Diana Passive not proccing Hunter's Talisman damage

As far as I know, Hunter's Talisman should work on all damage instance from a champion not just specifically something that procs spell effects, so it should work with Diana's passive. However, after playing a few games of Diana jungle I noticed it doesn't work properly and makes clearing the jungle early as Diana a bit difficult.