[Gameplay] No devourer stacks for Rift Herald unless continuous combat

As a jungler, I would prefer to give the buff to my top laner. However, I am often not tanky enough to tank it, and I usually have much more attack speed to make use of the backstab (I like devourer junglers). What I've noticed is that if I stop attacking long enough, I get no stacks. I wasn't tanking damage either. I was just standing behind it waiting for my top laner to take out the remaining 200 hp or so. This is problematic because I feel like I'm either robbing my top laner of a buff I don't need if I take it, or I'm being robbed of my devourer stacks if the top laner gets it.