Okay, so this is what happened.
Rengar is not immune to CC while casting Q, but he started the Q cast before you hit him with Headbutt. The Q cast and animation is not canceled by crowd control; if he's stunned in the middle of it, he'll still complete the cast, but for all intents and purposes be considered 'stunned'.
Now, when you headbutted him, because he was mid animation, you went passed him, and thus were not hit by his Q. Because no one was hit in the first 'cleave' animation of his Q, he gained the little hop that the Q does.
This little hop reset his position. You can see him start moving backwards from the headbutt, but the hop activates from the place where he started Q, and thus moves him back, and then he hops forward.
Not a bug, just weird interactions.