have you tried playing a strong invasive jungler and just invading him constantly before level 5? That should put him behind and if you have wards in your jungle and he tries to find food on your side of the map it's an easy counter invade.
You may say 'they'll do the same to me' but in reality that's what makes counter jungler's strong, they have the ability to not only take farm but also RUN AWAY!
Good examples being

Lee sin can bully him early and control him late by jumping towards him and punting the little freak away
shaco invades are legendarily annoying and strong while having good map pressure before finally being able to oneshot the freak once he gets 4 items
aand udyr is udyr, he can take and take and take an nobody's gonna catch him. Late game with swifties udyr should be able to dive in onto the koggles and at very minimum stun him. Losing a lot of hp in the process but that's why you open the gates instead and just push like yo mama.