I've played him Jungle more than a few times and I do so with an odd build where I focus on building damage items first so my clears require less mana and my autos hit harder, in fact, I start Red side and can give up the first blue if the mid needs it without even being worried about mana. Maybe it's just me, or maybe no one has tried that either. I do find that his ganks are only mediocre, not that they are bad, just so hard to manage them without a good gap closer, but if you get to land a Q on the enemy woo woo have to peel the Yorick then for sure!
It's just a matter of not being a very played champion at all. Try buying Warriors enchant on Jungler item sometime with him and just being a little more frugal with your spell casts while jungling, don't just spam them. Start on golems bros, leashed or not, auto, E, Smite, auto and E till dead then move to Red with W and rotate W and E on him between stuns from autos move to blue and kill it, then go kill Dragon using your 2 starting health pots to kill it, if you think they might ward it or you might need help or any danger at all toward early dragon then go counter jungle or gank instead. back buy Chilling Smite item and you can now gank much more easily, get your boots and a couple mana pots for when your blue wears off, jungle, counter jungle or whatever as needed. ward up rivers kill the river crabs as often as you can and aim for your big camps in your jungles avoid raptors and wolves unless you have an extra smite to get vision in your jungle or ward clearing power. Pretty straight forward, very high health coming out of the jungle. As for skilling order that's up to you, Masteries are 21/9 or 21/0/9 for mobility and longer buff durations. I like skilling E and Q back and forth for chasing power and because I have a slow on Smite, both scale with the AD so you hit pretty hard coming in, and the dueling capability is decent. Lee, Riven, Xin, and the usual laning all in champs that can just burst you down still give you the normal problems, but I find if you are focusing vision and warding appropriately it's not too bad.
I'm only gold elo atm though so maybe it's just easy down here. /shrug.