How do I deal with splitpush Udyr (Godyr)?

Basically, I get into a game as Cassiopeia and there's an enemy jungle Udyr, who ends up actually never ganking ever. After early game, the enemy team is pretty ahead, but then later on we start catching them out and coming back, and the game actually seemed winnable. That is, until Udyr emerged from the jungle. By the time any of us could get there to deal with him, he'd already have gotten an entire turret, and nobody could catch or burst him because he was super tanky and would just spam bear stance and run away. I ended the game 16/4 with a Rylai's on a champion who has a single-target spell up every 0.5 seconds AND a speed boost if I land my Q, and not even I could catch him. We could never engage in an actual teamfight with the enemy team because Udyr would just take an inhibitor in under 5 seconds, and then a nexus turret on top of that. I guess my question is, how should I deal with this? Is there a way to shut him down in the jungle before he gets all of those devourer stacks? Keep in mind I really only play Team Builder, so counterpicks aren't an option. I just have to use what I'm given.