I think I just start banning that stupid champ. There is counterplay against him, but only a little. It depends on reaction times and prediction of players if he can get stunned by a skillshot or AoE stun or he disappears into the air. Since he can use the Q multiple times, he can dodge more stuns than a Fizz would, making him one of the most slippery champions (Olaf can ignore stuns, but hey, he doesn't run at ridiculous speed using just ult and he doesn't have dash in his skill set either).
The original solution to champs like that was a targeted stun. But Riot seems to be removing those one by one in reworks. There is still Fiddle's fear - it makes enemy slowly walk away in fear (but Yi cannot be slowed in ult, so he runs super fast, hehe), and there are some melee champs who have a targeted stun in melee range (e.g. Irelia if she's lower on HP than Yi), but in a typical scenario, a Yi that can dodge skillshot CC rather well by his Q is almost impossible to counter. Which is quite sad. A fed Yi can be worse menace than a fed Darius or a fed marksman. He just 3-shots you in 0.3 seconds and then leaps to someone else while making you throw your skillshot stuns into the air hitting nobody.
So I agree, Yi is damn annoying to play against unless his team loses hard enough because of is unwillingness to gank.