Technically, you only need 1 guy to stop Yi, generally the jungler should already suffice.
Rammus, Xin, Udyr, Wukong, and Jax would forever remain as predators chasing to consume Yi. FIddlesticks would also counter Yi's very being if not for the fact Fiddlesticks is not currently in a good spot atm.
The trick is that CCs that have extremely short travel animation or outright no travel animation at all, ensure the Yi would be guaranteed to be hit and subsequently die because of how glassy he is. All of the junglers that hard-counter Yi have this property, and better yet, many of whom are also stat-checks themselves. This is also why Yi is less preferred the higher elo you go, as he's easily countered by enemy junglers who not only counter him but also do the same job as he does.
Outside of junglers, there are also other champs that mortally counter Yi, with the same logic of hard CC with short or no traveling animation. Lissandra, Lulu, Ryze, Annie, Cassiopeia, Singed, Orianna, Pantheon, etc etc all have this.
Similarly, things that are flat out impossible to avoid like Vi's R is also a sure-fire way to destroying Yi by itself alone.
......
The issue is that in low elo people often don't have the coordination to properly follow up on another's plays, and thus most teams have no focus and leads aren't followed, resulting in someone like Yi performing far better than his actual balance spot is.