From my understanding, it's because people do not want to deal with yi as a teammate or as an opponent. It's like yasuo, it's not fun to deal with them, op or not.
As a teammate, yi either doesn't offer anything (especially early game), or single-handedly carry the game and make you feel insignificant: people want to be the carry, not the one letting others carry.
As an opponent, it's similar concept. Even if enemy yi farms up and doesn't gank, you can't trust your own junglar to punish that by getting a lot of work done early. If enemy yi gets 2 items, suddenly you have no way to deal with him and you can't trust your pug team to coordinately deal with his engage - heck, you can't even rely on your team to be around when he jumps on you.