First of all you done goofed to begin with if you let the game get to that point, since Yi is equivalent jungle Nasus and just farms and never ganks so his team is fighting 4v5 half the game.
Second, damage is just Yi's thing. It's the only thing he brings to the team and as such his late game damage should be rightfully solid. That being said, his late game, while strong in it's own right, still has critical weaknesses. He's extremely vulnerable to CC. He's melee so you have time to hit him from range before he gets to you. His damage is mostly sustained (like an ADC) so even while he is chopping away at your teammate, you have time to focus and kill him (burst > sustained in competitive play, which is why he is literally the only assassin you don't see in LCS). He also has no way of initiating a teamfight. If you stay grouped and ward flanks so he can't make picks, you will find him easy to deal with.
You aren't supposed to build against him, he is quite possibly the strongest late-game duelist and you CANNOT 1v1 him late game. That being said, if you aren't teamfighting that late in the game and you are actively trying to fight Yi without help, then you screwed up twice.