Because playing Yorick doesnt feel good for the other team. Playing against him is like playing against tryn, you can beat him in lane but he will continue to split and have the damage to consistently fight you the whole game.
if you ignore him, he takes towers with TF + his ghouls very quickly. Even if you are ahead of him, he can go even in a 1v1 because of how much damage he can do with his ult up. If you send someone to gank him, now your team is 3v4 for another objective.
In most of the games I have personally played vs yorick, the yorick goes something awful on the scoreboard, like 3/14, but takes 1, maybe 2 inhibs, and usually is trying to win a base race alone when the game ends. Sometimes he wins, most of the time he doesnt.
So while Yorick might get beaten more often than not, he feels like shit to play against.