I don't know how you are playing him but his Mist Walkers are like his entire kit. He doesn't have much dueling potential without them, he can't really push without them, he can't really do anything without them because all he has is his Q and his E for damage. Without his Mist Walkers, you are like a worse version of Nasus without the stacks, you get 1 or 2 good hits in then you are fucked.
Build up your Graves until you have 3-4, raise your Mist Walkers, land your W, then your E and watch their HP melt in a matter of seconds. They proc Lifesteal so with Death's Dance you will be healing for a good chunk of HP, with Sheen you hit like a truck while your Mist Walkers chase them down. If they jump on you instead, pop your ult and watch their HP melt even faster while you heal for a crap ton.
EDIT: Also Old Yorick was a terrible champion, he was toxic to play against and pretty terrible to play as. He was fun, but he was completely unhealthy for the game. There is a reason Riot never put him in Free Champion Rotation and said if he ever became relevant they would nerf him immediately. He was bad design and bad for the game, new Yorick is better in almost every single way.