This is basically the exact strategy you want to take. The best way to deal with a Yi or any devourer jungler for that matter is to set them so far behind they won't be able to be remotely relevant until 40 minutes or so into the game.
I took this approach playing Evelynn, top laner and I invaded and killed both him and their top laner, stole his buff and that entire side of his jungle, cleared my own buff, invaded him at his second buff, killed him and stole that buff with my mid laner, cleared my side, invaded him on the ward I dropped and killed him again. Here he starts out 0/3, didn't even have enough money to purchase the second piece of his smite item let alone consider purchasing devourer. I naturally out ganked him after that, I think he may have had 2 stacks on devourer by the 20 minute surrender vote that flew out and the entire enemy team spent more time screaming at him than playing the game. That was also a 7 minute runeglaive purchase with change.
Counter jungle when you can, get deep wards so you can track him, try and invade at buffs. I also usually take the strategy of setting up a tent in a single lane at a time. If my other lanes can handle themselves, I camp a lane so hard that laner tilts and starts blaming Yi, who inevitably fights back in chat and now 2 people are too distracted to play properly, then you move on to the next lane.
A general tip against all Dev junglers, is to control the objectives, scuttle crab, dragon, rift herald. They need 30 stacks to get sated, and if you're controlling everything that grants more than one stack they need to clear camps and get kills 30 times, so it'll take a long time.
aren't supposed to kill 
1v1. But you can invade him at his 1st or 2nd buff and kill him when he is low. Always do the Scuttle Crab and take his camps if you know 
