You can keep speculating when you don't play the champ, or even watch anyone that does (apparently). He doesn't take enough hits in the jungle for the change to affect his first clear health very much.
Yeah, he'll have less armor/MR if someone invades, but that's not a huge nerf considering most champs couldn't safely attempt to kill him in the jungle with or without the armor/MR since they will STILL have significantly less HP than him when they attempt it.
Someone attempting to invade wasn't and still won't be a huge issue for Graves, especially once he grabs smokescreen. At worst, it means he's putting a point in W instead of Q at level 3, which isn't a huge deal.
Lee Sin MIGHT be able to do something against him after, but tbh, probably not in the majority of cases unless the Graves makes it completely, 100% free for him.