Losing the first blue buff can straight up lose the game.
For example: I was in a game, and the enemy invaded our blue at level 1. Jungler wasn't able to fend them off, and he died. They stole blue. Jungler was underleveled the entire game. Jungler couldn't gank for 15 minutes while he tried to farm (basically uncontested). He was executed twice. Jungler kept dying when walking through river/into lanes because he was so underleveled he was 2-shot by enemies. And if the tilt wasn't enough, the enemy jungler started counterjungling hard at 10 minutes, doing a full clear of our jungle, while our jungler couldn't go into the enemy jungle because he was way too starved. We lost because we had no ganks, a 0/7 jungler, and it all started because of a level 1 invade.
If losing one blue buff makes the jungler unable to farm back into the game, that's an issue. The jungler shouldn't be that impactful on a game.