To put this in easier terms to understand, this means if the enemy jungler steals red, blue, or raptors, you will essential be a level behind them, you being 3 and the enemy jungler 4 at full jungle clear. If the enemy mid or top are then dominating the lane, they would be level 5, making it impossible for you to do anything, and allowing them to roam without any danger from the jungler. Much like the 4-man-bot-gank meta that's going on right now, it would be incredibly hard to recover from and would essential determine the game in the first few minutes as well.
An enemy jungler would only have to take a buff camp(would now both give buffs and are be the only camps not affected by the xp drop) to both snowball and greatly reduce the xp available to your jungler. The jungle would essentially become a position where the objective is to steal the enemies camp first with one jungler always becoming useless for the rest of the game because of it(again much like our bot lane/adc meta right now).
Nunu, Ivern, and any counter jungler would become the the new meta. Not to mention any jungler that needs to farm till 6 would become irrelevant. Some currently being: Warwick, Master Yi, Amumu.