Jungle XP question

So I'll skip the pleasantries, as I'm a bit tilted from that match.

So, I played Shyvana vs Nasus jungle, not that interesting, I cleared my Red, then went to steal his Raptors and Red, stole a bunch of his Krugs as well, then a bit later we both die to each other (bad damage math on my side and a bit of greed).

Anyway, I proceed to steal his Raptors again, haven't encountered him again yet, but after I see him again a bit later, he is level 6 to my level 5. He didn't gank lanes nor was he visiting lanes to soak XP from there, as he didn't appear on the map between when we killed each other and when he showed up as level 6 while having almost half my CS, and he was not invading either, which after the jungle XP changes, he shouldn't be able to have that level lead on me.

So my question is, is there anyway for him to legitely get a level lead over me when he wasn't ganking nor being able to farm very much?

Yes, we lost and I ended 3-10 while he got big, but that's the thing, he got too big too fast and I just can't understand where his XP came from as no matter how I spin it, almost half my CS + no ganks + no invades =/= him having a level lead.