Am I counter jungling correctly or am I missing something?

Hello all, I am currently S5. I have just started to take ranked seriously and I love jungling. Recently I have played kindred in a couple of ranked matches and for the most part I've had success. I played a few normal matches today to try out some new routes and counter jungling tactics to use in ranked. For some reason whenever I counter jungle it doesn't seem like I'm having much of an effect to the enemy jungler.

For instance. I'm starting red side and my top leashes me gromp. I proceed to my blue and depending on who is the enemy jungler I plan out whether I want to steal the enemy buff and steal krugs or raptors while I'm at it or just ambush them out right while they do red buff. Let's say I just steal their buff & camps and leave a ward in the bush by red to know when they enter top side. I gank top since I'm already positioned behind the enemy top laner and get first blood.

I then proceed to do a full clear of my jungle from wolves to raptors to red buff to krugs. (I informed my mid lane and support to what I was doing so I had them ward my jungle heavily so they could collapse on the enemy jungler if he dared step foot in my jungle) so I know all of my camps are still there. After the full clear I would get scuttle crab and back. Around this time I would be level 4 or one camp from level 5.

Around this time I get vision of the enemy jungler and he's doesn't seem to far behind me. Yes, his cs count is low but he is still around my level. Now knowing this do I continue to counter jungle him since he is top side and steal his camps on bot side and gank either mid or bot, or do I try to follow him, either counter gank or ambush him to really tilt him/shut him down. Granted all this he's level 4 as well. To me it seems like everything I've done doesn't have an effect on him besides his cs count.

Am I doing anything wrong or missing out on steps I should be doing to continue my small lead? Should I start hard counter jungling him, placing deep wards to give information and maybe kill him if we ever confront. Or should I just continue to gank my laners and snowball them for the rest of the game and only counter jungling when I have a mark, or I have vision of the enemy jungler.