As someone who plays the jungle nearly religiously, I disagree.
Buy a pink ward on your first back. Ask your teammates to do the same. If it is a counterjungler that has CC instead of dueling powers (i.e. Nunu), just chase him out of your jungle and call your teammates. If it is a strong duelist (i.e. Trundle), stay away and go farm somewhere else.
If they gank bottom lane and you're top lane, pay their upper jungle a visit. Pay attention to the laners and have an extra ward on hand.
This may also be due to the junglers you are playing. If the enemy jungler knows that they can 1v1 you and win, you're very likely to be invaded (i.e. Alistar and Shaco).