How Does Ivern's Passive Cooldown Work

I have been playing Ivern for a while now and the only thing I am still unclear on is the cooldown of his passive. I have noticed that there are times when I am trying to clear jungle camps but his passive is on cooldown. I am wondering what determines this. I have noticed that the cooldown changes depending on where I am on the map. If you look AT 0:55 on the attached video link it shows Ivern's passive going on a 35 second cooldown but when he moves into the river it is increased to 70 seconds and is reduced again back to 25 seconds. Once he clears his blue camp the cooldown is reduced further to 5 seconds. Any information on why this occurs would be super helpful.