Um....so punish the team that's either trying to finish off the game or make a comeback? What if a slightly losing team is trying to get baron, get in that last team fight, then win the game? Now they have to get baron, get OUT of the pit, THEN recall while the enemy team gets drag and pushes into their base for the win. Why punish a team for making an already risky play that could lose them the game, and make it even easier for them to lose the game just for "baroning". A HUGE strategy in this game is doing a speedy baron, then immediately recalling to defend their base, because the enemy is either pushing, OR trying to steal the baron. You may find it hilarious, but any team that get's aced/almost wiped out because Anivia walled them in, just after they got the baron that should have won them the game, will definitely not like it as much.
This could be a funny gamemode "everything fights back" or something similar, but for the real game this would ruin much of the baron gameplay and strats. If this was implemented baron would be TOO much risk for its reward.