There are a couple of reasons why you've lost to the Baron buff.
When enemy teams are ahead, they take the buff not to win more, but to deny your team the chance to push back. The buff in this case is not winning the game for them, because if they're that far ahead, the game is probably lost to begin with.
Secondly, and this is extremely important, but the Baron buff is utterly useless without minions and lane presence. What this means, and it's something I've seen very few teams actually do, is that your team needs to push the minions back HARD. Making opponents with the Baron struggle to get the lanes pushed into your base wastes their time, so the moment the enemy takes the Baron, your team should be clear cutting the lanes to halt progress.
I've seen far too many games where there are big hives of minions knocking on the base door when the enemy takes the Baron, and no one clears them until an opponent shows up to rush the lane along. Preemptive measures need to be taken to slow down the Baron buff, and while killing the opponent is the best way to rid it (something which does not happen with thr Elder buff), pushing back helps the best.