You did let them take the elder, that's a game ending mistake. That's all.
Also I already won games in which the other team had elder and lost games in which I got the elder. The buff is strong, but not THAT strong.
This buff literally makes the dragons worthless, rift herald worthless, and baron worthless. It doesn't matter if you have 4 dragons, the dragon soul, and baron buff. If the other team gets elder, those dragon and baron buffs mean absolutely nothing and you just lose.
That's just false, as mentionned above. Most of the time my teamfights ends with a teams staying at more than 50% hp, especially if this is the team which is ahead, so the execute buff is worthless in those case.
Another option is to just try to depush and do not fight the ennemy team, the buff doesnt last long, you can even salvage an inhibitor if you are that ahead.
Of course if your first reaction to the fact the ennemy team took the elder is "wandering in the jungle without vision" you are going to lose, but, eh, that's totaly deserved.