Time penalty for leaving a game when I didn't leave it

I just had a game where we were in champion select. Our first pick selected his guy, then the enemy team selected their first. However the second pick on the enemy team timed out. I expected to return to the waiting Q but was notified that I had the 5 minute wait punishment for abandoning a game.

WTF?!!!