Can't surrender even when all voted up

Yesterday I had rly messy match. We palyed good, we played bad, happends but when one of my team-mates left in ragequit (after 20min) we decided to surr (all 4 voted up) Then only two of us "left/afk (dunno) the game and only I and top-laner stayed and we had to wait til enemy destroy the nexus ( we couldn't vote to surr again).