To be clear, you're saying that the group with fewer numbers decides for the team? Should 2 people be able to bully the rest into playing? Does this make sense anywhere? The matchmaking system doesn't elect political figures to represent each team.
It's not as though the vote can happen at any point. There's a 20 minute wait and an additional delay after a vote begins. 20 minutes is enough time to get a sense of how the game is going, and a come back is always possible, but continuing shouldn't be decided by the minority. If the game ends, players can play another or not. If the game continues, everyone is in until the end possibly against their will.
The game shouldn't continue because a duo is hopelessly optimistic. I've been IN of enough duos who have this ability to force the team to continue because we have the power to.