champ select cannot load

This has happened frequently, while in a lobby with more than 2 people but less than 5, I hit accept to the game and it plays the music, but sends me back to the lobby window, my friends would be in the game but I would not, I think its caused by not having everyone added in the lobby, either way, its annoying and the only way to fix it is by quitting the client and taking the leave penalty.