[CLIENT] Can't start game after pressing play again without the party owner

After I played a game in a 2 man party, I pressed the play again button in the after game screen, and since the guy who owned the party was afk I tried to leave the party. I pressed the X icon next to the grayed out Find Match button in the lobby screen, and and left the party. But instead of peing able to create a new lobby by pressing the Play button, the "play button" still said Lobby and it would return me to the lobby I just left and I had to wait for the Party Owner to return to be able to leave that lobby and play a game.