Declining a game in queue when I hit accept - Client

(Sorry I cannot figure out how to tag it as client) But when I am in Normal Draft queue and it pops up saying a game is ready, I hit the orange check. Out of the last about 20 games I've played, I have had it happen 2 times where even though I hit the orange check, it says "You have declined" and it returns me to the lobby and ends the queue. I experience no lag in the client or in game, ever, that would cause me to lag click it and I am sure I am not just missing it. My steps were sort've in my description...