Getting kicked out of Queue (client)

Essentially, if I am in a queue, with a party or not (and this happens every time) and someone declines the ready/check, the big symbol that says 'accept' won't go away, and it will kick me out of queue saying I was afk.