I keep getting thrown out of low priority queue.

I know why I have to wait 20 minutes before a match (I left a match due to low fps). But I keep getting thrown out of the queue in the last 5 minutes or right after the 20 minute wait. It's honestly kind of annoying since I have already waited out my 100 minutes but I still have low priority queue.