Why does matchmaking "hang" if you've been afk in your premade lobby for awhile?

By that I mean I made the lobby, maybe had to go walk away for 20 mins, and came back and started up queue

I've noticed that my queues will get up to times of like 15-20 minutes and still not pop, but as soon as I back off and remake the lobby I'll get a game within like 30 seconds.

Anyone else notice this?