Longer wait times if you fill as a group?

Recently I've been choosing fill for my role (like since season 4) for normal's, and wait times are getting ridiculous. Queuing by myself with an estimated wait time around 1:20 I'll find a match at around 1:40- 2:00 however if I duo queue, that same wait time jumps up to around 4:30-5:00. To make maters worst WE ARE BOTH FILL. What in the world is going on, I know a duo is treated differently but really? An additional 2-3 mins on average to the estimated time? Sorry buddy but I'm solo now.