Teambuilder insta-leavers

Sorry this is a vent post in a way. I'm tired and somewhat cranky so I am gonna vent.

I'm annoyed by people who join your teambuilder and then when you accept them they leave instantly if there isn't 4 other people there. It's so annoying and makes it nearly impossible to form a team when people keep leaving. Some people will wait a couple minutes or so before leaving as well. It's kinda annoying when trying to set-up a team. I usually don't mind waiting because I just play Hearthstone or do other stuff while I wait, but still it's so annoying when you think "We just need one more person!" then suddenly people start dropping from your team.