How does the game decide how long someone gets to connect before a game starts?

There seems to be some wild discrepancy sometimes in the time in which the game takes to load a game without someone. There have been games that have loaded near immediately with a person still not connected and there are also games where the game seems to wait days before starting the game. What dictates this?