Alt + tab while loading into game = late join

So when I alt tab during a loading time, I get crazy delays and usually have to restart my pc/client numerous times in order to fix this. It stinks cause I ruin the game for others potentially, and I hate the repetition of exiting the client and restarting it. Any help?