Think of it this way (although Gaxx already made a decent summary).
If you don't control your minion waves and the enemy team is able to get their minions to your base, your base is under immediate threat. The minions are the warning that the base is about to be under attack. If you get attacked, you should have seen it coming and it shouldn't be much of a surprise if someone shows up.
In a backdoor situation, you might have the minion waves under control and not expect the master yi, kassadin, or nasus TP to a ward in your base and wreak havoc by his lonesome. There is no warning.