What's wrong with spectator mode?

Everytime I try to spectate someone it kicks me out right after the game loads and says to reconnect or quit