[CLIENT] Loss of invite rights when giving back ownership

When in a lobby, if you were given invite rights by the owner, then ownership is bounced around between you and the original person, you lose invite rights.

1. Enter lobby
2. Be given invite rights
3. Be given ownership rights.
4. Give away ownership rights.

At this point you lose the ability to invite people. Happened 10/10 times.