Sure you can. Just like when you are in a restaurant, a movie theater, any public places, you have to mind other people around you. When you are in a restaurant with 3 of your friend and there's one random guy eating at the next table, if you start throwing vulgarities at your friends jokingly, you are still going to be asked to leave because the restaurant owner has to respect all his patrons and you are disturbing the other guy.
In this game, you can't exactly ask the 5th guy to mute the 4 of you and basically forced him to play the game in complete silence. He has the option to mute you all if he wanted to, but the choice should not be forced on him. If you are to throw toxic jokes with your friends (just like any private venues, e.g. Rito Games, in real life) and someone has to read them, you will probably be reported and 100% deserved the ban.