Friends of friends joining lobbies.

Is there a way to stop my friends joining the lobby that I'm in. This seems petty but sometimes I don't want to play with certain people on my friends list and it's a pain in the ass when someone join me when in someones lobby as I cannot prevent them from doing so, unless I ask the leader to switch the lobby to invite only which they won't do as their friends might want to join.

Is there a way to stop this? If not, can it be implemented?

<3