What you're suggesting is called "Prisoner's Island", which Riot has already stated numerous times that they will not be implementing in League of Legends. If a player truly wants to reform then they will put in the actual effort that it takes to do so. If they won't then that's what permabans are for, to prevent them from ruining any further games with their own poor behavior. From Lyte:
Well, in discussions we commonly refer to this as the Prisoner's Island idea--basically, all the negative toxic players get lumped together on their own island and get matchmade with each other.
This would definitely not be usable in Ranked because it would split the ladder into two demographics and no longer be about the best players.
In Normal Modes... there's a lot of potential discussion points that are just difficult to predict. For example, we know that toxic players tend to experience more toxicity because their teammates or opponents reciprocate their toxicity--that means on Prisoner's Island, the average game is going to be horrendously toxic to the point that the players might just quit (or create smurfs and wreck havoc on low level players).
Secondly, what happens when someone on Prisoner's Island invites a friend that is new to the game? The new player experiences Prisoner's Island and never wants to play again.
There's actually quite a few more issues, it's a pretty interesting topic around the offices.