As already pointed out, IP ban would have zero impact on the game, it requires no special skills or effort with the vast majority of providers to change your IP (which in most cases happens automatically when you reconnect to the provider).
I believe they should give each player a behavior score and given a lot of benefits to good behavior.
For example for match making, good behavior players get matched up and have priority in the queue and auto-fill protection, while poorly behaved players get tossed in with each other.
You could have a system of approval levels of behavior. For example you can queue by saying anyone with a 90% positive score or higher, that way though you might be in the queue longer you are guaranteed to play with well behaved players.
I also think the rewards for having high honors and good behavior should be extremely high. For example double essence for leveling, special reward tiers, advanced access to newly released champions, invitation to "good behavior" divisions and leagues.
Make it so that the benefit of being well behaved are so good, that players willingly change their attitudes so they can gain access to them.
I also think that people with high honors and a good behavior rating should have more impact when reporting people and people with bad behavior, should have their reporting impact and priority significantly reduced.
Punish heavily without banning and reward good behavior, that is the theme to how to fix a toxic community. It would not only improve the behavior but eliminate the benefits and logic of creating smurf accounts.