The punishment system does work because it's primary goal is removal: reform is left up to the player. Even if 90% or permanently banned players return (a high estimate for lack of data) you've still removed 10%.
Plus, part of the punishment system is designed to discourage others from taking the same action. Punishment may not help discourage future crime from an individual, but the idea of a punishment often discourages those who have not committed the crime from doing so in the first place.
I do think the system could do a little more to encourage reform, but I'm frankly not sure how to do that. It already gives numerous warnings in most circumstances, and escalates punishment slowly so your first two punishments don't actually stop you from playing the game. Short of tangibly rewarding toxic players for improved behavior, which I don't approve of because well behaved players don't get that, I'm frankly not sure what else you can do. Especially now, where rewards are tied to honor, so there is a tangible reward for behaving well. If the carrot of honor based rewards and the stick of a potential ban are not sufficient to help someone reform, I'm not sure what else Riot can do.