There are multiple systems in place. The one most people talk about is the system that handles behavior in chat. This one will automatically analyze chat logs for a player after a game if that player is reported (by even just one person). If the system finds punishable behavior in that game, it will basically flag it as a mark against that player. If the player gets enough marks against them over a certain period of time, that is when they'll receive a punishment of some sort.
Next up is LeaverBuster. This is another automated system, with a sole purpose of detecting AFKs or leavers. Leavers are fairly obvious, as are people who just AFK in fountain. But it can also detect if somebody is just running in circles in the base or perpetually staying as far from their team as they can while doing nothing.
The third that I'm aware of is a newer system, built to catch intentional feeding. I know less about it because it's generally been pretty quiet as far as the boards go, but you can read some about it here. I'd guess we don't hear much about it in this section because the players who get caught by it understand what they did and don't bother to come in here and try to fight it.
And on top of all that, while Riot doesn't manually review everything, they do perform periodic manual audits of all the systems in place just to make sure they're still working as intended.