how does riot tell if someones boosting or just using their smurf

Riot isn't going to reveal how they find out because that would allow abusers to circumvent their methods. As for making a smurf account, you're on the same IP as your main, Riot probably can figure out that you're just making a smurf and not boosting

I would guess a very sudden change in quality of play. If you've played 500 games as a silver player and then a diamond player hops on your account, it's going to become very obvious very quickly that you aren't simply on a lucky winning streak. Smurfs by comparison have no prior history to compare to so you should be ok.