I don't know exactly how the new system would work, but here is how the old one did.
You would go to the Refer-A-Friend page under "Community" on the top bar of this website. There would be two options. One where you could input a friends e-mail and Riot would send them an invite. I never got that option to work. The second was you could copy+paste a unique link tied to your account. Anybody that signed up using your link (or you invite that never shipped) would be credited to you as a referral.
Once a referred person hit level 5 you could go to the referral website yourself and claim a 250IP prize. If/when your friends hit level 10 you could go claim a second set of prizes. Most second prizes were just another 250IP(500 total per referral), however, some break points had special prizes. Like 5 referrals got you an extra rune page or 25 got you a special Warwick skin.
The real issue was that with like 50 or something referrals got a chunk of RP. Some players were running bots to level phony accounts by the dozen in order to get free RP. It seemed that you could actually play low level matches and be the only human playing the game.
Now Riot is reducing the threshold to earn some of the rewards and eliminating the free RP ones.