So now If I build ROA, Rylais, Liandrys as an AP, midlane mage I'm giving the enemy ADC a buff?

Simply because you picked an AP champion, you're already increasing the enemy team's chances of winning.

So far it seems that getting crit and ad is far more important now then getting armor pen and giant slayer.