I did a bunch more math comparing common full builds on various AD/AP champs. What you're saying about Rabadon's appears to be the closest thing to a right answer.
Ultimately the right answer in the case of TLD is that it outright favors AD without mathematical explanation. Guinsoo's and DFT math out only in the most extreme examples of AP stacking (Anivia) in the super-late game but only when compared to early-focused builds (Talon).
I think the conclusion is that TLD always favors AD. DFT favors AD against the majority of AP builds, and it still favors AD for the majority of a game's duration against the highest AP builds. Guinsoo's favors AD in 100% of builds that use it
Many AD items have crit and attack speed which provide zero benefit to abilities, meaning that's all wasted stats if you're trying to make your abilities deal more damage
They also have more hp and life-return in general. Mages have more CDR and mana/regen, but it's clearly less taxing on the item budget.
AP focused champions tend to have more of their damage locked into their abilities, meaning they have a larger up-front burst when they use all their abilities in a combo
You would be surprised. Here are some total combo ratios at max level:
Kat: 3.75 (QEWR, full channel ulti), 5.55g per damage
LB: 3.6 (WQRE), 5.79g per damage
Syndra: 2.99 (QEWR with 6-ball ulti), 7g per damage
Malz: 2.9 (full dot durations), 7.18g per damage
Anivia: 3.0 (both hits on Q, crit on E, 4 seconds in ulti), 7g per damage
Lux: 2.65 (QaaEERaa, includes 3 passive procs), 7.86g per damage
Zed: 5.13 (RWQE), 6.33g per damage (7.43 during ulti)
Lee Sin: 4.8 (QQER), 6.77g per damage
Riven: 4.28 (RQQQEWR, no AAs, but passive would put it at 7.78 bonus AD ratio)(full execute on ulti puts it at 5.52 for abilities only), 7.59g per damage (4.18g with bonus AD scaling from passive)
Talon: 4.62 (full bleed, both hits on W and R) 7g per damage
And bear in mind, this is bonus AD ratios only, and abilities that scale off total AD are calculate here only as bonus AD since we're comparing scaling from items.
That math disproves the myth that AP casters benefit disproportionately from ability scaling to make up for the lack of damage on AAs. Aside from Leblanc, it's almost completely untrue. Moreover, the reverse is often true with AD casters having better ability scaling by every metric relative to AP casters.
The same applies to base damages as well, so that argument can't be made in favor of AP casters. The answer really is that AP casters' abilities scale around similar rates to AD casters but only pull ahead on scaling per gold with a late-game Deathcap. In other words, the tax for long range is having no AA damage and inefficient to average ability scaling until late game.