AP isn't a damage type.
It's a stat that scales the power of abilities.
The word(s) you're looking for would be either "ability thornmail" or "magic thornmail", as AP isn't exactly a thing you're hit with.
I mean - when Tryndamere uses his AP scaling Spinning Slash to hit you - that's physical and ability damage for the purposes of all existing defensive items, not "AP damage".
As for why there isn't a magic-damage or ability thornmail, that's probably a case of their just being to many issues for it to be a good thing that adds to the game.
First issue: Thornmail's ultimate purpose is offsetting life steal and essentially negating it. Pretty much every physical-heavy champion is going to get a life-steal item of some type.
Spell vampirisim isn't a good equivalent on the magic-heavy side. For on-hit magic damage dealers like Teemo and Lulu, it doesn't even apply to their on-hit damage. For those it does - spell vamp isn't very appealing to their playstyle, costing to much mana in-lane for what little it offers, and the AP exclusive Zhonya's active just being significantly better at keeping them alive late-game when it's nice to offer a burst of defense on a burst-caster, as opposed to life steal offering sustained defense on sustained damage dealers.
Second issue: items aren't terribly redundant in LoL, and that adds to the feel of certain build-paths and itemization routes. Their isn't an AP QSS, there isn't an AD Zhonya's. Even basic stats are set in certain patterns - life steal and armor penetration are only found on items with a solid chunk of AD and neither are found on items with critical strikes, HP/5 is usually found on items with solid MR, etc.
Counterparts between classes are more life The Bloodthirster vs. Zhonya's Hourglass: one offers sustained-damage champions the sustain necessary to survive despite being squishy, Zhonya's offers burst-damage champions a burst of survivability so they can survive despite being squishy. A lot cooler than just separating 2 build-paths by name only via mirror-items existing IMO.
Bursty and magical champions function rather differently than the physical auto-attackers that Thornmail currently targets: partially because items like Thornmail exist that target physical auto-attackers, and items like Banshee's Veil exist that target burst-casters.
Third issue: Thorns is a power-without-gameplay issue. It gives a lot of power for essentially no work at all. Rather than forcing a player to accurately time and land their skills, hit multiple units with AoE, or utilize a certain combo to get the most of their AP: Thornmail just punishes your opponents for doing the very things they should be trying to accomplish for their win-condition.
Which just sucks once it gets to a certain level of power - as the only way of dealing with Thornmail is avoiding the person wearing it entirely, which isn't far removed from not playing the game and just giving up: so not really an option in regarding counter-play strategies if Riot wants people to actually play their game.
In Dominion it actually can be that oppressive vs. a solid number of champions, in Treelines it was removed: largely due to power-without-gameplay issues if I remember the Rioter's reasoning for that correctly. In Rift you'll usually have enough armor pen, life steal, and MR to deal with thornmail once you actually run into someone with it.
Fourth issue: auto-attacks are a fairly slow way to kill people that give a reasonable response/reaction time to killing yourself with Thornmail.
Most abilities just don't afford that luxury - often being front-loaded and bursty, and ones that are delivered over some time like Syndra's Unleash Power mostly lock the caster in their animation time without offering a realistic out. Like you mentioned - the AoE thing would also be an issue.
Anyways - the culmination of those 4 things makes an anti-ability thorn item an unlikely addition to LoL's itemization.