Never played Warcraft 3, but it sounds like "Backfire" from GW's Mesmers so I think I have a point of reference.
In GW, it was a "hex" spell that was cast on to a unit and would take damage on spell use (spell merely being a skill type in that game). This wouldn't translate well because there are two systems that lack the support for it, I think.
First, there's not a good hex/debuff/cc system. You can Cleanse and Mikhael's things on huge cooldowns to remove them so anything less than an ultimate would be oppressive to those laning against it. Since it can't be reliably removed as part of the system - it would need to have its own removal clause or would need to deal piss poor damage. Neither is a particularly great choice.
Second, there's no skill differentiation system either. Is Vayne's "Tumble" a "spell"? Is her "Condemn"? They both use physical items in her possession. THey both scale with her attack damage. But they would probably be considered a spell even though the main target of this Backfire-esque caster would be Mages and Supports, those who scale with AP or CDR.
It still doesn't seem totally unhealthy, though. Perhaps an ability that pings damage every time they use an ability unless they break combat for X seconds. Put it on a support and it's either heavy damage or efficient peel - but that's really up to the opponent. It'd probably suffer a lot against fighters and tanks that can kill a target or two while taking a beating.