Veigar's ultimate scaling with his opponent's AP would be fair
IF every AP champion had a viable AD build. Because then it would punish them for the choice they made in game to go for one build over the other. Since that's not the case, it just arbitrarily does a totally unbalanced amount of damage to someone because of the champion they picked, which is stupid since "don't build AP" isn't actually something that mages can do without being totally useless.
Soft counters are ok, but if your opponent can oneshot you purely because of the champion you picked, that's incredibly stupid.