I think instead of granting him AP, his passive should grant him stacks 'evil power' (or some other arbitrary name). The number of stacks he has will determine the percentage of bonus AP he receives.
This will fix two things.
He will be significantly weaker early game because he will not have high AP totals (compared to other champions of the same level with the same amount of gold invested in AP), so a small percentage, or even a larger percentage will not amount to much. Later in the game his AP totals from items, runes, masteries, buffs, .. and so on, will be higher so his accumulated stacks will grant a stronger percentage, allowing him to scale up to the monster he is now.
The other thing this will fix is the issue of Veigar building very low amounts of AP, and still being an AP monster by simply being an annoying little shit. He will need to build AP items so the higher percentage of bonus AP will amount to something.
If there is any concern for this making his early game too much weaker he can start with a decently high percentage with a low number of stacks, and the percentage increases more slowly or with a larger number of stacks needed to reach the next tier.
For example,
5 Stacks = +20% bonus AP
10 Stacks = +35% bonus AP
20 stacks = +48% bonus AP
35 Stacks = +55% bonus AP
45 Stacks = +60% bonus AP
XX Stacks = +XX% bonus AP
These are not intended to be balanced numbers, but they are meant to describe what I mean.
Notice how the number of stacks needed to reach the next Tier increases, but the bonus granted by reaching the next Tier decreases.