With Veigar's raw ap, I find he destroys structures plenty fast anyways with 500-600 damage auto attacks.
I wouldn't buy lich bane to use auto attacks against champions as Veigar. You're too immobile that being in auto attack range for a teamfight is just a no. Your cooldowns are sufficiently short that you can spam q and w with minimal downtime to use for repositioning, no need to throw in auto attacks at that point unless you're trying to finish off a low health enemy.
I generally get lich bane mid ONLY for taking structures when my ADC can't be trusted (either repeatedly dies in the won teamfight, or keeps chasing kills instead of winning the game). So I get lich bane to make myself able to take the turrets instead.
Veigar doesn't really need that because he gets so much more raw ap than other mid lanes. He may use it even better, but if you're doing 600 damage per auto already, you don't need to get it to 1200. You're better off investing in either more burst to win that teamfight, or more survivability.
Although I will admit that the CDR, mana, and MS, are all attractive still.