Lich bane is core to his burst though, and fits his play style perfectly (AA>AA>Spell>AA>AA>Spell>AA gives you two Lich bane procs, and yes I know it's only 2 AAs, but rarely does AP Varus have the time to plunk out three AAs between procs, especially considering that it only takes 4-5 stacks to kill them anyway). Zhonya's is overrated in my opinion, especially now that it gives 20 less AP (basically, unless you need it to avoid a lethal Nidalee spear or an Ashe ult from down town, the better option is usually to be able to out-burst your opponent and rely on your in built disengage for safety). The reason I take Lich bane+Luden's together without boots is that it provides a huge amount of mobility on its own (400 MS with Luden's, Lich bane, MS quints, and MS mastery, not to mention dragon buff's MS if you can get it). Playing without Zhonya's is a bit risky, yes, but playing AP Varus is risky in and of itself since he has to get in and AA for his damage instead of being able to sit back and poke for days. Really it comes down to personal preference, because when I play AP Varus, I go balls-to-the-wall glass cannon and blow people up, but it definitely works the way you build him too, that's just not to my liking.
When I said His AAs don't hurt much, I meant they don't do as much as people suspect, especially if they have much armor and MR. They chunk for a fair amount vs squishies, but the real kicker is that even though they really only tickle tanks, his W proc makes tanks cry.