An off-meta build post? With math involved? I'm home.
What we are trying to calculate is on-hit DPS. Your AD is going to be the same for all the builds at 72.3. His attack speed bonus at level eight is +8.66%, for a total of .679 attacks per second, without build any additional AS.
When your AP is greater than twice your bonus AD, you deal 50% of your AP as on-hit damage to turrets. According to champion.gg, the highest win percent Malzahar set up will give you 20.16 AP from Runes, and 6.64 from the Natural Talent Mastery (at level eight). Morellonomicon gives you 100 more AP, bringing our total to 126.8.
Let's examine our options!
Nashor's Tooth is a strong candidate because it brings a respectable 80 AP, and is the only option here to give AS. Its AP bonus is the same as Liandry's Torment and Void Staff, and greater than Rylai's Crystal Sceptre, so I will not analyze those items, as they could not possibly beat Nashor's where turret pressure is concerned. The attack speed bonus brings a level eight Malzahar's attack speed up to .991; the AP total is 206.8.
Rabbadon's Deathcap is the only AP item commonly built on Malzahar with more than the 80 AP that Nashor's Provides. In fact, with its passibe, Deathcap will bring Malzahar's total AP up to 333.18.
Lich Bane also gives a solid 80 AP, but that is not why it is being considered. After casting a spell, Malzahar's next basic attack on the turret will do 100% of his AP instead of just 50% (total is 206.8 again). This has a 1.5s CD, but can Malzahar make optimum use of this? With Morellonomicon and Lich Bane, he has 30% CDR, but still has rather long cooldowns. Q has a CD of 4.2s, W has a CD of 5.6s, and E has a CD of 9.1 (again, based off champion.gg's highest win percent skill order; even if you max E for extra CDR, you can only get it down to 6.3s). Doing this might actually be a good idea, as when you do, you can come pretty close to getting the spellblade effect as often as you can. Going forward, I will assume you can.
Alright! To actually compare them all, I will just do raw DPS calculations (AS * [AD + On-Hit])
Summation
AS: .991
AD: 72.3
On-Hit: 103.4 = 206.8 / 2
Raw DPS: 174.1187
AS: .679
AD: 72.3
On-Hit: 166.59 = 333.18 / 2
Raw DPS: 162.20631
Special: Actually weaving Spellblade autos into an attack pattern makes things tricky. Malzahar's attack speed is really slow, so to get maximum damage, you might have to skip auto-attacks. If you wait for Lich Bane's CD, your AS is effectively .666, which is almost imperceptible less than normal, so I will assume you are just doing that. This also means that only 75% of your AD is applied to the attack, because that is how Spellblade modifies basic attacks.
AS: .666
AD: 54.225
On-Hit: 206.8
Raw DPS: 173.84265
TL;DR
This is really close, but Nashor's wins over Lich Bane.
Cheers!
-Nixtarma
Note: Raw DPS does not account for the turrets' considerable resistances, nor does it include Malzahar's penetration stats.