That's why you don't all-in him, that's his job, and unless you're also an assassin or a burst mage, he's likely to do it faster than you can.
Turns out, one of Zed's biggest weaknesses is how he's also extremely squishy, and that makes him pretty vulnerable to poke harassment. Another big weakness is he has to play safely until "the right moment" appears, one of the same weaknesses a lot of burst mages and other assassins have.
It also turns out many assassins and burst mages are actually really weak to taking constant poke damage.
Also, if Zed decides he wants to ult you, turn around. As in, he literally reappears directly behind your champion, so if you have some sort of stun/root/knockback/slow/anything to prevent him from moving or dealing damage to you, use it directly behind your champion the moment he appears. Especially if he decides to turret dive you. That's a really easy way to kill him with little effort.
Zed is definitely not invincible, his playstyle is "stay safe, poke, and wait for a moment to ult>unleash combo". As long as you play safely, and deny him the potential of dealing any meaningful ult damage (it's not the ult so much as all the other damage he deals while the ult is activating), he's not gonna be able to do much of anything to you.
Also, does Zed ever max his W first, or even second? While it would give him a shorter cooldown, it would also limit the potential damage he's able to do in a combo.