I don't think it's khazix damage that's the problem.

It's his mobility. With the current duskblade and him being able to dive, kill, then jump out is what makes him annoying to fight. It's like if I see him and I'm playing a non CC champ, I'll just leave him be unless he's trying to dive me.