Her range is super short, and unless she finds some early kills she's got a meh early laning phase, easily bullied by someone like Lucian. I think part of why she's seen a ton above D5 is because she does require some skill to use.
That said, I can see how she's still a little too strong. The stealth she receives on her E after enough attack speed makes her hard to pin down. On the other hand she pretty much requires some kind of reposition tool because her range is so short and she has no real way to get back out or otherwise avoid damage after she ults in. Pairing her with Alistar is really killer, because he gives her some free passive stacks when he does his thing, and she gets to proc her passive easily after that, or ult in if she wants. The passive stacks is probably what needs to be looked at most, perhaps a straight nerf of numbers or a reduction of how many stacks her W puts on targets.
EDIT: I don't think people realize, OP title says "why is Kai'Sa allowed to be so busted?" I think she's STRONG and could use some nerfs. I don't think she's BUSTED and I don't think any Champion deserves to be dumpstered with excessive nerfs. I'd rather see a slower approach. Thanks for the discussion.