No, it really won't, unless someone is scared of conventionally attractive women in tight bodysuits baring cleavage.
Between her appearance, her incredibly monotone voice work, and the insulting implication that Riot felt this character was supposed to be an unsettling horrifying hero ala Spawn or Ghost Rider, this might just be the most fucking boring champion released in a while.
Sorry, but bad job Riot.
If a goddamn void portal opened up and a woman stepped out wearing some strange purple suit, holding weapons, with goddamn shoulder mounted turbines, you’d be fairly startled.
Also your last line is incredibly childish and more likely to make Riot ignore your opinions.
"The Void consumes everything-"
-except Mary Sues.
I haven't really listened to the VO yet outside a few of her special interactions with her strange not-Kiwi accent, but is it completely straight edge? I might not be a Kayn fan, but at least the entire point is self-aware edge memes. Hell, even Xayah's got a tiny bit of self-awareness going on.
That line probably has more power to it when she's fully suited up.
Context, context, context.......
Kai'Sa does not say that to you as the player, but to a random guy she meets after leaving the void. Runeterra is a land with mostly medieval standards, the only exception are Piltover and Zaun.
You mentioned Galio from Demacia in a comment: Galio moving is neither a regular sight in Demacia nor does that mean a Demacian random guy wouldn't be terrified by Kai'Sa.
There is a woman coming out of a void energy portal. She can shoot plasma bolts. The best explanation a Demacian has for that would be magic. Demacians fear magic, they are actively hunting down mages in that society.
Another example: Mount Targon. Considering that these are preparing to fight the void and some might already have some experience with the void, they could probably figure out the source of her powers. Void creatures are the enemy and now there is that woman staying in front of a Targonian using obviously void energy.
Same applies to Shurima. They have experience with the void, thanks to beings like Rek'Sai roaming the desert.
To be feared, you don't necessarily have to look ugly. People fear what they don't know, or what they don't understand. They fear the unknown.
Kai'Sa works in that way. Your argument that she can't be feared because she looks beautiful doesn't make sense. It would mean there is no way to fear Evelynn either, since she looks sexy.
