Kai'sa is walking to target after the E ability
Much like the other ranged champions from time to time, Kai'sa will walk to target instead of attacking like normal. This happens especially after her e or attack speed ability.
A way to test this monster is to use the e ability and about a tick or 2 after visual conformation of movement speed is done, try attacking.
As for the amount of times for this occurrence I'd say it happens every game at least 4 different times. This happens to all ranged champions in the game so you might see it on an of chance.