I'm not a good azir or anything but the worst thing for me is enemies that can easily get in my face.
This is because after using a soldier to charge azir suddenly has moved his zone of control away from himself, so instead of running back away out of range you can run toward azir and he basically can only auto attack you.
So people like zed, yasuo, yi, talon, ecko, Diana, kassadin basically force azir on the defensive because if he wants to use his charge to oppress their lane then they have free reign to jump him.
Also enemies that have long range or safe farm just stay out of his range and ignore him. So Xerath, morgana, malzahar, etc just kill minions and ignore his presence.
Once again I'm a bad azir, but these are things that I find annoying and that I have found useful when going against enemy azirs.