AP bruisers don't work because they don't work.
AP is inherently a burst stat. As its damage comes from abilities, in nearly all cases the best option is to stack up that AP so that single rotations do the most damage possible (and that's even on AP champs with more DPS leaning profiles, like Azir, Cassiopeia, and Ryze).
AD on the other hand has auto attacks, which are a direct scaler with said AD. That means that when abilities are on cooldown, you still have back up damage. In game, this plays out to bruisers and other fighters taking a few damage items, then specing into durability, as as long as they are auto attacking, they are doing their job.
AP bruisers don't have that option. Sure, you can make a mechanic where their auto attacks scale with AP instead of AD or something, but that only works on one champion (can't do it to everyone, as then they are all slight reskins of each other).
Until some kind of fundamental change in the game happens, making it so that AP is not just a burst stat, AP bruisers will never work. If they have good target access, then they will build heavily into burst and raw AP, ignoring any durability stats, and become binary "kill you or I die" assassins. If they don't, they will try to use whatever CC they have to become tanks. If they can't do either option, then they fade into obscurity with minuscule play rates (and that last option is most of the time forced on them as Riot has to nerf them so hard). We've seen it before, with Diana and Gragas. We've seen it presently, with Sylas and Galio, and to a lesser extent Ekko. And we'll undoubtedly see it again with the upcoming Mordekaiser rework (although that stands the best chance of working, as Mordekaiser has been the most functioning of AP bruisers as an AP bruiser, likely due in part to his hybrid scalings that were added during the juggernaut update, and his ultimate, which gives him access to potential DPS, both of which he is losing with this update).
Battle mages like Swain, Lissandra, and Aurelion Sol get around this issue by being allowed to have burst in their kit, functioning much closer to mages then to fighters (and even then, they will frequently cause large game issues if they are allowed to be too strong)
Could some new bruiser-intended items help with this? Maybe. The issue is that there is no way to guarantee that only your bruisers build it. Give it durability and AP? Every mage in the game wants that. Lessen that AP? Well then why would I buy that when I can just get more AP and be a suicide bomb?
Restrict it to melee only? 
say hi, and as we've seen with tank Ekko and Fizz, you do not want assassins to have durability.
And that's the issue Riot has now with making new items for this subclass. There is no way to protect against abuse, because as a whole AP is too homogeneous. Sure, you can say the same thing about ADCs, or about AD light fighters or bruisers and juggernauts, but those are subclasses. This is homogeny in an entire damage type. There's just no easy way to fix that without changes on par with a "League of Legends 2"
or 
?
