Brand's W is the only ability that is even worth it damage wise, aside from his ult. The others are literally for one thing and one thing only, to stack his passive. Jarvan's E is AOE as well and considering it's damage, and Jarvan's ult has 450 base damage. The Rylai's nerf kinda hit AP Jarvan indirectly(since it's not really a common thing to do), but Rylais was never necessary and another AP item can take it's place(
would be a good contender). Jarvan's passive also has literally no ties to AP or AD, though it's physical damage so if they build MR you can still deal some decent damage with an auto every so often. I'm not gonna say AP Jarvan is the best thing that ever was, or even that it's better than AD Jarvan(although in some situations it can be considering you don't have to initiate to deal damage meaning you might not die as often), but it's not terrible. Also might I add, you claimed Jarvan had 4 other damaging abilities besides his E. What are those? Cuz I only see 2 other than his passive(which has no ties to his AD), being his ult and Q.
(Though just to compare, I gave Jarvan a RoA+Ludens and Youmuu's and Ravenous Hydra to compare damage from a full combo, and the AP were just a little less in damage dealt: 1,220 with RoA+Ludens, 1,304 with Youmuu's/Hydra. If I added more AP Luden's Active in the combo would be stronger, but I don't feel like comparing full builds because it's too much math)