AP to building damage not working

Well, I just got out of a game as Veigar, I decided to dick around because we were screwed and gonna lose, so with my 906 AP, I bought a ruin elixir and ran at their exposed top inhib, only to be doing 80 damage with each hit, and I've done this before, run at an exposed inhib with a mage and have no creeps around, and deal ~300 damage per hit with all my AP making my autos do more, did something change that wasn't noted in the patch notes? Do I HAVE to have creeps with me to deal the 40% AP to building damage thingy? Does it only work on towers? Does the elixir of ruin turn that off?