I think we might see AP junglers eventually, but right now the game really isn't able to support AP junglers because Magus really isn't strong.
We can basically look at there being four kinds of junglers, one for each item: AD/Bruiser junglers that build Warrior, Tank/Bruiser junglers that build Cinderhulk, AP/Support junglers that build Magus, and On-hit champs that run Devourer. We need to look at how each kind of jungler will fare in the jungle between their starting clears, mid-game clears, and their impact later on.
Warrior: junglers that run the Warrior enchant will normally clear pretty well on the first clear, especially considering they are going to have the AD to clear early. Warrior will also greatly help champions clear the jungle throughout the early-mid game through its stats, and because the stats will apply on both their auto attacks and, in most cases, their abilities. Because these junglers can auto attack and will get faster clears and either get tankier (bruisers) or get sustain through lifesteal (AD's like assassins), then Warrior is going to allow most champions to do a lot better in the jungle than other champions.
Cinderhulk: more tankiness and faster clears. More tankiness means you can take more damage, and the clear speed off of immolate will make a lot of champions way faster, in some cases clear so much faster that any abilities that these champs use will actually have their mana costs restored simply because of Machete regen.
Devourer: you are getting a lot more clear speed just from auto attacks. This means that you don't have to spend much mana, if any, to clear camps, at least once you start getting devourer sufficiently stacked. Most users of this item also build tanky after they build this, and rely on its damage for the rest of the game as their only damage source or buy physical damage so they can deal tons of damage off of two damage types. This means that these champions are going to get the same kind of value out of devourer as warrior junglers get out of warrior.
Magus: AP junglers rely a lot on their abilities to clear and not on their auto attacks. This means that these junglers need to spend mana to try and clear the jungle, and there aren't too many junglers that can efficiently clear the jungle while still being mana efficient. This also doesn't include the fact that most don't have innate sustain, and will likely have too low health to really clear. Magus will increase clear speeds, but it doesn't fix any of the other problems that AP junglers have with jungling, which is their severe lack of sustain.
It's not a problem with designing an AP jungler. It's a matter of creating an environment where an AP jungler (that isn't Nidalee or Fiddlesticks or any of the other half-dozen AP jungler exceptions that I have no idea actually work in the jungle) can actually work while still being able to positively contribute to a team.