AP rengar is kind of a hit or miss.. it generally needs a lil more time to come online than AD rengar. i personally like doing it top more for this reason as you tend to get extra gold than u would in the jg.
You could still run flat AD to help you in the jungle, but if you are top id go AD reds with AP quints. You wana max your W first and generally your Q last which sounds weird, but E is going to scale better with an AP build. What you wana do is get a ludens and a lich bane, they both give you some extra MS and ofc.. burst. After that i suggest a deathcap or a zhonyas depending on the game. If you manage to get fed enough, you can generally drop 2 squishies with just 1 empowered W lol.
Its extremely trolly, and i dont recommend doing it all the time.. but every once in awhile its fun to do x)