I actually main Varus. He's the very first champion ever bought and I play him a lot in ranked. And you're absolutely right. He'll always be my #1 ADC despite his lack of mobility, but your positioning with him seriously needs to be on-point 100% of the time otherwise you'll get wrecked by anyone that can get close to you. Along with the possibility of rolling AP or AD. - He definitely takes some skill to play since his abilities are skill shots and he's extremely mana hungry, and has no escapes besides flash or blowing his ultimate away to escape. (Assuming you don't miss)
However a fed Varus with a skilled player is possibly one of the scariest things ever. He can solo baron, hard CC assassins and wreck their faces, AOE CC, and fairly decent poke potential (Caitlyn still can out-poke him). He can also absolutely shred tanks fast with his passive. - However since Riot has really been focusing on champs with mobility/gap-closers, Varus has definitely fallen down the popularity ladder besides the handful of other people that main him and know how to deal with those champs.
Sivir Hard-Counters him real bad, any lane match-up with her is gonna SUCK.
But about buffs for him, I'd love to see either: Mana-Regen, Run Speed Increase, Higher Range, a better passive. Besides the VERY MINUSCULE Mana Regen they gave him in 4.20 (Which is very unnoticeable BTW RIOT) I definitely think he needs a PROPER re-working to keep him relevant.