The main barrier to new maps, or even making the current alternate maps more popular, is that Riot seemingly refuses to do anything to support them. I'm less familiar with the other game modes, but the Dominion community repeatedly put forth suggestions whenever Riot asked for feedback, but Riot never did anything with those ideas. A large factor in why the alternate game modes are less popular is simply stagnation from neglect. Unless that issue is addressed, any new modes/maps will eventually collapse as well.