Here is the thing about the state of the game and how Maokai does in it.
Maokai and Nautilus fill very similar rolls. Being this big beefy tank why can utterly lock down a target to their doom. Maokai's longevity comes from his passive in team fights, which is excellent in spell heavy teams. Now with so many marksman (and Fiora) able to take the reigns, the most extreme example being 4 on the opposing team. Just having more auto reliant champions hurts Maokai's kit. While Nautilus doesn't effect him in the slightest.
Playing the same composition you are more open to being countered on a team level with a Maokai pick over a Nautilus pick. When and if there is a time where more spell heavy teams come to play I'm sure Maokai will see much more play.
Also if history repeats itself, when ever Nautilus sees exceptional professional play Riot will nerf him into the ground. Never to be seen until they give him some buffs that put him over the top. Happens every single season.