I think part of the problem with Nautilus is that his initial foundations have aged poorly as feature creep developed in League. When Naut released, he set a lot of records: he had the most CC out of any champion, plus he had manaless hard CC on his innate, which was unheard of. Nowadays, though, we're in an environment where a great deal of tanks have multiple AoE knockups, plus additional crowd control on the side, and that's considered standard on top of damage and/or utility. Ornn, for example, has CC comparable to Naut's, perhaps even more, yet has a fair amount of damage and some utility on the side. Naut's identity needs to be more than just "has the most CC in the game" if he wants to have a stable spot in the champion roster.
Part of the issue with this, however, is also that tanks as a class have gone a little overboard. Short of giving Naut stuff he really doesn't need to have, like damage, mobility or utility, he's not going to compete with the rest, but only because the rest has become overloaded. Tanks as an entire subclass need to have their AoE and damage toned down, and doing so could help Nautilus out by making him stand out better.
In the meantime, if there were two areas I'd seek to improve him, it would be his W and E: the bonus magic damage on W basically just affects his jungle clear without doing much anywhere else, so it might help to instead tie it to his innate, which provides damage but also hard CC (which doesn't really change his clear). Having his innate tick down faster one way or another while he's shielded (it could even be on-hit cooldown reduction) could allow that steroid to have some use outside of the jungle.
Regarding his E, I think it might be good to reward Naut for landing multiple waves against the same target: for example, if getting hit by all three waves stunned the target, he'd be able to punish opponents running away from him much harder. Additionally, I'd remove the damage reduction on subsequent waves, even if it would mean toning down the damage per wave, as I believe he deserves the full rewards for landing multiple waves against opponents. Having his Riptide waves originate from him, rather than a static location, could also allow him to maneuver the ability much better, which could allow him to express more skill both in his jungle clears and his ganks.