Jax vs Nautilus

They serve completely different roles. It is like comparing Janna and Kha'zix

IF you need a tank, Nautilus is one of the best things you can ask for, if you need a carry, Jax is one of, if not the best out there. Nautilus can't really kill jax, he's not supposed to kill people by himself in the first place but jax can't kill him either unless ahead, even later on when he actually reaches his full potential since Nautilus is just too tanky to be brought down by one person who isn't Vayne, Fiora, Twitch or Azir (the best hyper-carryes of the game) and even then, they do take a bit of time to bring him down by themselves thanks to his shield and his huge amount of cc. So tl;dr: if you need a tanky dude Nautilus with his cc is far better, if you need damage Jax is the way to go, but they are pretty power-neautral, neither of them can actually against the other through the whole game.