Nocturne can carry harder, Volibear can snowball harder.
Other than that,
Nocturne struggles in skirmishes, he has alot of damage in his kit, but he has squishy base stats. Nocturne is stronger offensively in 5v5 teamfights through his utility in his ultimate. Either as an initiate or clean-up. He also scale well on most items, so he has a strong lategame.
Volibear is strong in skirmishes, and can snowball waaaay easier than Nocturne. In 5v5's Voli struggles at initiating, and excell more as a peeler and tankbuster through his bite. Volibear shines midgame, especially when ahead. However his damage falls off unless you get on hit effects like wits, botrk, triforce or bc (very underestimated item on him)
They depend on your adc's really. If they have Cait, Ez, Lucian or hard to catch slippery fishes, go Nocturne and kill them.
If you have Kog'Maw, Graves, Vayne or Sivir, go Volibear and peel.
They're both strong duelists, good gankers and counter gankers, good invaders and have decent clear time while being very strong through different stages in the game.
Btw, I prefer taking Voli toplane, because... Free win.