He got hit through the wall because of the lollipop hitbox thing that means when the hook is at the end (and hasn't hit a wall in the case of naut) the hitbox is a circle for an instant, that circle intersected Kayn's hitbox, and so the hook got him through the wall
So the hitbox travels like this:
=
=====
=========
========(--)
This gives some margin for error when you throw long range hooks, but also means you can hook a champ behind a minion as Thresh if you throw your hook right beside the minion, and hook a champion over a thin wall as Nautilus if your hook doesn't quite touch the wall they're behind
Honestly this lollipop hitbox thing should go, because it's an issue on all hook champs

And also some other skillshot champs