Mordekaiser can damage kayn during his ult when he is supposed to be untargetable during his ult, however mordekaiser can still damage kayn with at least his passive. I coudln't tell if he could damage kayn with the rest of his abilities, but his passive definitely hits kayn in ult. i'm pretty sure this is a bug because kayn doesn't get damage by similar abilities like fiddle R or nunu R.
I can only assume that you are referring to the Kayn jungle match you played today. If not, I apologize for the assumption.
The Morde on the other team had Liandry's Torment at the end of the game. If the specific moment you are referring to was after they purchased this item, that could be the reason for your death.
Kayn is untargetable during his ultimate, that is true. However, he is not invulnerable. Meaning, if he already has a DoT on him, the damage continues to be applied as normal. Other DoTs, such as Morganna's Ignite would also apply damage while you are untargetable.
If you were low enough health when you initiated your ultimate, you could have died from the burn.
