It's certainly a touchy subject. I think there's a lot of debate that could go into this discussion. I personally don't think it's unfair because regardless of the outcome of the game, you've put undue stress onto your team.
Not only is playing 4v5 not an ideal experience, the fact that the game dragged onto 55 minutes (often times, games where there is a good portion of it being 4v5 get quite lengthy because a comeback has to be made) means that I'm probably taxed by the end of it. What if the game could've been finished earlier? You being gone for 5 minutes may not have been the only reason, but it surely played a factor.
'Well why did would I bother to come back and play the game?'
Because you're not a toxic player and you truly have good intentions for your team. You've already spoiled the game for the same reason I mentioned above. Your responsibility as a teammate (in my perspective) is to at least help your team not get an auto-loss.
'Well, can I at least avoid LP loss if we win?'
Do you really want to encourage any form of rage quitting? I understand that you had some real life family stuff going on but if we believe you, we have to believe everyone. Allowing people who rage quit to mitigate their LP loss if they can secure a win is going to encourage more people to rage quit and come back to the game.
'But isn't it good to encourage players to come back and help their team?'
From an LP perspective sure. But I don't want to deal with people leaving the game (even if only temporarily) more often. That's not quality gameplay. Besides, your odds of winning when a teammate is gone for a while and come back to play aren't as high at that point anyways.