As a Viktor player i dislike having a Nasus on the other team. A farmed Nasus can 2 shot a Viktor (like any other carry) so i'm very surprised you had an issue with Viktor late game. Him having more AP doesn't stop him from you being able to kill him. I have to assume you didn't build a tri force...for whatever reason.
Viktor Is deceptively immobile. His upgraded q gives him a really shitty shield and speed boost that only matters when he is against slow mages. Nasus, with his W, and speed from a tri force should have no issue catching and killing him. Viktor has his little slow field, but it has such a long CD he will only get it out once, and maybe twice in a team fight. If he throws it on you (the tank), then he is pretty fucked, and if he throws it on your carries, then you...as Nasus just walk up to him and kill him.
All that fails, you need to avoid fighting in the jungle. His skill shots become infinitely easier to hit in small spaces, and with his slow field pulling people to the center he can easily kill any non-tanks on your team all at once with his E and R. Key all your flanks warded and try to get a fight in the lane where he has to choose between peeling for himself or attacking your back line.
Also, Veigar is probably the best counter to Viktor.