Pantheon shouldn't gain passive stacks from attacking minions

To me, this just makes it basically impossible to farm or trade or ward or do anything against this guy in lane because he can jump on you whenever the hell he wants and melt your hp bar while also taking zero damage himself because of his E. I don't have an issue with the rest of his kit. I think it's fine. But how easy it is for him to stack up his passive really bothers me as there's really little you can do about it except for wait until the late game which rarely even happens anymore because he got so fed that you have to ff@15.