I always wondered why do aa stacks don't count back down from max stacks such as graves and jax?

I mean when you build the stack shouldn't it drop the way it climbed back up? for example if you want to run into a team fight but want to build your stacks up first before you run into it, it can be pretty useless because sometimes you just don't have the time to to aa the enemy 10x for example as graves to get the full benefits. Stacks are stacks if you gain them you should lose them in same manner as well. It does make sense if you think about it.