Penalties

I had 5 penalties, i played a game, went to a D&D thing, and then played 2 more when i got back, Why does it say i still have 3?