FPS Drop Issues?

So I've been playing league since beta and have never really ran into an issue like this, but recently whenever I'm playing my fps will drop from my 144cap down to usually 40-50~ and gradually gets worse, but whenever I leave the game and rejoin it's back to normal, is this an issue with the game or maybe a GPU problem?