Odds are you're playing ranked players who use un-ranked to try out new stuff. My understanding of MM (which is really old, but as far as I can tell it never changed) is that you actually have two (or more) ELOs:
- Your Ranked 5's ELO
- Your un-ranked ELO
- (and 4, and 5) all other ELOs.
In short, playing ranked (and getting to Master or whatever) increases your Ranked ELO, but not your normal ELO, and vice versa. I believe ELO also drops over time if you don't play a game mode (but I'm not sure about that). So at the end of the season, when ranked goes away, everyone's normal ELO gets used. If you got to master by playing a lot of ranked games, but maybe only touched normals to learn new champions, odds are there's going to be a vast ELO discrepency. This happens every pre-season. There are really simple fixes to this (when ranked ends, your unranked ELO becomes the greater of your Ranked and Unranked ELO), but tbh I don't think it's one of those things that Riot cares a lot about.