Ranked Matchmaking

I am a silver 3 player who has been stuck in silver for a long time. I started looking up the ranks of people who I play with and against and have noticed that the majority of them are gold rank.

For example, the last ranked game I played I had once silver 3 on my team, and three gold 4 or 5 players. The opposing team had four gold 4 or 5 players and one silver 4.

Clearly if I am playing gold players I deserve the rank of gold, but I do not understand how I will ever get out of silver if none of the people I play against are actually in my rank.

Can someone please explain this to me because it seems punishing and incredibly unfair.