High downlink buffering (according to netalyzr)

Netalyzr says I have an downlink buffer of 4500ms. I seem to get inconsistent lag and disconnects when I'm playing. Sometimes the connection is perfect and sometimes it's unplayable. I share an internet connection that has unlimited monthly bandwidth with 4 other people and a family. The first thing you might assume is that it has to do with too many people using the connection all at once. But sometimes it happens when nobody is home. Does anybody have a solution to this?