If you are not a tank building heavy defenses actually sets you behind, because you won't hit your peak when you need to. At some point, non-tanks need to be dealing lots of damage to effectively swing team fight, and if you've built nothing but Sunfire and Visage, you are not going to be contributing anything once late game rolls around.
If you are behind, yes, you do need to modify your build. But that doesn't mean building things that don't actually compliment what your champion does. For instance, if an ADC is behind and can't afford a BFS, they can make do by rushing Youmuu's for the AD and lethality to still give them some sting. They can still build damage and make a comeback, because eventually they can ramp up into a full carry if the game drags on