Would it be OP? Probably not. Would it be annoying? Absolutely. You're playing against a time bomb in the truest sense. If you went against them, they would just be absolutely the strongest in pretty much every stat come late game. I would build to have more autoattack damage than anyone while being tankier than anyone except a pure tank. I could have the strongest abilities by far. It's already annoying playing against someone like Veigar or Nasus or Vayne or Poppy if you know you'll be outscaled and will lose options as the game goes on, but he'd be even worse because his scaling is just sooooo out of line.
Of course, it could be annoying for the guy with the passive too, because he's going to have to be balanced around being weak up until the point where he gets those items so that his guaranteed win late game isn't too strong. Being weak for that long is never very fun.
Overall, I'd say it wouldn't be OP, but it also isn't worth putting in the game. It would be more irritating than fun in most situations. The only gameplay it offers is "beat me before X minutes or I'll win off of pure stats", and that's just not very interesting.
Also, I don't see much of a point of giving them extra item slots at 6, 12 and 18. They'll be fully leveled before they get to use the extra slots. Might as well just give them all the item slots at once.