Bought Items and Walked Off Pad and System Still Thinks I'm Afk

I bought my items, and walked off pad to lane, and come back right at the three minute mark just to watch my team surrender, because I was "afk" (I know I technically was). I'm pretty sure my account got dinged for this, which is pretty frustrating since I literally bought items and walked off the pad to my lane. Has anyone else had this happen to them?