Think of it this way: If someone says that it really offends them, you should probably stop.
Example: I've lost count on how many times I've been called autistic on my climb to 30. I do have a young family member who is autistic and also gets picked on a bit because of that, so when someone uses autism as an insult, it's a very personal insult to me. I've told people to stop it multiple times, and they continue, except they're even worse because they know it incited a response from me. That's when it becomes verbal abuse and harassment.
There's joking, and then there's being a jerk. I was once that kind of person, and after growing up and experiencing life a while, I learned that if I didn't have anything nice to say, I probably shouldn't say it. It's that old adage that kids were told in preschool-kindergarten. I personally didn't learn that words do influence others' decisions and was a complete jackass and a douchebag towards everyone until I went to college and learned that.