hmm... well considering yourself asexual is considered abnormal and describing the idea that you're not sexually active as such will make anyone around you uncomfortable. regardless of what asexual means using it to describe a lack of sexual interest is assumed to mean you create offspring by budding and a lot of people will think you're ignorant as well as just denying pure instinctual desire. i explained that to another of my old friends back in high school and they hated me for it, but there is truth to it. it's better to just say im not interested in sex. people will assume you're lying, but that's better than being assumed ignorant of human sexual reproduction since the definition "not related to sexual feelings" is not commonly known.
people assume conservatives are infringing upon freedoms just the same as liberals are assumed to be anarchists. being scared of one or the other is silly since most longstanding governments are based on a combination of both. in other words conservative freedoms.
talking about drugs in the first place is a faux pas. if you dont consume them just dont consume them. when you tell people who consume drugs you dont like drugs they assume that you wont like them and it makes conversation awkward. make sure to approach it casually and tactfully. of course if you can try to avoid these types of situations as much as possible. it sucks being the guy that has to report on your friends for their involvement in meth production, but it's your duty as a proper citizen to do so if the laws prevent it even if they're your friends.
there's nothing wrong with not being interested in sex and there's nothing wrong with not being interested in drugs or alcohol. everyone has their own vices and finding someone who is the exact perfect person for you to talk to is asking too much.
refusing to be around people who drink, smoke, consume drugs, are sexually active, have a different form of religion, have a different view on proper government structure, are less educated, are more educated, have identity issues, have medical issues, social issues, etc. will only lead to one thing : being alone.
i for example am what is known as "straight edge" in that i dont drink alcohol, consume drugs, or smoke. i have conservative views and i am agnostic. i am hyper sexually active, but i prefer monogamous relationships that are meaningful and long term. if i were to try and find people who share my exact opinions on the world i'd be alone every day. i just dont share my personal opinions with anyone who doesnt need to know. if a situation makes me uncomfortable i avoid it.
my point is the problem you're having is more or less you accepting others and not so much others accepting you. or maybe it's both. either way being picky about who you talk to while sometimes is a good thing can also be a bad thing as you'll be treated as a snob. it all depends on how old you are where you live your body type and your gender. beautiful people, wealthy people, skilled people, and those that put effort into their living routines are usually admired more easily than those on the opposite end of the spectrum. that's part of natural selection. beauty is given at birth and requires effort to maintain. wealth can be given at birth or earned throughout life, but requires effort and knowledge and/or skill to maintain. skill can be given at birth, but can also be developed through rigorous effort and practice in some instances. effort naturally grants admiration and is self explanatory.
intelligence will only grant admiration from those seeking to be as intelligent as you are. more often than not intelligent people suffer for their views and gifts as understanding is not the same thing as knowing.
so choose your path. you have to live a life of worth to be treated as a person of value.