anybody know if its possible to gain champion mastery points in a 5v5 human custom game ?

yes.

I believe it has to be a match made game mode, though I could be wrong.

I don't think so, I mean you cant even get ip in that kind of game.