If the Ursine are a species, how is Volibear Ornn's brother?

Volibear is their god, either he got retconned, or he is like Targon aspects.

Volibear is a demigod now, the Ursine are weird hybrids of humans and bears that aren't really natural I think.

It could be a god-demigod-halfling type situation. Volibear is generally pretty monstrous, so the aspect of him having children either consensually, or more likely, un-consensually is entirely on the table. Orrn, Volibear, and Anivia are all related by blood, but the Ursine may be spawns of Volibear, like the Demi-gods heros of old greek myths.

Given their nature, it's not like they'd really have any other choice then to align themselves with Voli and the Winter's Claw. It's either that, or be killed, and even more generally, if you were a massive bear-man abomination would you actually go out of your way to keep a structured people like Ashe? I doubt it.

Anivia is their sister

Magic is the answer.