Maokai has fell out of popularity due to repeated nerfs, most of which are small/indirect but added up to be huge.
- First they reduced his healing on his passive
- Next they nerfed Rightous Glory and Rod of Ages, both of which builds out of catalyst which is an excellent early item for him
- Then they nerf Frozen Heart which means that all of his first built items have been nerfed and he can no longer sustain as well as he could before
- Lastly they nerf his ult meaning that he can't use it for wave clear and it's not up in lane all the time like it was before.
He's not unviable yet. He still does his job of locking down targets and tanking a lot of damage. However he gets less stats for a build that cost more and now your ult isn't up anywhere near as much as before level 16+. It was multiple nerfs that killed his popularity and not just one giant nerf.