And vision is more limited than it has ever been in League. Why should I waste a ward to monitor a location that contains a plant, when I could just get rid of it and place a ward where I need it?
While jungling I already have my mind on the mini-map, checking lanes to see which is favorable for a gank, which might be ganked that I could in turn counter-gank, jungle camps, dragon, and Baron. On top of all this is the presence of the other jungler, who I am "laning" against, who may or may not invade, who may or may not counter-gank.
Throwing plants into the mix is just another objective to keep track of, which are trivial compared to other objectives, but could nonetheless shift a fight in someone's favor. It benefits me more to get rid of them instead of watching them and potentially using them when they actually matter.