With the exception of wards, invisible "traps" should be damageable by skillshots

Imagine how much less annoying playing against if, when they drop their traps RIGHT IN FRONT OF YOU, you could actually use your skill shots to take them out/damage them before they go off.

This would hurt ultra aggressive playstyles since they have to actually put their traps in places out of your vision or else you could destroy them. I get why wards aren't hit by skillshots, but what makes an invisible unit so special that it can only be auto attacked?