Destroying wards shouldn't use auto attacks (but still take time)
it inherently favours ranged and high auto attack speed champions and adds to the problem with completely negating trap champions using red trinket. It also favours ranged supports over tank supports.
If all wards and traps had a dotted circle around them indicating a range where you could channel for x seconds (while still being able to move and cast spells) red trinket wouldn't have needed to disable traps in the first place (since melee champs would activate them anyway, defeating the purpose of using red to sweep an area).
It would also allow a bunch of new restrictions that make destroying traps and wards not an instant loss. It could slow the champ down while channeling, or destroying traps in combat would no longer be possible. While I'm not sure what would be the best choice it would still be healthy for the game as it adds a bunch of levers to balance with.