Maybe not one passive stat, but you have to commit your main tree and keystone to Inspiration, the ultility tree. Damage dealers lose out on damage, and tanks lose out on durability just so they can swap one summoner every 3 minutes provided you take something like barrier/ignite and you to use at least one on CD, because at the moment it requires you to use a summoner spell 3 times before you can swap (they are working on something so it can't be abused with smite).
Compare swapping a high CD summoner spell for something like new Grasp, a different new Defensive keystone for tanks, the new Thunderlord's decree, or even the insane movespeed boost. You lose out on a lot more than a passive stat just to be able to swap summoner spells, and that's the whole point. You have to sacrifice something just to be able to swap a spell every once in a while, because let's face it, you won't be using those spells on CD, making the time building shards and between swaps that much longer.