They would likely take away the hammer and try to destroy it and after that they'd likely lock up Jayce for some time and interrogate him about it(to make sure that jayce himself isn't a mage).
If Jayce can convince them that the hammer is just a magical artifact,then they'd likely just punish him for bringing a magical artifact into demacia(though they likely wouldn't kill him),if they think that Jayce himself is magical,then they'd likely banish him or try to "cure" him from his magical abilities.
They don't just execute mages in demacia,the guy who was executed by Garen in the short story about Nocturne was executed because he murdered other demacians(though he did it under Nocturnes influence),not because they believed him to be magical.