I didn't consider this

In the older Elder Scrolls games, a boon like this would actually turn out to be something of a burden due to the obtuse leveling up system. In this game though, they’re definitely something you will want to use. You may want to consider not actually assigning it to the skills that you spend most of your time using, instead assigning it to one you use a bit less. If you’re playing a fighter who uses magic occasionally then activating the mage stone will help your magic improve fast enough that it will stay effective throughout the game.