I have two thoughts for you:
First, remember that this is a game that you
want to play, so it's in your interest to respect the "rules". Maybe try just putting it in an envelope, so you are forced to tear it open to access your key- a clear failure on your part to follow the rules. This should be enough ideally. Check out this post by Denying Thumper (IMHO Best Chastity Expert on the Web):
https://denyingthumper.com/2015/02/13/the-best-male-chastity-device/ 
But let's be real- we all feel the fantasy of wanting to be locked up and unable to get free, but the urge to let ourselves out, to be comfortable and to pleasure ourselves, makes it hard to be "good". So we want to be prevented from accessing our spare key, but we also want the safety of a way out in case of an emergency situation. You'll never get this if you know you can get at your spare key. The answer is to make it difficult, unpleasant and/or costly to get to that key. But it always also needs to be possible to get to the key, to minimize your anxiety.
I've read tons of great/crazy ideas online (try searching "self bondage" forums for tons of ideas). My solution is to use a realtors key lock box. I settled on this model:
https://www.amazon.com/gp/product/B0002YP1VC/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1It costs around $20. It's reliable, and if it were used as designed (hanging on a door handle) it would be much easier to break through a door or window to enter a house than it would be to break this lock and access the key inside. I have a bench-mounted vice, and with great effort I can break the case using my vice. I'm pretty sure you could achieve the effect with a large hammer as well. The body is cast metal- squeeze it or beat it enough and it will break.
I know I can always break one of these if necessary, but it will take me a ton of effort, cost me $20+, and most importantly deprive me of my fun chastity key-safe. It keeps me honest, simply because it's too challenging & costly to defeat easily, so I don't. A few times I've messed up setting or recording a combination, and I was able to get out without problem or anxiety. It's a lot easier to enjoy the thrill of locking yourself up when you know you can get out in an emergency, but also know you're not going to use this option unless you really must.
Cheers Buddy
