JIT and GIL removal are not even my most anticipated Python 3.13 features

    Fun story: I came to Python through data analysis - so I wasn't a "python/programming expert" - and for an unreasonably long time used exclusively csv, parquet (thanks to pandas) and later JSON to persist data from my projects (local projects, of course. Mongo, RDS and Athena were always there). At work this was 100% enough (mostly parquet), but it always felt a little clunky to use JSON for a quick cache, especially when said cache grew larger for longer running processes or longer projects. Opening, parsing then serializing back again becomes slow surprisingly fast.

    Then after a few years I decided to start using SQLite. I knew it existed, but always felt like it was a bit of a hassle to define a schema and "get" and "set" functions in all projects just to have a basic persistent k-v functionality. It was OK and had the extra benefit of allowing me to safely check this cache on a different jupyter notebook process when running longer processes. I still recommend doing this if you're logging / caching results for a somewhat long process that you feel may break and could start again from where you left.

    I can't remember exactly how, if it was from a blog post, a forum reply or just perusing the standard lib documentation, but I then stumbled into shelve. It was great: simple to use (just wrap a `shelf.open` in a `with` block and treat it as a dictionary), faster than using JSON and didn't require installing anything else. Since finding out about this module it's a given that at least some kind of cache or persistence in my projects, even if occasionally combines with SQLite as described above.

    There was just a small catch: shelve behaved differently in different systems due to using dbm as a backend. You can read the documentation for more details, but suffice to say that it could cause incompatibilities between Linux (and different distros), Mac, Windows. This is not a huge letdown because, again, I mostly used it in relatively isolated projects, but it always left me with a feeling that I was being hacky.

    Which finally brings me to 3.13: in the What's New in Python 3.13 it says, under "dbm": Add new dbm.sqlite3 backend, and make it the default dbm backend. (Contributed by Raymond Hettinger and Erlend E. Aasland in gh-100414.)

    So, if I'm reading this right - and I think I am - Python will soon have an efficient and widely compatible persistent key-value store based on SQLite baked into the standard lib. This might be rather trivial for moderate and especially advanced programmers, but if someone's starting out or, frankly, doesn't want to invest time in something more complex, this should completely scratch that "persistent dict" itch that most of us had at some point and many still do.

    And thanks Raymond and Erlend, for the contribution! Super appreciated.


Popular posts from this blog

An interview with Steve Wozniak by Jessica Livingston cured my AI anxiety

What if regular exercise is the best cognitive exercise?