JIT and GIL removal are not even my most anticipated Python 3.13 features

    Fun story: I came to Python through data analysis - so I wasn't a "python/programming expert" - and for an unreasonably long time used exclusively csv, parquet (thanks to pandas) and later JSON to persist data from my projects (local projects, of course. Mongo, RDS and Athena were always there). At work this was 100% enough (mostly parquet), but it always felt a little clunky to use JSON for a quick cache, especially when said cache grew larger for longer running processes or longer projects. Opening, parsing then serializing back again becomes slow surprisingly fast.

    Then after a few years I decided to start using SQLite. I knew it existed, but always felt like it was a bit of a hassle to define a schema and "get" and "set" functions in all projects just to have a basic persistent k-v functionality. It was OK and had the extra benefit of allowing me to safely check this cache on a different jupyter notebook process when running longer processes. I still recommend doing this if you're logging / caching results for a somewhat long process that you feel may break and could start again from where you left.

    I can't remember exactly how, if it was from a blog post, a forum reply or just perusing the standard lib documentation, but I then stumbled into shelve. It was great: simple to use (just wrap a `shelf.open` in a `with` block and treat it as a dictionary), faster than using JSON and didn't require installing anything else. Since finding out about this module it's a given that at least some kind of cache or persistence in my projects, even if occasionally combines with SQLite as described above.

    There was just a small catch: shelve behaved differently in different systems due to using dbm as a backend. You can read the documentation for more details, but suffice to say that it could cause incompatibilities between Linux (and different distros), Mac, Windows. This is not a huge letdown because, again, I mostly used it in relatively isolated projects, but it always left me with a feeling that I was being hacky.

    Which finally brings me to 3.13: in the What's New in Python 3.13 it says, under "dbm": Add new dbm.sqlite3 backend, and make it the default dbm backend. (Contributed by Raymond Hettinger and Erlend E. Aasland in gh-100414.)

    So, if I'm reading this right - and I think I am - Python will soon have an efficient and widely compatible persistent key-value store based on SQLite baked into the standard lib. This might be rather trivial for moderate and especially advanced programmers, but if someone's starting out or, frankly, doesn't want to invest time in something more complex, this should completely scratch that "persistent dict" itch that most of us had at some point and many still do.

    And thanks Raymond and Erlend, for the contribution! Super appreciated.

Comments

Post a Comment

Popular posts from this blog

An interview with Steve Wozniak by Jessica Livingston cured my AI anxiety

Note: there's no hidden knowledge contained in this post, it's just a reflection on how reading someone's reaction to an analogous situation changed my outlook Like everyone who works in an information job, I have been worried these last few months about how AI would impact my place in the labor force in the coming decade. Even though I didn't get to the depressed extremes we've been seeing on the Internet, I could not shake off the feeling that what I could offer to a company - no, to the whole economy, which just means "other people" - would get a lot less valuable. Come mid January, I wast taking a week off and had taken to read a few books since I read so little non-technical content on a day to day basis. One of these books was "Founders at work" by author and YC co-founder Jessica Livingston. The book is a series of interviews conducted by Jessica with the founders of some of the most important tech companies - sometimes specific products -...
Read more

What if regular exercise is the best cognitive exercise?

File this under highly speculative. I’m sure there is better research on this topic than this writing This past week, for no obvious reason, I considered a few thoughts at the same time: Exercise is supposedly one of the best anti-aging “drugs”, and also highly effective (relatively speaking) against cognitive decline There’s this vague idea that after you become “professionally inactive”, you accelerate your cognitive decline To avoid cognitive decline, one must keep “exercising the brain”, so the incentive for seniors to have hobbies, do crosswords, sudoku, etc Now, from this point it seems that I’d go to a very commonplace conclusion about how exercising is probably better than sudoku or something, and to be frank that’s exactly where I’m going. But another two things crossed my mind at the same time: Recent advances in AI, and less recent advances in computing in general showed that a lot of the things we consider “cognitively hard” - from symbolic manipulation to arithmetic, “...
Read more