Similarly one of my most important projects reads a lot of api calls returning serialized json. Those calls are expensive so I have, over time, tried many complicated cacheing mechanism.
These days though it's _so_much_simpler_and_cleaner_ to just wrap the call in a decorator that caches the request to sqlite3 and only makes the call if the cache is stale.
I don't worry about parsing the results or doing any of the heavy lifting right away - just cache the json.
Sqlite is so good at querying those blobs and is so fast it's just not worth munging them. Nice.
And using something like datasette to prototype queries for some of the more complicated tree structures is a breeze.
These days though it's _so_much_simpler_and_cleaner_ to just wrap the call in a decorator that caches the request to sqlite3 and only makes the call if the cache is stale.
I don't worry about parsing the results or doing any of the heavy lifting right away - just cache the json.
Sqlite is so good at querying those blobs and is so fast it's just not worth munging them. Nice.
And using something like datasette to prototype queries for some of the more complicated tree structures is a breeze.