Thank you for the kind words! I wanted to get something out even if it doesn't have the scalability stuff built. But I have been thinking and architecting that behind the scenes, and now I "only" need to turn it into code.
I think the queue itself will end up using a number of technologies: SQLite for some data (ie, organizing messages by date and queue), RocksDB for other things (fast lookup for messages), DuckDB (for message statistics and metadata), and other data structures.
I find that some of the most performant software often uses a mix of different data structures and algorithms by introspecting the nature of the data it has. And if I can make one node really sing, then I can have confidence that distributing that will yield results.
I think SQLite is a great start, but I really do want the software to be able to utilize the full resources the underlying hardware.
I think the queue itself will end up using a number of technologies: SQLite for some data (ie, organizing messages by date and queue), RocksDB for other things (fast lookup for messages), DuckDB (for message statistics and metadata), and other data structures.
I find that some of the most performant software often uses a mix of different data structures and algorithms by introspecting the nature of the data it has. And if I can make one node really sing, then I can have confidence that distributing that will yield results.
I think SQLite is a great start, but I really do want the software to be able to utilize the full resources the underlying hardware.