Yes, but the Rust code is the kind of thing you'd write if you wanted to build that kind of thing yourself. I do think normal Rust can get too typey, but this is probably not the best example of that.
What you’re describing sounds exactly like Elixir (any Erlang family language really, Elixir is just most ergonomic IMO). Truly seamless cooperative scheduling with transparent distribution between any number of nodes.
Shameless plug, but there’s also Oban, a widely used database backed background job system.
The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo.
Starting with the project's tagline, "Robust job processing in Elixir", let's see what else:
- The same job states, including the British spelling for `cancelled`
- Snoozing and cancelling jobs inline
- The prioritization system
- Tracking where jobs were attempted in an attempted_by column
- Storing a list of errors inline on the job
- The same check constraints and the same compound indexes
- Almost the entire table schema, really
- Unique jobs with the exact same option names
- Table-backed leadership election
Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX.
Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different implementations before ultimately apparently landing in a similar place. I'm not quite sure what to say on the spelling of "cancelled" though, I've always written it that way and can't help but read "canceled" like "concealed" in my head :)
As I think I mentioned when we first chatted years ago this has been a hobby interest of mine for many years so when a new database queue library pops up I tend to go see how it works. We've been in a bit of a mad dash trying to get this ready for release and didn't even think about crediting the projects that inspired us, but I'll sync with Brandur and make sure we can figure out the right way to do that.
I really appreciate you raising your concerns here and would love to talk further if you'd like. I just sent you an email to reconnect.
As if Oban invented anything, like queues on RDMS is not a new concept at all. Oban is 4 years old, do you know how many queues baked by DB were created in the last 10years?
I don't see Sidekiq credited on the main page of Oban.
I’d argue strongly that Oban did invent things, including parts of the underlying structure used in River, and the authors agree that it was a heavy influence.
While there is no overlap in technology or structure with Sidekiq, the original Oban announcement on the ElixirForum mentions it along with all of the direct influences:
Not sure if you meant to compare to Sidekiq (which uses Redis). But delayed-job and que are both projects that stretch back much more than 4 years in the ruby ecosystem that leverage relational databases as well.
As far as I've seen, Americans are just plain inconsistent on this spelling.
Cancelled has nice pairing with cancellation, canceled can be typed nicely without any repeated finger use on qwerty, both clearly mean the same thing and aren't confused with something else... I say let the battles begin, and may the best speling win.
To be fair, "referer" is just a misspelling, I don't think it was ever accepted as a correct spelling by a large number of people. I'm sure you know the backstory though.
> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level.
Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poolers like pgbouncer, which prevent NOTIFY ever triggering, and the extra db load of trigger handling wasn't worth it.
That makes a lot of sense, I've had the thought a few times that the NOTIFY overhead could get overwhelming in a high-throughput queue but haven't yet had an opportunity to verify this or experiment with a mechanism for reducing this overhead.
The very first paying Pro customer, as a matter of fact =)
You said back then that you planned on pursuing a Go client; now, four years later, here we are. River looks excellent, and the blog post does a fantastic job explaining all the benefits of job queues in Postgres.
To do anything safe and interesting you’ll need transactions. Using SKIP LOCKED won’t be your bottleneck, your application will. Job queues are about side effects and the rest of your application needs to keep up.
Time, know-how, and long term dedication are the key ingredients from my perspective. Maybe he's working 10-20 hours now, but there's support and development every day for the past 11 years.
> I don't imagine any equivalent product (background job processing) in the other languages is raking in that much, if making any revenue in the first place.
I guarantee, there are background job processors in other languages making revenue.
> I guarantee, there are background job processors in other languages making revenue.
I had no idea until someone just hinted me that this was self-referential... LOL, apologies for the now-embarrassing other comment I made LOL (I only know you as "sorentwo", whoops!)
Jobs may crash due to VM issues or OOM problems. The more common cause of "orphans" is when the VM restarts and jobs can't finish during the shutdown period.
