> I must not segfault. Uncertainty is the mind-killer. Exceptions are the little-death that brings total obliteration. I will fully express my cases. Execution will pass over me and through me. And when it has gone past, I will unwind the stack along its path. Where the cases are handled there will be nothing. Only I will remain.
They did an episode of Signals and Threads (Jane Street's technical podcast) about their build systems back in 2020¥. Highly recommended, as are the other episodes, especially The Future of Programming with Richard Eisenberg° and What is an Operating System? with Anil Madhavapeddy§.
Compose key shenanigans can make such typesetting significantly easier. That's built-in on Linux/X, but there are also easy solutions for Windows systems.
I have to respect Jane Street for proving that HM-typed functional programming can be production-ready and extremely fast when given enough love.
I still occasionally hear things about how the more academic-styled functional languages can't work in production, but Ocaml shows that it absolutely can work, even with high performance requirements.
I was fortunate enough to work at Jet.com before Walmart completely destroyed it, which was an F# shop. I really liked it, and I never really felt "limited" by it.
The stuff I was working on didn't have nearly the same requirements as high-frequency trading like Jane Street does though. I never did any super low-latency stuff with F#, so it's tough for me to say how well it fair with that kind of environment.
Thank you for mentioning this, because—not having followed the link—I thought someone was using Mercury for a backend. And... I wasn't sure I was prepared to follow that link.
Yeah, I actually interviewed for a Haskell position there in 2023. They didn't hire me, but it was kind of fun to do technical questions in Haskell for an interview instead of the boring JavaScript or Python that I usually use.
Certainly. There's also a good amount of F# in production. It's a fantastic option for building high quality programs with broad interop with existing systems.
NoRedInk[0] also uses some Haskell in their backend, as well as Elm for most of their frontend. They've also worked some with Roc according to a blog post from a few years back; not sure if using it.
Sadly I don't think it's used any more - it was migrated away from. I worked in integrity there and quite enjoyed whenever I had to make some changes to the Haskell rules :)
If anything the type system improves the run time speed, because the static analysis enables better code generation.
But I think what OP meant was more about the "functional programming" side of things than the "HM-typed" side of things. Naively, anyway, you might think that "the FP-style" of avoiding mutation and preferring recursion would require lots of garbage and high-latency garbage collection, copying, function call overhead...of course, that's not the whole story, but having Jane Street to point to as a crushing counter-example is nice.
Eh, this reads like damning with faint praise to me. Lots of robust and fast real world systems are in FP languages. Ocaml is estabilished and mature, far from "is it production ready?" category.
Not really meant to be "damning", I'm a pretty big functional programming advocate (I've spoken at the Lambda Days conference four times and the Clojure Conj once).
It feels like the trend right now is to bolt on one or two "functional libraries" into your "normal" language and pretend that that's the same as writing Haskell or Ocaml. People have actually expressed such sentiments to me because Java has the optional type and a "map" function for the Streams API. When I suggest writing something in a functional language, the response is always "it's too hard" or "we won't be able to hire for that", as if engineers are somehow unable to learn new things.
I dunno if anyone really doubted HM, functional programming could work. I've written a bit of OCaml and apart from the bad syntax (code very easily becomes a word soup) it's not especially different to other programming languages, just more convenient sometimes.
I think it's purity that actually is a big difference, but OCaml isn't pure.
My humble opinion is that the every-function-must-be-pure aspect of Haskell is the reason why Haskell won't receive mainstream success, not that that's what they're after.
It complicates things that are simple to express in an imperative language
Whenever I see posts from Janestreet blog I laugh inside thinking how working on the financial industry at Jane is probably more fun and rewarding than some 90% of the jobs in the so-called tech industry.
Jane Street and a handful of few other elite shops make up a tiny percentage of the finance industry’s tech workforce though.
There are a ton more people working in tech in finance that don’t quite have it as fun (or lucrative) as Jane Street let alone your average tech company.
OP should've been more specific. HFT firms, not any other finance companies, probably have a lot more exciting work due to the nature of reducing latency using all sorts of novel techniques.
I wonder if they disable all the fancy exploit mitigation protection in linux kernel just for a tiny performance hit
Mother capitalism deems that our brightest young minds best serve humanity in two tasks. Keeping the public passively scrolling, and moving money at speed to make wealthy people more wealthy.
Actually, not. These market makers are often prop shops. That means they use their own fund (prop = proprietary) to do the trading. They can do that because they don't need much capital to run.
So the story here is that over the last twenty years they stole the lunch from the traditional market makers like eg banks.
Of course, they got rich in the process. But they started from relatively modest means, compared to the companies they took on.
Michael Lewis's 'Flash Boys' is an hilarious account of this process. Well, it's involuntarily hilarious, because to tell his story, Lewis needs to cast Goldman Sachs (!) and other big banks as the victim. See the rebuttal 'Flash Boys: Not so fast' by Peter Kovac for more insight.
Perhaps some start this way. But in terms of the general trend of talented engineers and mathematicians being sucked into this quant vortex, it is a matter of making wealthy people wealthier.
Automation in trading makes all investors wealthier via lower fees. Trading costs basically nothing nowadays, and that is because far fewer people are employed to do it.
Obviously, the people who own the automation will want a cut of the rewards, like any other business.
Obviously NASDAQ and electronic trading systems are a good innovation. But firms basically doing arbitrage or exploiting uneven network latency are not that economically productive.
Well lowering market spreads is all about increasing the returns for capital, and incenctivising overfinancialisation. It's hardly curing cancer is it?
At worst it's actively harmful if you believe that the current state of turbo-financialised capitalism has its drawbacks.
> Network latency
Not really sure what you're talking about but surely spending billions of dollars to bring rtt latencies to 50 micros or whatever is not really a great use of money and top engineering talent. Again, it's playing an arbitrage game but not really delivering any value.
lol I said "closeted dictator" for the record. But alrighty why don't we start over and see if we can both argue in good faith. I can certainly be a dick on the internet sometimes.
I honestly can't tell what your concrete points are. I come from the position that economies are naturally occurring phenomena which cannot be centrally planned or controlled. If people can find ways to profit off market inefficiencies, they should! The HFT/Quant firms make their arbitrage money (value for them) and all market participants in return see: (non-exhaustive list)
If your bar is that "all smart people should be working on curing cancer or andrepd-approved endevours" then almost nobody in the economy is providing value. Is my lowly SecEng job at $MEGACORP good enough? What about my buddy who writes firmware for toothbruhes? Are professional starcraft players wasting their talents?
> EDIT: The funny part is even the exchanges and hft firms agree with me see PLP/speed bumps on exchanges like Eurex lol
This debate has been going on for ages, and it's silly to pretend that it's been settled and everyone agrees with you.
> I come from the position that economies are naturally occurring phenomena which cannot be centrally planned or controlled.
This is a challenge to untangle. It sounds like you're saying that there is no point trying to regulate, legislate or control what happens in the economy at all. But that sounds bonkers to me.
For starters, there are (and should definitely remain) absolute limits to business activities. We've moved on from Victorian-era child and slave labour for good reasons, even though such a situation was "naturally occurring" at the time. Moreover economic activity is dictated by cultural mores - if your service is morally reprehensible in some way then you won't get much business whatever your matgins are. Economies are inherently subject to the laws and customs of the agents.
Secondly, some regulation is pretty clearly beneficial. For example, there's a recurrent tendency for market power to concentrate in modern economies; we need robust anti-trust regulation to prevent consumers from getting ripped off and to prevent fragile supply chains. A well-conisdered balance of public and private provision supports the least well-off in society while allowing room for the fruits of individual flourishing.
Thirdly, we must consider what makes one economic system better than others. One way to measure this is to look at how efficiently it converts resources to social utility. I'm far from convinced that it's efficient to employ our brightest minds to build trading models with brief lifespans so that investors who are already well-off become slightly more so. It's worth investigating what regulations and incentives could put those minds towards things of greater value - solving climate change, cancer, sending humans into space etc... .
> This is a challenge to untangle. It sounds like you're saying that there is no point trying to regulate, legislate or control what happens in the economy at all. But that sounds bonkers to me.
I really do not appreciate this mischaracterization of my position. Focus on my actual words. I don't care about 'winning' this online argument. I take effort to engage because I am disturbed by the number of intelligent people who believe if only _THEY_ were in charge (or at least the right person), we would be able to fix all of society's problems.
> For starters, there are (and should definitely remain) absolute limits to business activities...
I agree with everything that follows. Government needs to be around to keep the peace. I want to be explicit: When I say "centrally planned/controlled economies" I am NOT talking about the general concept of regulation. If you are debating in good faith, this should be obvious. Look at all the history of failed states who tried to implement top-down control of their economies.
Also, YSK that not all regulators are government entities.
> Thirdly, we must consider what makes one economic system better than others. One way to measure this is to look at how efficiently it converts resources to social utility.
Never before in history has mankind been so prosperous. What system would you like to emulate? The US capitalist system is not perfect (and never will be)...but it blows all of its peers out of the water in terms of economic prosperity. Here's a couple data points: (Please read the technical definitions if you are truly interested in this subject)
> I'm far from convinced that it's efficient to employ our brightest minds to build trading models...
This is where my "closeted dictator" quip comes from. Nobody is "allocating" these minds...they are acting on their own free will. Why should you or anyone else be the arbiter? What if individuals disagree with your beliefs? Space exploration is a great example of a debatable "worthy endeavor"
Do you mean that high salaries indicate a demand in the market? Not much argument there, although it is sometimes the case that large companies hire talent purely to starve competition. But what I'm really questioning is whether those high salaries translate to value to society.
Some fraction of young minds, not all. I was happy to work at a small aerospace company with extreme concentration of brightest minds. Only because they loved the domain, and didn't mind a salary cut. What a joy and relief it was for me after FAANG!
No, I eventually left because money, I needed to save for a house.
Most of the staff were locals, so already had own/inherited property to live in. After FAANG it was new to work with mostly locals, way more stories about the surroundings.
Absolutely. The moral aspect is certainly questionable. Although, I wouldn't say "all brightest minds" are going to neglect their moral concerns for getting rich
To be honest, even this is morally questionable. It makes superficial sense in that someone else will do the work if you don't. But this happening on a large scale is still a drain on talent that could hopefully be used for work of greater social value.
Is the problem with the system or with the minds? The minds that want to scroll are the same as the ones that make money on the scrolling, that made the scrolling itself, and that made the system we're in. Why is it that defeatist comments always focus on the capitalism part and not on anything else? I don't think it's perfect like we aren't perfect but unless you have some particular suggestion this type of comment just reads "boohoo the world is bad and it's not my fault".
> Why is it that defeatist comments always focus on the capitalism part and not on anything else
Because endless growth is the only reason these once fun spaces have been hyper focused to be as addictive and stressful as possible to the "whales" of scrolling. That's why, when their own internal reports say "people spend unhealthy time on our platform and it's making them unhappy," it gets passed up the chain of command and whittled down by internal incentives until it dies as an issue.
Individuals hold some blame, but to put most of it on them is to ignore what growth demands. You're supposed to doomscroll and engage and worry. That's the business model. Facebook is in the same business as Cigarettes and Casinos. When I see someone on an air tank playing slots, literally crying when they spend their last dollar, I will not waste my breath blaming them. Just like I won't blame the doomscroller, anxious that they need to stay "informed," who hasn't met the basic needs in their own life.
> The minds that want to scroll are the same as the ones that make money on the scrolling
No? Where are you getting this? I don't think the people guiding these companies want to spend 6 hours scrolling TikTok. This is not the way most people live their lives.
> No? Where are you getting this? I don't think the people guiding these companies want to spend 6 hours scrolling TikTok. This is not the way most people live their lives.
I meant they are all humans. We're all sort of the same. If you disagree just think of your opinion of any other species, or about a group of people a thousand years ago, and you see what I meant.
Would they also change the aliasing assumptions to something close to C/C++? Otherwise I imagine it would be relatively easy to make mistakes and get "surprises" at runtime thanks to the optimizer.
Anecdote which doesn't prove anything but I have a colleague who left JS to work in FAANG after 6 months because working there was just too boring in his opinion (despite a higher salary).
Also, it's apparently significantly harder to land a position at JS than at a Google/Meta.
Actually, I applied there a while ago, the interviewer was actually pretty unpleasant, which hasn't happened to me at big tech. Didn't leave a really good impression.
Yeah I sent an application on a whim because I've spent a lot of time in FP land professionally and otherwise, I figured "what the heck, maybe they'll see value in that," and I was surprised to actually get a response but it was dismissive and borderline insulting..."there's nothing relevant here" talking about my resume is the thing I remember the most.
It was probably the first and only time I would have rather have been ghosted lol
Is it? Hours are long and the standards are very demanding, and in the end you're just manipulating numbers for profit, so that's not (for most people) a very meaningful endeavour.
I still rank it above making people click on ads though.
People really over hype and overrate what kind of work happens at Jane Street or similarly exclusive HFTs. They have their mix of interesting projects and mundane work like any other company.
Maybe, but could you live with the fact that you're providing absolutely no value to the world? It always makes me sad when the greatest minds get gobbled up by these companies because they're creaming off the top of an enormously complex system they created.
As someone that used to work in this world I find this opinion of “no value” pretty bizarre. There’s a lot of value in efficient markets with high liquidity and low fees, which companies like JS make possible.
Capitalism is the best system we humans have come up with at this point in civilisation, for all its flaws. It’s largely responsible for the technological, medical, and quality of life advances in the last 150 years.
Please tell me a better system that would cure cancer. I don’t think it’s Leninism or its derivatives. You need a macro system that is rich enough to allow for significant investment in medical research.
Having worked in both, there is a huge number of really smart people who turn down offers to work in quant shops. And people really underestimate how much tech will pay top researchers/overestimate how much the average quant makes.
I did work in the industry briefly and can confirm it is actually full of people who would post a comment like this. Just in case anyone needed another reason to avoid it!
Work on real problems. Try to make real people's lives better and happier. There are real problems in finance but my feeling was it's all very simple and solved decades ago, now it's just pointless complexity that isn't solving anyone's problems. I recommend John Kay's Other People's Money for a primer on what finance is actually good for and where it's gone wrong.
The real big problem in finance IMO is digital cash. Bitcoin started out trying to solve that problem, and there are still some people in the community interested in it, but it's mostly of interest to the finance guys now. Just another "instrument" in their "portfolios".
> Work on real problems. Try to make real people's lives better and happier. There are real problems in finance but my feeling was it's all very simple and solved decades ago, now it's just pointless complexity that isn't solving anyone's problems. I recommend John Kay's Other People's Money for a primer on what finance is actually good for and where it's gone wrong.
If the last few years have taught me anything it's that a large % of the population will actively aim to make their own lives worse long term because they are told lies. What benefit is there really in trying to undo their own self-inflicted damage.
The only thing worse than a complex codebase is a complex build system. I think CMake is an abomination. “Hey go learn this new language to compile this other language!”
>> Complexity has to live somewhere. If you are lucky, it lives in well-defined places. In code where you decided a bit of complexity should go, in documentation that supports the code, in training sessions for your engineers. You give it a place without trying to hide all of it. You create ways to manage it. You know where to go to meet it when you need it. If you're unlucky and you just tried to pretend complexity could be avoided altogether, it has no place to go in this world. But it still doesn't stop existing.
>> With nowhere to go, it has to roam everywhere in your system, both in your code and in people's heads. And as people shift around and leave, our understanding of it erodes.
>> Complexity has to live somewhere. If you embrace it, give it the place it deserves, design your system and organisation knowing it exists, and focus on adapting, it might just become a strength.
I think that's true up to a point, but it misses the distinction between accidental and essential complexity. Essential complexity has to go somewhere, accidental complexity doesn't, and most complexity in software is accidental.
And even when the complexity is essential, IMO it's better off not in the build system. I'll gladly accept more complex code for the sake of a simpler build (even though that theoretically means worse performance). Worst case if I need to do something complex at build time I'd rather model that as "the build system invokes a program that does something complex" than try to express the complex thing in some Turing Tarpit "configuration" language.
Also the point of software engineering is to take the real complexity that actually exists and you can't get rid of and work out how to write simple code to deal with it. Good software architecture does not include complexity because there is complexity in the business. It manages the complexity with simple code.
The trouble is when people say "complex" you don't really know what they mean, though. They often just mean "difficult". Every programmer who wants to use that word needs to watch this: https://www.youtube.com/watch?v=SxdOUGdseq4
> I think that's true up to a point, but it misses the distinction between accidental and essential complexity.
Very respectfully, I think you may be missing the author's point. When you fail to make a home for necessary complexity, it rears its head as unintended complexity in unexpected parts of the system. The source of 'accidental' complexity is unaccounted for complexity.
> When you fail to make a home for necessary complexity, it rears its head as unintended complexity in unexpected parts of the system. The source of 'accidental' complexity is unaccounted for complexity.
If that's what they're claiming then I completely disagree. No, that's not the reason, that's got nothing to do with it. If that were true we would expect e.g. projects with more complicated builds to have simpler code, and IME that's not true.
Again, respectfully, I don't think that follows (or makes much sense). There is not a set amount of complexity for all software that gets apportioned between the build system and the business logic, so that an increase in one necessarily means a reduction in the other.
Instead, each domain space has some degree of inherent complexity, which varies from problem to problem. Failing to account for this inherent domain complexity appropriately will cause it to bubble through at unexpected points throughout the system.
Build systems inherently have a very complex job. A good build system grapples with this complexity and tries to harness it; a bad one pretends it isn't there, and becomes a tangled mess once the (inevitably complex) demands made of it exceed its limited assumptions.
> Build systems inherently have a very complex job. A good build system grapples with this complexity and tries to harness it; a bad one pretends it isn't there, and becomes a tangled mess once the (inevitably complex) demands made of it exceed its limited assumptions.
I don't think this is true. I think that when looked at in the right way the job of a build system (when used appropriately) is actually fairly simple, and most build system complexity is either accidental complexity (either just straight-up bad design, or misguidedly overengineered flexibility in directions that don't matter) or comes from trying to accommodate things that the build system shouldn't have been doing in the first place. When I've seen overcomplicated builds they've never been because the build system made assumptions that were too limiting.
I am painfully aware of bitbake. I’ve probably written 3-400 recipes.
Most of them are about 20-30 lines long, because I refused to hide the compilation mess inside a recipe. I fixed the problem _before_ getting to the bitbake part. Most of my recipes at this point need only a repo name, the recipes are identical after that.
so simple huh? since you fancy yourself an architect can you let me know how we might fix the architecture of https://github.com/llvm/llvm-project such that CMake is no longer needed?
That’s such an obnoxious tactic. It’s basically saying “Your point is invalid unless you can cut this Gordian Knot on the spur of the moment in an internet comment!”
He did write ‘awful,’ which is a fair description of the experience of writing Lua.
I don’t mean that as an insult to Lua’s creators. They seem like really smart fellows. It’s just that the language is (to my eyes, with my background) viciously ugly. And 1-based arrays, of course, are evil.
It has some neat ideas, though, and it is supposed to be very easy to integrate into a project. But man, that syntax …
I don't mind CMake. It's awful, but I'm already using C or C++, so the bar is set pretty low. It does do a few things right, and at least some of the things it does wrong are manageable.
I'll eat at least a bit of shit if it means I can get more than one platform's-worth of build process out of a single set of human-editable configuration files.
Meson is quite nice if you want to build software exactly and exclusively how the Meson designers want to build software.
The world does not work like this though.
CMake is weird but once you've learned some non-intuitive stuff, it works very well and there's a reason why pretty much everyone is using it.
Depends on what you need; you can easily call out to external Python scripts to solve many problems, which will work everywhere because Python is needed to run meson itself.
I initially tried to use Meson for a small emscripten project targeting WASM, but it seemed stuck on the idea that I was cross-compiling (technically true), and this seemed to make everything more complicated. With CMake I was able to hack together a working build configuration without too much trouble. I just had to tell it that I wanted to use emscripten as the compiler.
The ideas embodied in modern target-oriented CMake are worthwhile. It's all hobbled by a dreadful script language but that could be improved with sufficient motivation.
I think what annoys me about it the most is that for every good idea in there there's one piece of tedious crap you have to work around, one actively bad misfeature that doesn't work properly, and one terrible aspect of the awful scripting language.
(And two minor annoyances that people determined to hate CMake will never shut up about. But then, if you're determined to hate something, there are worse things than CMake to do it to! So I can't be too critical.)
> I think what annoys me about it the most is that for every good idea in there there's one piece of tedious crap you have to work around, one actively bad misfeature that doesn't work properly, and one terrible aspect of the awful scripting language.
It's hobbled not just by that but by the fact that it's not the thing doing the building. It's a build system for build systems, and suffers from brittleness and confusion as a result.
CMake can generate a makefile, and this is the usual usecase. So a new language generating a makefile is even _more_ brittle than make, if your assertion is correct.
F#, while missing some of the more interesting features of OCaml (e.g. the expressiveness of its module system), also has indentation-sensitive syntax that ditches a lot of the scaffolding, including semicolons in most cases:
“If your intention is to compile or interpret files as scripts, double semicolons can and should be avoided when writing in OCaml. Leaving them does not raise errors, but they are useless. The compiler tolerates them to allow copy-paste from UTop to a file without having to remove them.“
https://ocaml.org/docs/toplevel-introduction#loading-librari...
you can, for now at least, still use reasonML with js_of_ocaml to compile to javascript if you like reason syntax but don't want to a more complicated relationship with the ocaml ecosystem (we use this for https://hazel.org)
It's funny because the thing that always turned me off from Haskell was that it feels like a stream of whitespace and identifies without good visual cues as to structure.
I'm sure once you get the zen of it, it's fine. Like Lisp, I guess, you learn to think in its structure. But looking at a screenful of Haskell to me is intimidating.
Of the bunch I found SML/NJ to be the most readable.
But in any case, you can use curly braces and semicolons in Haskell just fine. You can also write your Haskell like Lisp, and add lots of parens everywhere, and use all operators in prefix-form.
I wonder if anyone has used Jane Street's open source OCaml libraries and tools for somewhat serious projects.
I took a brief look at these things, and my impression is that their stuff isn't "ready" for anyone outside Jane Street, even though they put a lot of effort in building the ecosystem and open source their code.
I do! Their standard lib replacement was used on the project where I originally learned ocaml, so I have just always used it. You have to be fairly sophisticated at reading fn signatures to find things sometimes but otherwise no complaints with it. It has some very nice features too for example convert functions to/from sexps for all base types can save you a ton of time if you know it's there and plan for it.
I've used some of their other libraries too, their logging and unit test ppx are common maybe even de facto standards as much as the ocaml world has such a thing. I've also used, off the top of my head, their code formatter, one of their test frameworks, their implementations of some advanced data structures.
Sometimes you do run into one like the other commenter said, where that shit just does not work. It depends on an undocumented something they shipped separately, or needs a secret bit of config or whatever. These aren't malicious, I open a ticket and come back in a year or two often they'll be working.
It's not zero frustration but I appreciate their approach of just throwing everything over rather than spending more resources testing and polishing fewer releases. Their code quality is generally very high and even if I can't get something working directly, it provides a rigorous & vetted example implementation.
Thanks! The "depends on an undocumented thing" happens quite often for such projects from companies, and is enough to hold people back -- in a company where a third party library requires approval, such issues mean that a fix could take a while before it is available internally, which could delay your own work etc. A lot of this probably comes down to the ecosystem -- not enough people are using these things to discover them early.
I'm running tests for an involved math research project, and to my great surprise Lean 4 << Haskell << OCaml in execution times (so far).
I love Lean 4, but good luck getting help with it from AI. Today's project-in-progress is digesting their reference manual to fit well within a 200K context window. We'll see if that helps.
kinda surprising that it still makes sense to develop your own new build system, and then do the migration to it, instead of adopting an off-the-shelf one like buck2 (or bazel/etc...). Not saying that these fancy build tools are easy to adopt, but compared to building your own? is there something remarkable that makes dune much different?
Without getting into any specifics of it - I'm sure there's people with much more experience with these tools who can comment - I'll point out that neither buck nor bzl existed when JS decided to start building their own tool in 2012. Bazel's first release was in 2015, Buck's was in 2013.
JS does have a bit of a NIH culture, but I'm not sure if that was really at play here. There just...weren't very many good build tools available at the time, particularly for a company using an unorthodox tech stack.
> I'll point out that neither buck nor bzl existed when JS decided to start building their own tool in 2012. Bazel's first release was in 2015, Buck's was in 2013.
But Dune started (according to this blog post) in 2016 and JS started seriously improving and adopting it last year. So to me Jenga sounds like a reasonable step in 2012, but pouring significant effort into migrating from Jenga to Dune (and improving Dune) in 2024 sounds more weird
Yes and no. This is all spelled out in the post, but it's a little thorny.
Dune is a rename of Jbuilder (2016). Jbuilder uses Jenga (2012) configuration files.
> By 2016 we had had enough of this, and decided to make a simple cross-platform tool, called Jbuilder, that would allow external users to build our code without having to adopt Jenga in full, and would release us from the obligation of rewriting our builds in OCamlbuild [...] Jbuilder understood the jbuild files that Jenga used for build configuration.
So in 2012 it made sense for them to build Jenga, because there weren't any good alternatives - Bzl etc. didn't exist, so they couldn't have solved their problems.
And in 2016 they had open-source code they wanted others to be able build; those people didn't want to use Jenga, and JS didn't want to rewrite their builds so that they could use something else. Thus, Jbuilder was a shim so that JS could still use their Jenga builds and others could build JS' code without using Jenga. Bzl etc., even though they existed, wouldn't have solved these problems either.
I don't know ocaml particularly well but my understanding is that only with buck2 (of those mentioned) do you have a build system with the semantics to actually compile ocaml properly.
In particular I was under the impression one needed to be able to run ocamldep before hand (or compile twice) - buck2 can do this, bazel needs hacks iirc.
I loath autotools with a passion, and it's awkward to set it all up, but it handles OCaml code just fine. We use it for multi-language projects (including OCaml) like nbdkit & libguestfs.
My team has a large ocaml codebase and a dual build system (buck2 and dune). The two have roughly similar: dune faster at the raw speed of invoking ocaml build tools, buck2 winning out when a lot of stuff has to be rebuilt and it uses distributed build system.
The major pain point is LSP integration, which has to be closely tied to the build system, since it's only by building that the LSP server can know a file's dependencies. Everything is all neatly available with dune. We've cobbled something together a bit with buck2 but it's not as nice.
I respectfully have to disagree. As Fedora packager for OCaml packages, dune's configuration is yet another one-off, with a weird Lisp syntax, hard to understand, difficult to patch. I'd far rather people stuck to using more familiar Linux build systems. As an upstream OCaml developer, the whole thing falls down the minute you need to integrate other programming languages into your build (or OCaml code into a code base written in another language).
(This rant more or less equally applies to other language-specific build systems.)
"Familiar" is not a property of any system. It's a relation between a system and a user.
Some Linux build systems maybe be more familiar to some users, but will be less familiar to others. When picking a build system, you can't just look at the system itself and declare it familiar or not.
It's not even enough to look at the total number of users familiar with some thing. Hindi is one of the most familiar languages in the world, but you're probably gonna have a bad time if you use it for the menu in a cafe in rural Texas.
You have to look at your actual cohort of users (and potential future users) and see what's familiar to them. This is one of the key reasons why usability is actually a deeply hard problem. So much of usability hinges on familiarity, but familiarity is a human-specific highly variable property.
As a user of Linux, I wish Linux distributions would abandon their build systems which really only work well for C and maybe C++, and also only work on one platform (not even the whole of linux!) meaning that packaging has to be duplicated for every OS, and stuck to using more familiar language-specific build systems that work everywhere.
(this rant more or less equally applies to all os-or-distro-specific build systems)
---
This rant is only semi-serious. I do see some value in the Linux distribution style packaging. In particular, I do sympathise with the need to do cross-language builds. But goodness are they a pain to work with, and probably the biggest barrier to me shipping software on Linux.
My hope is that eventually an evolution on build systems like bazel/buck2 will lead to a truly universal build system that is both cross-platform and cross-language. But unfortunately it doesn't look like it's coming soon.
> [Distro-specific build systems are] probably the biggest barrier to me shipping software on Linux.
Don’t[1]. Ship source tarballs (or VCS tags).
I’ll grant that most distros’ build systems are antiquated and, in places, silly. (That includes Nixpkgs, first released 2006.) We could really use some fresh ideas there.
But they’re also not for you (or me) in your (or my) capacity as a software author. They’re there for a person who works on packaging software for, most of the time, a single distro, and their balance of complexity and flexibility is calibrated accordingly. One of the functions of that person is also to keep you honest and represent the interests of users before you, because they have more expertise than the users but not as much of an attachment to your software as you. The ecosystem is less healthy when the author tries to fill in for the packager.
“But then the users will come to my bugtracker to complain about bugs in patched versions!” Pre-Google, we used to have a solution for that: a configure option to set the bug reporting email, present in all GNU software. Nowadays it’s not clear what a good solution could be, but it does seem like, unfortunately, the author will have to maintain a table of packager contact information for the end users.
That doesn't scale though beyond the most popular open source apps especially those in c/c++
Less popular apps or closed source apps don't work. Things with old dependencies won't work.
Other language with package manager which depend on dependencies and dependency versions that may not be in distro packages may have trouble.
For the core the distro model might work.
For the rest maybe something like flatpack from the devs might scale
This is why I'm really looking into SerpentOS and their moss build system. Kind of like a combination of Nix and AUR. Simple syntax, simple to create packages, atomic.
Make is primarily used with C and C++. It is not commonly used in Java, Rust, Go, NodeJS, or hardly anything besides C and C++. Make is not "generally used with other languages".
This doesn't prove anything at all. Of course the toolchain has to be built somehow. Some toolchains use make to do that, rather than depending on the previous version of the toolchain's build system. Some toolchains are written in a language completely separate from their downstream language, so they obviously wouldn't be compatible with their own toolchain.
Downstream projects in these languages do not typically use Make.
I clicked on the Rust one, and not only did it seem to be specific to some old testing infrastructure, but I found this note:
> There are two kinds of run-make tests:
> The new rmake.rs version: this allows run-make tests to be written in Rust (with rmake.rs as the main test file).
> The legacy Makefile version: this is what run-make tests were written with before support for rmake.rs was introduced.
So, it's an obsolete system that is being migrated away from.
But, again, the main point is that what the toolchain does with its free time has little to do with how end user applications are developed, and the complaints in this thread were strictly about building applications in distros, not about building toolchains.
If an application in one of these languages uses make, it is typically just a little syntax sugar around the toolchain commands, which does absolutely nothing to absolve the project of the complaints Linux distro maintainers have about how dependencies are managed.
In case you're not trolling (and it's really hard to tell), those makefiles are for building projects whose source code is written using C or C++. The projects they are building are things like the Java runtime, Go runtime, or the Rust compiler, but they are not building projects whose source code is written in Java, or Rust, or Go etc...
What people are claiming is that make is used as a build system for projects whose source code is written in C or C++.
Make is the common denominator in most projects I come across regardless of language. I see lots of frontend projects and certainly Go and Rust projects using Make quite often.
Ironically many modern C/C++ projects use Cmake to generate Makefiles. If anything the inverse of your observation is mine.
Are those Makefiles doing anything more than calling "go build" and "cargo build"?
Because if they're still using the language-specific build tools and dependency management systems, then I think you would find that the Fedora maintainer higher in this thread would not be any happier that there is a sugar coating of Make. That's not what they're asking for, based on other rants I've seen from Linux distro maintainers.
I haven't been doing much OCaml in recent years, but I do sometimes check in to see what's going on in OCaml-land, and it seems like dune is what everyone is using now - what are the alternatives?
> This rant more or less equally applies to other language-specific build systems.
And yet those language-specific build systems are overwhelmingly winning, in pretty much every language.
> As Fedora packager for OCaml packages,...
I honestly think traditional Linux packaging is in the wrong here and the problems are essentially self-inflicted (not in the sense that individual maintainers are doing something wrong, but in the sense that the policy that traditional Linux distributions are following is inherently unsustainable. It's designed for a pre-CPAN world)
> As an upstream OCaml developer, the whole thing falls down the minute you need to integrate other programming languages into your build (or OCaml code into a code base written in another language).
True up to a point, but frankly the worst case is falling back to a terrible C-style build, and "always do terrible C-style builds in case you need to integrate with C code" is not a proposition that has much appeal.
Much as I wish the whole world would standardise on Maven or Cargo, I can't see a realistic path to there without first eliminating C, because the C people are never going to agree to follow a standard for package repositories.
The issue is there. Not you or your fellow packagers of course, but in the idea that every linux distribution needs it's own packaging system, and each version of a distribution it's own packages.
> I must not segfault. Uncertainty is the mind-killer. Exceptions are the little-death that brings total obliteration. I will fully express my cases. Execution will pass over me and through me. And when it has gone past, I will unwind the stack along its path. Where the cases are handled there will be nothing. Only I will remain.
reply