We’re experiencing this at my place of work. Our backend stack is Python, and coming from an Elixir/Phoenix background, adding a durable queue to our infrastructure should be trivial but because of the nature of Python, it is all but impossible to run the queue in the same codebase and application server.
Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.
Interesting, most of my work has been 'across the stack' but for desktop applications. This looks like a better match for me than most other approaches to web applications.
It is a very good point: I did a lot of desktop apps at one point (the largest being a mixture of C# and legacy C++ wrappers), and the first time I tried LiveView it somehow reminded me of WinForms!
Every time I see microservices, or Kafka, or Redis, ... I can't help being disappointed because we seem to be stuck reinventing BEAM/Erlang (and Elixir) 30 years after the fact, except with technologies that are slower, more fragile, more complicated to use and harder to deploy.
Someone in the Erlang community named our processes “nanoservices” as a joke. I thought that was pretty apt though, a gen_server is pretty close. And you can have 3M of those on one node!
> it is all but impossible to run the queue in the same codebase and application server.
I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.
It sounds mad but because of the process isolation within the language you can trust running services within your application and message passing can be local or to a different machine in the BEAM cluster. It opens up a whole new way to think about a runtime once everything is an actor model process with a mailbox you can send messages to.
It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.
But why? I have one project that is a single Python codebase that embeds the whole http server, task scheduler, a durable queue implementation, and job processing implementation using those queues all in a single process.
I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?
Are there any examples online of doing these things with simple (or just not as complex) elixir code compared to using dedicated libraries in other languages
FYI "all but impossible" usually means the opposite of what you think it means - "all but" meaning "everything except". Seen this usage a lot and it bothers me, because both usages are common and theyre the exact opposites.
If I didn't have context clues, it'd be difficult to know which one you mean.
Agreed, to me the meaning unpacks like: "it meets or exceeds every category of difficulty that is less severe than Impossible."
Compare to a disaster described as "the destruction is all-but-total".
That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".
“All but” = “almost” (which presumably historically was “all most”).
Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).
I'm not sure how it makes sense to claim that an action is "everything except impossible." This is more in the classic sense of "the battle was all but lost" for a battle was practically but not actually lost. You could implement a queue in a Python server, but it would be technically awful to do.
In this context I meant “all but” == “everything except” i.e it is almost impossible. Are you saying it means the opposite of how I used it? I’m not certain what you mean.
In a lot of these BEAM threads I see certain features listed as "part of the benefits", but when I look to see who is using them in production, most of the advice is "don't use that, use an established service to do that".
This is usually things around either ETS/mnesia for state storage or doing hot-releases, both of which I find most people in the community recommend _against_.
A few examples:
- Server-wide state
- Persistable data
- Background Jobs
Having worked in a heavy Erlang codebase and enjoyed some of the features of it, but realized some of the shortcomings where when reliability is a primary concern (medical/healthcare), a lot of the advice changes.
I dunno, the blogging community of Erlangers seems to have a different opinion of hot-loading and mnesia scalability than I developed working at WhatsApp.
Specifically, hot loading allows for tremendous agility in rolling out code: for better and worse. There's sharp edges, but it's worth it IMHO (of course, it should be noted that AFAIK, hot loading is either unused or very rarely used at WhatsApp now. I believe they now use fairly conventional rollouts where new instances are started with the new code, load balancers shift the load and old instances are shut down eventually... back in my day, we didn't have load balancers or easy access to spare machines to rotate through, etc)
Mnesia has lots of sharp edges too, but if you are willing to learn them as you grow, and maybe do a bit of work here and there, disc_copies and ram_copies tables scale pretty good (afaik, we only used disc_only_copies for the schema table, and that's a small table, so it works; I don't think that table type works well with large tables). And having data and application colocated makes a lot of things pretty nice. IMHO, if you are doing the volume of data and accesses we were doing with any other database, you'd need in house engineering on the database as well, so I don't see it as a strong negative that it's needed for mnesia. If you have a smaller need, maybe it works out of the box. IMHO, mnesia works best with stable nodes and stable network though: the original use case was for two nodes in the same chassis, and the farther you stray from that, the more work you may have to do. But we had replication setup between Reston, VA and Dallas, TX and it worked fine as long as the backbone between the two wasn't being extra flakey.
Again, I think post-acquisition, mnesia was phased out, but FB doesn't value stability of nodes or networks.
I think it's more that it's not recommended within the web development community which, of course, was Elixir's initial "purpose". As a web developer who has worked with a lot of web developers, I think this is pretty sound advice. However, I do wish there was more multi-node literature out there for web developers as I'm very interested in it myself. I'm talking running different OTP applications on different nodes. Almost every blog post I've found on the topic doesn't go much farther than `iex --sname foo@localhost --cookie monster`. Hopefully more in depth stuff will start to show up.
I should think so. I certainly haven't heard differently, and I'd expect some sort of engineering blog about rewriting an chat system evolved from ejabberd in C++ for a second time.
Yeah, like anything, the BEAM world can get a bit hypey.
From the first slide in the linked article, I would almost never use Erlang to store data. I'm always going to reach for Postgres there unless I've got some very specific use case.
I do think having most everything else "under the same roof" is nice for a smaller company that doesn't want to run a whole bunch of different services, though. Having an internal cron thing, and an internal job thing are probably fine for a lot of people getting something off the ground and easier to deal with than having something external to deal with. I think in particular this is often nice when trying to replicate the production environment on both a staging and local development system. If the infrastructure is complicated, that's harder to do. It's nice to be running substantially similar systems everywhere.
I'm not 100% sold that it's that big a win compared to having more libraries and such available with, say, Rails or PHP or Python or something. But it's definitely nice to have.
The main issue with Phoenix on the front-end is all the great component libraries target React. Developing accessible, battle-tested components (even a simple dropdown is much more involved than you think) is extremely hard and can drain a lot of development time. I've seen a few blogs about using React alongside LiveView--even sharing state--but I think we need something official.
I did heavy LiveView development for a while. The library problem definitely exists. It's very rare to find what you want already implemented, so you have to rebuild it yourself. It takes a lot of time.
What I ended up doing was loading react components using LiveView hooks and syncing state using pushed events. Saved me a ton of time because libraries were all there, and worked seamlessly alongside my LiveView app.
I personally wouldn't use React for components in LiveView, you are just adding more complexity to your application for no reason. It really not that hard to build components using JS and LiveView. In the end you get less complex components than what you would get with react.
Thanks. I took a cursory look and they just don't seem as high quality as something like shadcn/ui (built on Radix UI) or Tailwind's upcoming Catalyst. For example, on Moon, the dropdown arrow support seems broken and it doesn't switch positions based on the viewport. It's these little details that matter, especially for accessibility.
What about wysiwyg editors? Something equivalent to react-grid-layout? Date pickers? I want to build a startup, not rebuild complex components for the sake of purity.
Simple components are taken care of, but I think that's not where the discussion is very relevant.
I'm sorry to disagree here, but I think it's disingenuous to act like the problem is solved when there's really quite a large gap.
(Btw, of course I built my components with pure LiveView when it made sense.)
I always find good vanilla JS for everything I've needed (though my needs haven't been too special).
Depending on what you're building you could also say: "I want to spend time building my business, not writing an API for myself just so I can communicate with my own backend." Which is not even to mention how much simpler concurrency features are. But again, it depends on the application.
True, I used a vanilla wysiwyg editor integrated via hooks.
To be clear, I heartily endorse LiveView. Not using it now due to pretty heavy chrome extension needs, but it was solid. It's just rough in the "just works JS integrations" department. I'd argue you have to be really good at JS to handle any heavier library integrations.
> I'd argue you have to be really good at JS to handle any heavier library integrations.
You aren't wrong there! LiveView was initially developed to make a certain class of web-app without needing JS. Of course, people went and pushed it further and I see it as a "only write the JS that is necessary" type of framework. I'm very fullstack-minded so this doesn't bother me in the least. I love writing JS for DOM manipulations and whatnot, I just don't want to write any business or server-side logic with it, so LiveView is a great fit for me. But even as an advocate, if your application needs to be very JS-heavy, I wouldn't necessarily recommend it. No frameworks are one-size-fits all, but of course most people want to work that way.
With all due respect, if you want to build accessible components, you cannot rely on JS commands alone. I've been down this road and ended up writing JS commands combined with JS Hooks for just about every component to get what I actually wanted.
Elixir/BEAM seems like an ideal solution to many common problems. Why has it not caught on more yet? Can anyone who uses it give a devils advocate position or is everyone just lagging behind on something thats going to get more and more common?
A lot of devs really don’t want to learn anything but the very most popular languages. A lot of managers don’t even understand that what language they use makes a difference in terms of their product or their productivity—they optimize for whatever stack they can most easily the hire the cheapest devs for.
That leads to the market responding much more slowly to newer options, even when they have clear advantages. Smaller startups and indie hackers don’t have the same switching costs though, so you’ll see more of them making a bet on a less popular technology they believe has a productivity edge for their use case.
I'm pretty new to the community, but I think there is something of a first-mover advantage which Python, PHP, JS, etc. have which Elixir does not: The former languages all have mature solutions to a lot of problems (in terms of packages available, not language capabilities) which Elixir has not had time to catch up to yet.
Plus since these solutions already exist, there is less incentive to switch things over to the newcomer when you already have something that is working.
BEAM is pretty cool, but some of what that article is talking about, the fact that you can do everything under the same roof... it's nicer than, say, Rails, but it's not like 100 times better either.
I hear so much good stuff about Elixir and Phoenix (often from fly.io blogs) but I've always dismissed it for the lack of type support. Is there a TypeScript equivalent on the horizon? Or is it not so much of an issue?
Eh. I designed some kind of static type system for Erlang ~20 years ago. It wasn't very good and I changed job before I could test it on anything serious, but it was fun!
Glad to see that some people are doing a more serious job than me :)
(also, it's a small world – I know Giuseppe Castagna, wasn't aware that he was working on typing Elixir)
The language is still strongly typed, and with the language being purely functional, I found myself not missing static typing nearly as much as I thought I would. Things just work, and they break during compilation if they don’t.
I will say that editor support for static languages is categorically better due to the language server being able to more completely reason about the state of your code. That’s the one drawback.
Parent is making a distinction between “strongly” and “statically” typed. By some definitions. Type conversions in Elixir are generally explicit (contrast with JavaScript) and the type of a variable matters at runtime (e.g think about pattern matching), and so some people would call it “dynamic and strongly typed”.
You also can't do "foo" + "bar" to get "foobar", you have to do "foo" <> "bar" giving it even more type "strength" than many strongly typed dynamic languages have.
Is that a common source of typing errors? Genuinely wondering as I don't have a whole lot of experience with statically typed languages. I've personally never had a bug related to it.
I wasn't meaning to imply everything is perfect, though. For example equality operators take anything. I believe there may be changes coming around there.
I think it is pretty standard in functional languages not to overload operators. There may be technical reasons as well, I'm a bit out of my depth here, but it probably is better for performance with pattern matching knowing the type? Like `"hello " <> there = "hello there"`. Of course it's not just strings—concatenating lists uses ++ and for dates have their own functions.
OCaml takes it a step further with numbers where `+` only works on integers. You need to do `+.` to add floats. This lets it be statically typed without having to actually specify any types.
Again, a little out of my depth in terms of rock solid explanations.
I've done both strongly typed & "less typed" languages over the last couple of decades, and I do not find the current situation a problem in Elixir. We're using Dialyzer on my main project, without too much interest I must say. I am following the discussions on that topic (improving types in Elixir) with interest, but it's not an issue for me at the moment (and large refactorings occur without much troubles either).
I think the fact that most functions are "pure" / "static" is really helping with refactorings.
I’ve been using Elixir for most of the past six years and really haven’t ever been bitten by the type system. I’m fairly sure that pattern matching , structs and immutability is why.
This topic has come up quite a few times on HN, but it seems like it’s generally people who haven’t ever used Clojure or another similar immutable functional language that worry about this.
IMO, Elixir and Clojure hit the sweet spot between a language like Ruby and one with an ML-inspired type system.
It not as big of an issue as people make it out to be. Maybe it becomes they haven't used a functional programming language. I have yet to have a problem with Elixir where static typing would be the only thing that would solve the problem.
Elixir has great support for pattern matching, guards, documentation, testing and being a functional language all help negate the problems of a dynamic language.
Elixir has features called guards which allow you to check for a condition to be true before a function is called(and will match another version of the same function if it is false). This is used as a crude form of type checking.
They were saying guards are a crude form of type-checking, not intrinsically crude.
Really they are a perfectly regular form of type hinting--as is pattern matching in function heads--when being used for no other purpose than to ensure the correct type is passed at runtime.
It might get type support. They have been very upfront that they might fail and will only proceed with it if they can come up with something that is right for the language. It's not a rush to get a type system just to attract more users. I'm very optimistic they'll succeed, though! :)
I hear this a lot. I get that it is helpful. But I know people run large projects without it in Elixir and seem to do fine. Discord and remote.com seem likely to be fairly massive. Last I spoke to Remote they did 70 releases a day and no type static typing.
I know from Elm that it does change how things feel. I could see an upside. But coming from a dynamic life I can't quite grok considering it a requirement.
It really depends on your definition of your massive. Discord seems to have a few hundreds to a thousand employees in total. I am from a typical tech company. So I could estimate that their total engineers count is probably just a couple of hundreds, which is about our director/senior director's team size.
In our definition of massive, we could have even hundreds of ppl to touch the same portion of the code base frequently. We regularly have to deal with code which are about 10 years old and it's a moving target. You have to deliver quickly with migration requirement in mind. Also we frequently collaborate with the other teams and you have to be able to understand others' code quickly.
Typing is critical as you don't have time to deal with the constant stream of the new projects to dig deeper in to the objects. And you really want to have some understanding of the code in the first glance. We also want to offload lots of the things to compiler and ide as much as possible.
It doesn't have to be full on static typing like Java, type annotation in python/ruby already helped a lot
> massive as in the top 10% of tech companies in the entire globe
But are you talking about the size or the impact of the project? I am in the fintech space and the complexity and cruft is just incredible thick. We just needs so many ppl to do the dirty work. There is no way for you to shrink the code base fundamentally
> quote: WhatsApp scaled to 1B users with only 50 engineers [1]
Again, are we talking about the size or the impact? Are you saying all the projects for in faang can be handled like this? Meta currently have about 70k engineers. What do you think that all those people are doing?
> What do you think that all those people are doing?
justifying the salary of middle managers.
at my current job the number of employees has increased 30x in just 5 years.
there's no important single task we couldn't do when we were 1/30th of now, 2.5x would have been the perfect number to make all of us work no more than 3 days/week, now we are just working on more tasks that are basically DOA or scratch the itches of higher ups who want their name on something when they meet with other higher ups but are useless. Moreover, more often than not, they are a minor variation of something that already existed for a long time, works better, serves a real business purpose and it's written in legacy languages because it just works (I'm sure many of us have attended the and now with AI! BS meeting).
We also know Facebook's 70k employees are there so that many of them won't end up working somewhere else and help the competition (see the Silicon Valley No-poaching Case).
I'm an advocate for not burning up your employees and hire more than you need so everyone can relax, I'm a union representative (on of the) in the company I work for (yes, my country is highly unionized and thanks god it is) so I deeply support the needs of the workers against the absurd demands of the companies, but at some point too many people becomes a burden and severely hinders communication, it basically creates a layers of bureaucracy that resembles much more politics than the job you were hired for, ending up stressing people more, creating harmful frictions, complicated workflows, absurd chains of indirection and countless authorization requests that in the end reduce everyone's output, regardless of the commitment of the single employee.
Also, larger companies get a lot more slack taxation wise and receives a lot more cuts from the government, because unemployment makes us look bad.
That's just ridiculous if you think this applies to all projects. We have incredible amount legal, tax, sanction etc in the code. Have you seen how complex real world financial and legal system can be, do you know how much work to translate those into code? And keeps updating it?
And to just to interface with other financial system worldwide, we have to keep the old system in place, at same time upgrade as the industry move forward. It's a also a big challenge. Despite the above and we had to ship fast, we are actually still SHORT on people. Our engineer to manager ratio is also high. The world is a complex place, sometimes you just have to deal with it.
Again, why do you think 50 man team applies to all projects? What is it so hard to believe that some complexity cannot be reduced?
> We also know Facebook's 70k employees are there so that many of them won't end up working somewhere else and help the competition
You get this from the TV show. Have you actually worked in the companies at Meta caliber? Do you really think most faang employees just do nothing everyday?
> Have you actually worked in the companies at Meta caliber
you're taking it personally.
I work for a > 30k employees company, not based in the US.
All I'm saying is companies optimize for what's better for them, so at some point you'll have on payroll thousands of people you don't need for your core business because they solve a lot of non-core functions. such as dealing with authorities or devise a strategy to reduce the pool of talents competition can pick from. HR, accounting and legal departments will also increase in size a lot consequentially. so their massive size doesn't says much about the complexity of the tech problem they are solving.
the fact that Discord runs on Elixir at that scale is remarkable, regardless of how many people work at Discord.
This is a great resource to map "what library/thing should I use for x situation" which can sometimes be daunting to someone new to an ecosystem.
The "niche" label cast at Elixir and its ecosystem hasn't been relevant for years; it can do what you need it to and with massive reduction in complexity.
If or when you do need to write your own library: you'll find that straightforward as well.
I recently deployed a Phoenix LiveView app[1] on a shared-cpu-2x 512MB fly machine[2]. Configuration and deployment was non-trivial with their CLI. I ran `fly launch`[3] which auto-detected the project's codebase as a Phoenix app which then auto-generated a Dockerfile with support for rust NIF built using rustler so I didn't have to do anything extra to support the project's rust dependency. Overall, it was a convenient experience and wanted to share.
Love Elixir and Phoenix, but the last time I checked (granted, two years ago), the major lack was learning material for people who weren't coming from Ruby experience.
I agree with this sentiment of lack of learning materials. I've been through the Liveview for elixir 2.5 times, and the programming pheonix book when it came out, and the they cover the simpler things of elixir. But they don't cover some of the benefits listed in the article (background jobs, service wide state, etc) with solid examples. But maybe I'm not thinking about it the right way because I am coming from a rails or whatever perspective.
I've been brainstorming writing my own book that covers elixir pheonix crud, auth, background jobs, live view, (+ more?) using a twitter-clone example, which feels sufficiently complex. Basically writing a book that I wish existed.
The Elixir and Phoenix books by Pragmatic Programmers are very good. Beyond that, a little bit of practice and getting used to the docs is all I've needed (so far).
It's been well over 2 years since I tried it out & I thought the Elixir docs were really awesome when I did it. They had great walk throughs to getting started.
I also did Rob Conery's Elixir tutorial which was fun - https://robconery.com/course/take-off-with-elixir/#/ Absolutely loved the way he did this one as a story but it also didn't feel like I was sitting there forever waiting for him to get to the lesson either. Looks like it's out dated though.
I have found learning materials to be plentiful. I came from Python which I guess looks sikilar enough to Ruby. But of course what paths you hit can vary wildly.
Honest question, if its so great an ecosystem why is it not adopted more?
I can cite some easily available public stats: Scholar, which is compared to numpy, has like 0.001% of its popularity. (While there are rumors people buy likes on github I doubt this is something affecting the numpy project).
Maybe the reason believers in elixir/erlang are more passionate is precisely because there is no obvious reason there isn't major adoption?
My best guess is that the missing ingredient are major corporate interests that directly or indirectly are promoting the technology. E.g the usual big tech suspects seem to be absent. For better or worse, they are the ones dangling lucrative jobs prospects etc.
But if its a technology delivering the goods people would gravitate to it sooner or later. Would be interested if more informed people can add their insights
Comparing Scholar or any other Elixir ML component is bizarre. Elixir has been around for many years and its ML ecosystem has only existed for the last few. Obviously Python, the most used ML ecosystem in the world, is going to be way more popular than Elixir.
ML isn’t even close to the primary use case of Elixir either. This is just a bizarre comparison.
Also, Scholar would be equivalent to Sklearn, not Numpy. Nx would be the Numpy equivalent. Scholar has literally only existed for 2 years.
I would like some community feedback on this. I have been attempting to try Elixir and Phoenix and did with some small projects. However, when it comes to db, that is where I found it challenging. Ecto supports SqL and Absinthe provides graphql. However, if I want to roll out some of my own with Surreal, edgeDb and Mongodb, Neo4j or something else. I am forced to eject out ecto. I am fine with that, but how do we support other Dbs easily as Ecto or Absinthe. Are there any libraries I am missing? Has anyone tried with other options. I saw somebody mentioned mnesia. It can be an abstraction maybe .
Mongo appears to have a decent driver for it. I'd lean to that if i were looking for an alternative to SQL.
You shouldn't have any issues integrating it into Absinthe, because it doesn't have any ecto bindings out of the box. (There's a helper library, but even that is fairly small.)
I have heard of people using mnesia, but it's not really that common. I'd personally avoid it unless I had a good reason not to.
This post is missing one huge point: its tooling is bad. Like, real bad. The language server will regularly crash, does not support code actions _at all_, messes up formatting regularly by ignoring the configuration. Then credo, the popular linter, will tell you that your imports are not alphabetically ordered but there’s no fix mode, meaning you’ll go back to order them yourself.
I love the language, am using it during my day job right now, and have published multiple libraries, but the development experience can be infuriating.
I also develop Elixir professionally and I find the tooling to be excellent. I've seen some transient language server crashes[1] - but I wouldn't call it common? I personally dislike Credo, but it's very customizable, and you can simply remove the "alias order" readability check[2].
I find the tooling excellent. `mix` just works, and `alchemist` mode in emacs provides everything I could imagine needing. I've never heard of credo, but adding a call to `elixir-format` on save solves any linting needs I've had.
Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.
I miss it.