Thank you for the reference (not finished it yet).
Worth mentioning functions that can error, should follow the {:ok, response} | {:error, reason} pattern. Because if such a function returns response | {:error, reason}, then if we are inside a with clause and we want to capture the response and use it in the next with clause, such capture value can be either response or {:error, reason} - which goes around the pattern matching.
with response_f1 <- f1(),
{:ok, response_f2} <- f2(response_f1) do
# do something
else
{:error, reason_f1} ->
# we will never come here
# because the returned value from f1
# is already matched to the variable response_f1
end
your solution also works if you define the function headers with a pattern match against the tuple, but then you have this extra function hanging around. Feels like a style thing more than anything else.
This approach is not equivalent since it uses a strict match in the function head inside `then`. It will raise a `FunctionClauseError` if a value not matching `{:ok, _}` is passed in.
I mean, yes, you're right that the failure mode of the `then/2` approach works differently than `with`. I think most semi-experienced elixir devs would recognize that the function used with `then/2` needs to have a pattern that matches expected returns—as does `with/else` if you want to be able to continue on the happy path of your program.
If that's the behavior desired, experienced Elixir devs would use the match operator since it's more conventional, requires fewer characters, and eliminates multiple unnecessary function calls. The thread was about monadic transformation operations similar to `and_then` on Result and Option in Rust or Promise#then in JavaScript.
I've seen that recommendation often but I still dislike it, since it falls apart whenever you need to do the same logic on multiple pieces of data. It only makes sense if every single with-clause is doing something hugely different.
A simple example:
with {:ok, cleaned_first} <- sanitize(data.first),
{:ok, cleaned_middle} <- sanitize(data.middle),
{:ok, cleaned_last} <- sanitize(data.last),
{:ok, final} <- combine(first, middle, last) do:
{:ok, "Name is" <> final}
else
{:error, :illegal_character} ->
{:error, :illegal_first_name_or_middle_name_or_last_name}
other ->
other
end
The above sucks because it's not clear which data was bad or which step failed.
So let's talk about the alternatives...
__________________
First, the "Lots Of Methods" approach, where you just add as many different private methods as you need, where each one works almost the same except for a distinctly different error:
I feel that sucks because it's tedious, error-prone boilerplate, even if they're just wrappers around a sanitize/1 that does the real work. Also any new error-atoms being introduced are scattered down the file rather than near where you might want to match/test them.
__________________
Second, the "Augment One Method With Causal Data" version:
This is a marginal improvement, but we're still contaminating methods like "sanitize" with junk they don't actually need to know in order to do so their job, passing a piece of opaque data down and up the stack unnecessarily.
__________________
Third, what if we carefully isolate the concerns/complexity to the with-statement... Hey! We're right back to where we started!
The "Augment The With Clauses" approach, which I argue is least-bad:
i agree that with statements aren't the easiest to understand, especially for a beginner, but I think the value in having the entirety of the happy path in the initial block is very helpful for understanding the flow of the feature, once you grok the syntax.
> I do wish migrations were just generated from schemas though, a la Django.
This is a weird one to me. I really dislike the magical way Django does this and I'm glad Ecto doesn't. It also allows you to have separate Ecto structs representing different parts of a table in scenarios where that is desirable.
I think its also important for people to realize that
1. its opt-in, if you don't like it don't use it :)
2. we don't do magic migrations. We generate a regular ecto migration from your resource changes. So you can use it as a starting point and hand edit it (and often you should or even must as we can't do things like generating data migrations).
3. you don't have to map your resources one-to-one with tables. We support multiple resources being sourced from the same table, and turning on and off migration generation per resource. It's very common to have an Ash resource backed by a postgres view, or two ash resources backed by the same table. The migration generator merges resource definitions together to create "the strictest table definition that supports both resources". You can also remap field names. So at the end of the day, you can have your cake and eat it too on that front.
I'm aware of that, and can see why people like it in Ash, but as with Django I think it makes things less explicit, harder to break up domain specific structures backed by the database, and more magical. It's fine IMHO as an add on with something like Ash, but a data mapper like Ecto shouldn't strive to be an ORM like Django especially in a functional language.
I mean that could just be a mix task that you would be free to use or not.
Currently it's true (for me) that writing the migration is copy pasting the fields from the ecto schemas and modifying them because the dsl is "almost the same but not totally".
Agreed, I hate the way Django does it. I don't want my database migrations to be tightly coupled to my models - what's the point?
I'm not sure that the Django approach would make sense in Ecto anyway because Ecto schemas are explicitly _not_ "models". The function of a Django/Rails "model" is divided between different concepts like schemas, Ecto.Query, Ecto.Changeset, the Repo etc.
Same. I was annoyed at first with what felt like duplication (and indeed is duplication for the first time the table is created), but over time the Ecto/Elixir method really showed it's merit and is IMHO the cleary superior way for a maintainable long-term application. The magic and the convenience in the short term it brings are not worth it. The straight forward, explicit and crystal clear declarative way for migrations and the schema are IMHO a joy.
Interestingly, though I am not a great Rust programmer, nor a great Elixir programmer (I have written programs in both languages). And, gotten a little ways through the amazing Exercism (https://exercism.org/tracks/elixir).
What I loved about this writeup was the way I could review both by comparing two languages in which I have a moderate understanding. The comparison of imports was very engaging to my brain.
I really feel like I could accelerate learning a new language if the tutorial was written by comparing that language to another language I already know. Partially I just think that would keep my interest. I'm never excited when I start learning a new language and read the description of how to import a module in that new language; my brain would say "doh, I'll review this when I actually start writing code, let me zone out here..." I feel like most language tutorials feel obligated to cover this minutiae and there is always a lot to review when you are writing a tutorial. People would probably get upset if you skipped that stuff, but for me as a reader, I really prefer this approach here.
You bring up something interesting. Structured learning is very helpful but I sometimes hate that I can't or feel I shouldn't skip around. Videos are the worst because it's very linear. I'd love to be able to see abstract concepts such as the interpreter/compiler for a language and it's syntax and important concepts and models etc the way you can just turn a 3d model around in a CAD system. That would be truly amazing. Let me explore it with my visual cortex, instead of of the need to follow along some story I'm not really interested in.
Edward Tufte advocates handouts over slides in presentations for just this reason. Your audience can refer back to the written and illustrated content of your presentation at their leisure.
FWIW this is the #1 best use case I've found for ChatGPT. I recently started learning golang and as an experiment decided to use nothing but ChatGPT for reference/documentation. So far, ChatGPT has satisfied roughly 100% of my questions. A lot of the questions were like "what's the go equivalent of X in JS/TS?".
The returns seem to be diminishing after ~6 weeks, but the first 2-3 weeks were the least friction I've ever encountered learning basically any new topic. It was kind of wild, I don't remember getting stuck even a single time, it was just super high-density progress, non-stop.
More generally, the situation where "I have a good amount of generic knowledge, so I know what I want, but I don't have the specific knowledge I need" seems to be a sweet spot for ChatGPT.
If you like the presentation format rather than written format, I did what was called by one reviewer "a delightfully thorough" intro to Elixir for Rubyists: https://www.youtube.com/watch?v=uPWMBDTPMkQ
This is why a survey in programming languages is beneficial. You want a variety to fill in the gaps and this increases capacity to learn new ones but you are also easily able to identify short comings and foot guns where people who are siloed to a single language could never be aware of which leads to instances of NIH syndrome like we've seen in Python & Go.
Testing is one of the areas that we have felt the most in Elixir while building Batteries Included. ExUnit is pretty good, but bare bones. That combined with Phoenix (most popular web framework in elixir) made for some places we didn't test. So we created a test library that does polaroid snapshot testing of Phoenix components. We called it Heyya and added other utilities to test phoenix live view too.
Does anyone have solutions for Ecto testing with processes?
If have an `Application` that starts processes that cache ecto state, or periodically write to db, etc, it's hard to test the whole process tree. Additionally Umbrella projects start all applications in the project with one config. All of those combined mean you have to change your code structure quite a lot to make that testing possible.
Testing single processes is pretty easy. Making sure that two processes and ecto can tolerate failures, delays, etc requires too much.
> …though I really have a hard time expressing why.
This one is interesting, I'm new to Elixir an I really enjoy writing tests for my Elixir app, in a way I kind of never have before.
This might just be an effect of the ecosystem being built more with testing in mind than I would have expected.
For instance when I wanted to add email sending to my Phoenix App, the Swoosh library came with a Test adapter and an assertion to use in tests for it.
I would have normally expected to have to write some mock of the email library or use some testing SMTP service, but it was so surprisingly painless instead.
Same with testing functions that interact with the database in Ecto, I feel like the way to do this is well thought out as part of Ecto and I never have to work around any testing-specific issues.
For the email validation I would have used an ecto schema, since most cases you won't just be validating an email address in isolation:
defmodule EmailSchema do
use Ecto.Schema
import Ecto.Changeset
@primary_key false
embedded_schema do
# here is your type validation right off the bat
field :email, :string
end
def validate(email) do
%__MODULE__{}
|> cast(params, [
:email,
])
|> validate_required([
:email,
])
|> validate_change(:email, fn :email, value ->
cond do
not is_email_address?(value) ->
[email: {"invalid email address", [validation: :email]}]
not EmailAddresses.is_available?(value) ->
[email: {"is unavailable", [validation: :email]}]
true ->
[]
end
end)
|> apply_action(:insert)
end
end
case EmailSchema.validate(email) do
{:ok, %{email: email}} ->
{:error, %Ecto.Changeset{} = changeset} ->
changeset.errors[:email]
# Can be all of these in the same list, or be any one depending on the validations
#=> [{"is required", [validation: :required]}]
#=> [{"invalid email address", [validation: :email]}]
#=> [{"is unavailable", [validation: :email]}]
end
For me it would be the Pin-Operator. Which is only needed cause variables can "mutate". IMHO it's not that common that we need to "reassign" variables, we could life without the looks-like-reassignment.
I touched Erlang before, it's hard to get my brain to accept elixir is different in regards of variables :)
Nitpick of the nitpick: name shadowing is one of the most important QOL improvements a functional language can add. The pipe operator can alleviate some of the pain of lack of shadowing, but sometimes you really do want to have a chain of transformations to a value that don't fit well as a pipe, such as when there's more than one intermediate value being used in parallel. In those situations, forcing programmers to give each intermediate value a new name each time is pointless overhead and doesn't actually improve "purity" in any meaningful way.
Genuine question: From an application developer's perspective, what's the difference between mutating a value and transparently rebinding an old name to a new value? Is it just that in the latter case other references don't pick up the changes? So with rebinding we don't have something like
On the contrary. I'd like the pin operator regardless of whether rebinding is allowed or not.
def func() do
s = g()
...several lines of code
l = t()
case l do
{s, q} -> #blah
{q,q} -> #blah
{p, r} -> #blah
end
end
Without the "^s", you need context to determine "s"'s behavior. If s is currently unbound, then it'd be assignment. If it's unbound, it'd be pattern matching. It's a rare enough operation that it's nice to make it explicit.
Erlang didn't have rebinding, and Elixir learned from the mistake. Rebinding is not complicated: the scope rules are simple, you learn them and then you're done. It's Day 1 type stuff, which is absolutely not worth optimizing for if you're trying to build a useful language. Without them, you end up coming up with a bunch of bogus variable names that don't add anything to the conversation. Imagine modifying an entry in a struct (or, more exactly, you're making a copy of a struct with one value changed)
With rebinding, it's easy:
def func(dict) do
dict = dict |> Map.put("apple", 1)
end
Without rebinding, you need a pointless new variable name:
def func(dict) do
dict_with_one_apple = dict |> Map.put("apple", 1)
end
I do agree with the pin-operator, after more thought. I actually think it's good, it's just the rebinding that I don't like.
I understand the argument for rebinding, as an Erlang developer I have to deal with not having it all the time. I guess I don't really see it as that big of a deal to have to use more variable names. The way I see it, if you're transforming something, it's fine for it to have a different name. I haven't used elixir, but I would guess pipelining covers 95% of the time that rebinding would be wanted, and the other 5% of the time I think I would prefer not to have it, but it's really just nitpicking, it's not too big of a deal.
I love how pins make it explicit you are not assigning something without having to look at the context. I prefer it being there even if you couldn't shadow a variable.
Interesting. In our codebase we do this all the time. A quick search revealed 280 occurrences of `some_var = some_var |> ...`.
I also find the pin operator much more readable, as the meaning of `{foo, ^bar} = result` doesn't require to know the context. `foo` is being assigned, `bar` is being matched on. No need to know the code before this line to interpret it.
Yes, good luck writing a non-trivial Phoenix and/or LiveView app without ever writing `socket = something(socket, …)` or `assigns = assign(assigns, …)`.
I do remember finding the pin operator confusing when I first started learning Elixir, probably because I'd never seen anything like it in another language. But the confusion didn't last long; it's really not hard to understand. I've never felt like the pin operator was bad for readability.
It's true there are few in the standard library, but look at 'Enum.reduce_while/3'.
Use of throw/catch for prompt return is possible, but is not considered good style, except perhaps if the recursion is very deep, or through an intermediate library layer that does not support prompt returns.
However, it's trivial to write your own prompt returns. It is a standard idiom (esp. in Erlang) to write a function as:
1. Public head, argument tests in guards, immediately call internal private implementation function, with additional empty accumulator argument.
2. Private function (clause 1), continuation case, does the work, then decides whether to: increment accumulator (often O1 prepending head to linked list) and recurse; or prompt return of some result, like found search, or error when some condition exceeded.
3. Private function (clause 2), termination case, empty arguments, reached the end of inputs, just return accumulator, often Enum.reverse of a list.
For example, specialized version of Enum.reduce_while over a list might be implemented as:
def foo(xs) when is_list(xs), do: do_foo(xs, [])
defp do_foo([x|xs], out) do
case bar(x) do
{:ok, val} -> do_foo(xs, [val|out])
:error -> :error
end
end
defp do_foo([], out), do: {:ok, Enum.reverse(out)}
I think when you grok this simple pattern, you will never use, Enum map, filter, reduce, reduce_while, flat_map, .... ever again, because it can emulate all of them.
I fully agree about how hard it is to understand what the DSL features of elixir are doing. It was a major pain point for me and I wish that Elixir had a mode where it would expand all macros and output the resulting code so you can just see what your code looks like. I think that would help a lot.
On the flip side, most of these complaints feel like the author is fighting the language and wants it to behave more like a language that makes different tradeoffs. Maybe akin to complaining that Rust is annoying because "you need to use unsafe to code normally."
For example pairing with statements with specific atoms is specifically called out as an anti-pattern in the latest elixir docs[1] (this is an example of the elixir devs trying to talk more about how best to do things!). I don't mean say that the author "should have known not to do that" - I use it from time to time. I mean that the language is not setup to support this code structure well. The language wants you do use different structures for doing a series of things (the Ecto approach of returning a structure with the error embedded in it is an example).
I'm also sympathetic about the confusion around umbrella projects (and other such features) but I kind of feel like the current state is the optimal one? There's a tutorial showing you when you might want an umbrella project, but also showing you a similar structure that avoids using one[2]. On some level choosing which approach you want depends on a ton of details about "how elixir / erlang works"...but that seems pretty unavoidable? I think umbrella projects make very little sense until you grok the application system - but ultimately you can't save people from skimming over why they might use something and diving into learning it. I think this is just an inevitable drawback of languages that have fundamentally different tradeoffs than ones that generate native code with mutability.
I enjoyed reading the previous articles, so I was excited to read this one too. I really appreciate this style of feedback. Comments below.
-------
ERROR HANDLING
In my opinion, the "with {:is_email, true}" style is missing the forest for the trees. The whole point of "with" is to match on a consistent result. If you need to distinguish individual clauses, then you should either use case/cond or normalize the result types, in the same way you would do in Rust. So in your case I'd add two functions: "validate_email" and "check_email_availability" that returns ":ok" or "{:error, reason}". Then you end-up with:
with :ok <- validate_email(email),
:ok <- check_email_availability(email) do
...
end
> Sure the state is all encapsulated into processes, but then those processes are hidden behind an abstraction layer that makes them invisible, so really you’re just touching global variables.
They are not invisible. You can use Observer, the Phoenix Live Dashboard, and many other tools to traverse, explore, and navigate the supervision tree, processes, and see where the state is!
-------
IMPORTS
Agreed. We had several discussions on how to improve this but nothing satisfactory. Maybe it is time for another tango.
-------
MIXED MESSAGES
I'd say we actually do a good job on the official docs on the topics that are directly related to Elixir:
The trouble is in finding this information, as it can be a lot to absorb. If anyone finds we should link to them from other places, pull requests are welcome. In general, PRs to improve docs are always gladly received, be in Elixir, Ecto, or elsewhere!
-------
OTHERS
> Anecdotally, when Elixir started off there was some bad blood between them and the Erlang community, which is the origin of this schism
No bad blood, really. I asked the Rebar team (not the current Rebar3 team) if they would accept PRs to also compile Elixir, they said no (which is understandable) and then we move forward with Mix (which was a contribution from a Clojure developer inspired by Lein). The projects drifted apart but we often share whatever we can in other places (such as https://github.com/hexpm/hex_core).
> In fact the Elixir compiler almost never gives you an outright error, basically it only fails if a file can’t be parsed. This feels spooky as hell… but its warnings are basically always correct and seldom miss anything
Yes! Our goal is to avoid halting compilation as much as possible and instead rely on precise warnings. It is easier to debug a program that compiles (and then raises) than one that does not compile at all.
If you ever get to what is bothering you on unit tests, I'd love to hear (feel free to reach out).
> Lots of other tools are a bit short of the “first-class” level of polish; Image/Vix
I'm the author of Image and I'd welcome feedback on improving areas where you see lack of polish (there's a reason its not yet 1.0, but it is getting closer - primarily rewriting the color model).
> Rust-style returned tuples of {:ok, val} or {:error, e}.
Tuples fall under product types. Rust's APIs return errors as sum types. A more apt comparison would be with Go, where errors are actually returned in tuples.
Glad I'm not the only one who doesn't get Ecto.Multi. I can see how it's useful in theory in some cases, but I've always found it much easier just to use Repo.transaction.
with true <- is_email_address?(email),
true <- EmailAddresses.is_available(email) do
then I have to go line by line replacing = with <- and adding a comma at end of line. Sometimes there is a long list of lines to wrap. Surely it can be automated by an IDE or a macro of the editor. Anyway it's still a bad choice IMHO.
I understand why with could not work with the = matching operator (it's a macro) but that's about language design, ergonomics and in part not making developers do the job of the compiler.
`=` is useful inside `with` if you have a clause that _must_ return a particular value or it's an error, i.e. you don't need to match on any other possible return value. But if you don't have at least one clause with `<-` then `with` achieves nothing.
Also, why would the fact that `with` is a macro mean that it can't use `=`?