In my experience, macros in lisp codebases are mainly abstracting away common boilerplate patterns. Common Lisp has WITH-OPEN-FILE to open a file and make sure it closes if the stack unwinds. This is a macro based on the UNWIND-PROTECT primitive, which ensures execution of a form if the stack unwinds. Many projects will have various with-foo macros to express this pattern for arbitray things that are useful in the context of that project(though not all with- macros need UNWIND-PROTECT). Another example is custom loops that happen over and over.

Let's say I'm writing a chess engine. I regularly want to iterate over the legal moves in a position. So I might make a macro so I can write (do-moves (mv pos) ...)

I find that because doing this is so simple, well written lisp codebases tend to be pretty easy to read. There's less of that learning curve in a new codebase of getting used to the codebase's particular boilerplate rituals, and learning to pick out the code that's interesting. In a good lisp codebase all the ritual is hidden away in self-documenting macros.

Of course this can get taken too far and then it becomes nightmarish to understand the codebase because so much stuff is in various complex macros that it's hard to tell where the shit goes down.

What I don't understand is where the advantage of macros over functions lies in your example. In essence, what can't I do with the following?

    var position = new Position();

    var newPositions = GetLegalMoves(position)
        .Select(move => position.Clone().Perform(move));

Moreover, besides this "sending remote commands example" portrayed in the article, when would I ever want to pass around unevaluated code?

Perhaps I should pick up lisp.

I'm not a JS programmer, but I didn't find that code terribly readable. And from a performance point of view, I would not want to copy the board state n times, though that's beside the point. It's a bit of a contrived example of course. But what this looks like in a real engine would be something like(no language in particular):

  pseudo_legal = move_generator(pos, PSEUDOLEGAL);
  while true {
    mv = pseudo_legal.next();

    if !mv {
      break;
    }
  
    if mv == tt.excluded_move || !pos.legal(mv){
      continue;
    }

    pos.make_move(mv);
    ...
    pos.unmake_move();
  }

The pseudolegal stuff and all that has to do with various performance considerations. This is roughly what Stockfish' move loop looks like. It looks overly verbose because abstracting away these things with functions adds runtime overhead.

With macros you can have your cake and eat it too. While it's technically true you're passing around unevaluated code, none of this is happening at runtime but at compile-time(technically macro-expansion time but that's beyond the scope of this comment). Think of it as the ability to inline pretty much anything you want.

In this particular example, there's little to none: truth be told, I don't like WITH-FOO as a good example of why macros are fun. That's because WITH-FOO macros are commonly implemented/implementable as simple wrappers around CALL-WITH-FOO functions.

For instance, if we had a CALL-WITH-OPEN-FILE function (it's not too hard to write one yourself!), the following two syntaxes - one macro, one functional - would be equivalent:

    (with-open-file (stream #p"/tmp/foo" :element-type 'character)
      (read stream))
    
    (call-with-open-file 
     #p"/tmp/foo" (lambda (stream) (read stream))
     :element-type 'character)

Notice that both of these accept a pathname, additional arguments passed to OPEN, and code to be called with the opened stream. The only differences are that the functional variant needs to have its code forms wrapped in an anonymous function and that the function object passed to it is replaceable (since it's just a value).

---------------

For a more serious example, try thinking of how to implement something like CL:LOOP (a complex iteration construct) without a macro system.

Sure, LOOP is a very complex macro. But my point was that most macros in real codebases are these simple boilerplate wrappers that help readability.

I don't like codebases too full of DSLs, necessarily.

A less trivial example is defining different types of subroutine. In StumpWM, a tiling WM written in common lisp, there is the concept of commands. They're functions, but they executed as a string. "command arg1 arg2". And these strings can be bound to keys. But args might be numbers, windows, frames, strings etc.

Commands are defined through a defcommand macro. It takes types! And there's a macro for defining arbitrary types and how to parse them from a string. A command is actually a function under the hood, with a bunch of boilerplate to: parse the arguments, stick the name in a hash table, call pre- and post-command hooks, set some dynamic bindings. and so on. Defcommand abstracts this away and you can just write it just like a normal Lisp function except for the types.

These don't look like functions to me:

   var
   new
   .
   =>

If any of these were removed, how would you code them back in?

Well there's an old debate about macros and functions, especially when you have closures, since closures can also delay assembly of bits of logic (half the work macros do, the rest being actual syntactic processing when people do that).

You have to understand too that mainstream languages didn't have closure until very recently, so a lot of things look less obvious now.

I've been using Java since version 1.1 and I've seen features added that required a new version of the language spec to go through committee and get implemented before we got to use them where you could just add these features yourself at the top of any Lisp file and then immediately start using them. For example consider the try-with-resources [0] syntax sugar. So instead of thinking "I never actually ran into a need for them", think of new language features that you have started using: those are the kinds of things you could've added yourself.

Also look at any kind of code-gen tooling like parser generators or record specifications like Protocol Buffers as examples of what you could do within the language.

[0] https://docs.oracle.com/javase/tutorial/essential/exceptions...

> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? On occasion I have had to write DSLs for the user input, but in these cases the (non-programmer) users didn't want to write Lisp so I used something like Haskell's parsec to parse the data.

If you're talking about Haskell, you should be talking about folk who write template Haskell, which is the macro system for GHC. There are plenty of Haskell programmers who know how to write Template Haskell, and there are plenty of Haskell programmers who don't. By contrast, I don't think there's a single Lisp programmer who can't write Lisp macros.

That's homoiconicity. Once you learn Lisp, you automatically know how to write Lisp macros. Once you learn Haskell (or Ocaml or Rust), you don't automatically know how to write macros in that language (and the macro system may not even be portable across compilers).

> This is useful for compiler programmers

But that's incredibly powerful.

Now, stuff that would have to be implemented in the compiler to update the language, can now be written just as a "normal" program, that adds whole new features to your programming language.

For example, the entire object system in Common Lisp was implemented as macros.

Yes, most programming tasks don't require this kind of power. But it does mean programming in Lisp it's very very rare you are going to be stuck because your programming language doesn't implement some feature you need for the task at hand.

> For example, the entire object system in Common Lisp was implemented as macros.

It wasn't. The original Object System implementation of Common Lisp is has tree layers: at the bottom layer it is object-oriented (especially the Meta-Object Protocol), then there is a functional layer and on top there are macros for the convenience of the user.

I actively want people to NOT change constant to c. I want the language to be a predictable shared base, not something I have to relearn and customize for every project.

I'm not a language designer. That's hard. That takes time and effort. And I don't do anything that can't be done in Python or Dart or whatever. Customizing the language is time not spent on the project, and batteries included stuff already has everything I need.

I think lisp is good for people who "think inside their heads", as in, they think "I want to do this, oh, I could do it this way, then I'd need this resource, let's build it".

If you think "Interactively" as in "I want to do this, what does Google tell me others are doing, oh yeah, this was made almost exactly the same use case, I'll start with this resource, now I'll adapt my design to fit it", you might not have any ideas for language tweaks to make in the first place.

I basically never code and think "I wish I could do that in this language" aside from minor syntactic sugar and library features. New abstractions and ideas don't just pop up in my mind, what the common mainstream languages have is the entirity of what programs are made of, as far as I'm concerned.

I don't entirely disagree that _changing_ the language is a bit of a no no, like changing the behavior of existing keywords and operators.

However, any program that declares a variable or new function could be said to extend the language, since, if you declare some function, well, that function is now, at least in any proper language, as much part of the language of that program, as any builtin function is.

Sure, if all you have is an empty .c file, you can say "this program is standard C", but as soon as you've declared a variable or funciton, your program becomes in a way a superset of C, it is all the standard C plus the functions, datastructures and variables that you've defined, and to extend that program, it is not enough to keep strictly to the standard language, you must also take into consideration the superset of functionality that is part of the program..

In this way, programming is much more about creating a language that speaks natively in the abstractions of the domain, and then using that language to solve specific tasks within the domain.

And so it becomes that, you're always tweaking the language, it's just the degree to which you can tweak it that is different..

> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it?

It is also useful for anyone wanting to implement language-level features as simple libraries. Someone else brought up Nim here: it's a great example of what can be done with metaprogramming (and in a non-Lisp language) as it intentionally sticks to a small-but-extendable-core design.

There's macro-based libraries that implement the following, with all the elegance of a compiler feature: traits, interfaces, classes, typeclasses, contracts, Result types, HTML (and other) DSLs, syntax sugar for a variety of things (notably anonymous functions `=>` and Option types `?`), pattern matching (now in the compiler), method cascading, async/await, and more that I'm forgetting.

https://github.com/ringabout/awesome-nim#language-features

And anyway modern languages have proven that you don't need S-Exps for powerful hygenic macros.

E.g. sees Nim's take on it: https://nim-lang.org/docs/tut3.html

Dylan also did this, in the 90s, that's not a new idea. Who has been saying you need s-exprs for hygienic macros?

All the lisp advocates, generally.

Some Lisp advocates may tell you that Lisp even does not have hygienic macros. Lisp dialects like Scheme have. Lisp usually has procedural macros, which transform source code (in the form of nested lists), not ASTs (like Nim).

That Nim has 'powerful hygienic macros' is fine, many languages have powerful macros.

As part of delivering an e-commerce solution in Scheme, I wrote a module which allowed for SQL queries to be written in s-expression syntax. You could unquote Scheme variables or expressions into the queries and it would do the right thing wrt quoting, so no connection.prepareStatement("INSERT INTO CUSTOMERS (NAME, ADDRESS) VALUES (?, ?)", a, b) type madness. Wanky, you bet. But oh so, so convenient.

It's not tied to complex DSLs and compilation. I was frustrated by the lack of f-strings in emacs lisp so I hacked a macro, now I have embedded scoped string interpolation. I can write (f "<{class} {fields}>"). Having these freedom is really not a niche, it's mentally and pragmatically important.

I remember (faintly, from 20 years ago) using Lisp in game programming to create rules for mobs, from within the game.