Allow me to include my mandatory comment about how other languages need to (re)discover Common Lisp's condition and restart system.

The condition system from Lisp gives handlers the option of running before the stack is rewound; other languages put various amounts of state in their exception objects, which only gives you a portion of the power of just not destroying that state in the first place.

Restarts let callers communicate down the stack what to do when certain conditions happen, which removes the need for second-guessing and allows for better locality of code; having the "When X, Y" closer to the code that is responsible for the decision rather than the code that just happens to be closer to X is almost always a win.

Running various callbacks or whatever is still overthinking the problem, similar to throwing exceptions. There's still a preconceived notion that anybody would want to (or could) handle a certain state right then and there.

In general, some responsible code does something about the state that is created by the library. But also, in general it isn't the code that issued the request, nor should happen right when the library detects the error.

In theory, such callbacks are at least as powerful as only storing the state so the client can inspect it later - because a trivial callback is free to only store the state so the client can inspect it later. However, in practice, callbacks lead to overly complex code (beyond some trivial lambdas), and having to provide a callback encourages bad program structure - creating a temporal coupling between the occurence of some event and the handling of it

> Running various callbacks or whatever is still overthinking the problem, similar to throwing exceptions. There's still a preconceived notion that anybody would want to (or could) handle a certain state right then and there.

Sometimes you want to and can, other times you don't. Sometimes you want to log some state that will be destroyed when you unwind the stack. Not being allowed to handle an exception before unwinding the stack is a hindrance.

The common case is that you will unwind the stack; lisp even has a form that does so (handler-case vs handler-bind) because it is so common.

"Where" an exception is handled is two dimensional. There is "where in the source" and "when at runtime" Most exception systems unnecessarily couple the two. If you make an API call that you know will cause a network request somewhere downstack, and at that point in the code instruct it to retry up to 3 times when a connection-refused exception occurs, then you have a usable exception system.

If you can't do that, then just give up and return errors.

There are valid usecases for libraries to take callbacks to parameterize stuff - namely, when the library needs something from the client that the client should always be able to provide synchronously, and it would be too much work to create an interface to let the client pass in the information asynchronously.

Example: memory allocation?

But it doesn't work the other way around: The library shouldn't try to restrict the user about all the things it should want to do. Taking dozens of hooks to cover all possible situations will only make the API harder and harder to use.

You may be misunderstanding. These aren't explicit hooks any more than a thrown exception is. Imagine this pseudo python:

  tryNounwind:
    Foo()
  except SomeException as e
    # This block is run as soon as SomeException is raised, before unwinding the stack

There is still an equivalent to try that doesn't unwind the stack. One major use for this is to call dynamic restart points that intermediate functions have provided. A function implementing an RPC shouldn't have to decide how to handle e.g. a network error, but there are several reasonable things that it could do (retry, fallback to a different host, &c.), it can provide those hooks while defaulting to propagating the error up, and since the caller of the function can run its exception handler without unwinding the state, those hooks are accessible to it.

It also dovetails nicely with a debugger, because the restarts can be invoked interactively.

No no, I understand. It is callbacks. Or some fancy syntax that creates and registers a closure implicitly. But technically it's just a callback - the library calls back into the user code, synchronously. The syntax was never the problem, it's more like this syntax sugar (or in the case of LISP, let's call it semantic sugar) is making it way too easy to do something complicated like this. My strong opinion is that this is not a good idea to do.

RPC is bad, it's an extremely leaky abstraction (network requests pretending to be synchronous function calls). Maybe that's ok for scripts, or in a compute cluster that can guarantee high availability and low latency, but it's nonsensical for general programming (another reason for slowness and clunkiness of so much software).

> No no, I understand. It is callbacks. Or some fancy syntax that creates and registers a closure implicitly. But technically it's just a callback - the library calls back into the user code, synchronously. The syntax was never the problem, it's more like this syntax sugar (or in the case of LISP, let's call it semantic sugar) is making it way too easy to do something complicated like this. My strong opinion is that this is not a good idea to do.

This is essentially how traditional exceptions work, it's just that it also happens to include a non-local transfer of control first. Every place an exception is raised is an additional surface to the API.

> RPC is bad, it's an extremely leaky abstraction (network requests pretending to be synchronous function calls). Maybe that's ok for scripts, or in a compute cluster that can guarantee high availability and low latency, but it's nonsensical for general programming (another reason for slowness and clunkiness of so much software).

Perhaps this is overfocusing on my example? File system access, database requests, memory allocation, are other examples of things that can fail that have multiple plausible ways of handling the failure, and separating the implementation of handing the failure from the choice of how to handle the failure can be useful.

[edit]

I mostly agree with you on RPC; I picked that example specifically because it has the most non-local exceptional situations of things I could think of on the spot.

> Libraries that are designed with exception systems are in the business of second-guessing what's an error for the caller

Okay, let's take some concrete example: a library that does a network call in order to accomplish it task. Imagine it sends/receives all relevant data, and then the TCP connection is closed from the remote end with RST instead of FIN: for the library it looks like a RemoteHostClosedError being thrown. What next?

Some libraries would second-guess the caller that it's not an error, swallow this exception and return the data as normal — that's bad, we don't like those libraries.

Some libraries second-guess the caller that it is an error and rethrow it, throwing away the would-be returned data — that's bad, we don't like those libraries.

A good library would instead not second-guess the caller and do... what, exactly? Return both the resulting data and the exception? But that's the case when the error happened on the wind-down, what if it happened in the middle: should the library return some sort of "resumable context" together with an error? Or what?

So, the library received some data. It also received an RST. It should store both pieces of information in the connection handle (library data structure).

The caller can then inspect the state and decide what to do.

There are other options. The library could for example throw an exception and never report the received data. It's a different paradigm - making choices to present a convenient API that works in the common cases, at the cost of taking away control over the less common ones. APIs like this are popular and maybe useful for scripting languages, but probably less useful in larger systems.