Hacker News new | past | comments | ask | show | jobs | submit login

"I want a defined point at which the system is in a known good state or fail in some detectable way."

I think we're (as an industry) getting to scale and complexity of systems that warrants systems to heal themselves for a range of predictable and well-understood failure modes, in a way that doesn't require my manual interference.




Absolutely! But that doesn't need to be the app's job. It's the role of an orthogonal process, the monitoring or init daemon, to say "Hey, this process bailed. I should restart it."

That way all your app has to worry about is "Every time I get started, I should try and connect to my dependent services. If it doesn't work, bail."

And the monitoring process gets to worry about things like "I should retry X times before giving up. There need to be at least Y instances of this process running."


Agreed, but I don't think we can do that without having transactional boundaries. When a transaction fails, it doesn't necessarily mean that a human has to intervene. It just means that we have a reduced set of possible states that are known to be consistent. I don't see how we could ever hope to define correct self healing algorithms without reducing the number of possible states a system can be in.


Yes. Consistency guarantees in the face of distributed failure is a popular topic. (CAP theorem, etc). The whole "cloud native" (12 factor apps, microservices, immutable or disposable infrastructure) movement is also trying to describe ways to simplify most codebases so that you keep as much of the system as stateless/ephemeral as possible.

What is interesting is that , for the stateful/persistent data processing, most large scale systems are rejecting transaction boundaries as we know them (fully serializable isolation and consistently) for relaxed consistency. There are some good articles and papers on how programming needs to change to enable better self healing / "recoverable to a known state" behaviour, such as CRDTs.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: