* Have the tests all ran and passed? Is CI green?

* In addition to the tests passing, has a third party (not the devs themselves) clicked around and kicked the tires on any new features?

* What is the roll back plan? Can we roll back cleanly?

* What is in this set of changes? Should all of these changes go out?

These checks seem stupid, but the simple act of taking a breath and slowly going down the list really seemed to make a big difference. After instituting the list we had almost no failed deployments.

I don't know if this is an approach that generalizes. I tend to think that for clean, well-maintained and well-tested web applications, a continuous deployment approach is safer and faster in the long run. But when you're working on legacy code, a checklist works wonders.

You can restrict deploys to only CI-checked revisions, you can mandate some sort of functional testing (with the tester stamping the revision), you can check that the rollback procedure works and works cleanly, you can put the changes list in front of somebody and ask that its items be validated.

The field seems to be moving very slightly forward, but it's not always easy to get people to relinquish useless control to automated systems. I'm currently trying to get an integration bot[0] set up in my company and the pushback surprised me. But fundamentally, you can get something similar for deploys, ask for a deploy and it kicks off the checklist automatically, mailing people and updating statuses on dashboards as needed.

And then you can even start collecting stats and charting time to deploy or amounts of rollbacks.

[0] by which I mean something similar to the Rust project's "bors": when changes are proposed, the bot requires validation by a "core developer" (who may be able to self-validate, it's only software so the validation process for a change are flexible), runs the linter, runs the tests, sends the changes to functional testing for validation[1], and merges the changes to "mainline". Humans only get involved where they make sense and can not be automated away: proposing and reviewing (technically and functionally) the changes.

[1] optional, and requires a significant functional validation/testing team, but definitely possible

She's been through 4 years of shit getting a diagnosis to start with thanks to several GPs being disinterested or just incompetent, now a year of immobility after a fucked up surgery and now faces at least 2 years of recovery and rehabilitation.

When it's over we're rolling out the legal eagles but based on past experience I expect to have to do a few weeks of forensics first and possibly get a court order to hand medical records over. Historically we've requested them and they've either been too expensive to prepare (ha) or completely disappeared. It's like chasing cockroaches with a torch.

And this is a minor surgical issue (hernia).

I'm sorry that your wife had poor care - nobody should have to go through that.

But isn't the approach you've described exactly what we want to avoid in a healthcare system? Blame is almost poisonous in its ability to encourage people to cover up the truth, "misplace" documents, make excuses, and so on - this often actively hinders improvement.

Applying the principles of air accident investigation to the NHS is absolutely something I applaud, because the focus should be on why accidents happen, and how they can be prevented — not on who is responsible for them, outside of the most egregious cases of professional failure. Most medical professionals are competent and skilled, and really try rather hard to avoid causing harm to their patients. But mistakes and accidents will always happen regardless of this.

I'm actually very surprised that such a system isn't already in place in the UK. We've already got an independent body which investigates rail accidents, for example - the RAIB looks into all major irregularities or failures on the UK rail network, and rather than finding fault, identifies root causes for failures. That often includes things like the design of systems to reduce the possibility of human error, or even such things as shift patterns which risk causing sleep deprivation. They then issue recommendations for improvement to different bodies as required, and I think they have some legal clout to follow up if they're not implemented.

It seems like a system similar to that would be effective applied to the NHS, but I'm very cautious of heading down the same litigious route as the US.

Whilst the primary purpose was to help us piece together everything that had happened[1] and process that there definitely seemed to be an element of the midwife going through things taking down notes on places where things had gone wrong for later follow up.

[1] After five days of labour, including three extended trips back and forth to the hospital, it got a bit difficult to even work out on what day things happened, never mind what order they happened in.

I don't know if I believe this. I have found many medical professionals (not all of course) to be arrogant and cocky and even if you pointed directly to the cause of the problem, would most likely not think that those rules apply to them.

"not on who is responsible for them"

Mistakes happen because of people. If there isn't some sort of punishment or blame, the majority of people will just continue on with what they are doing (and the mistakes will continue to happen).

We need to blame people, so they change their behavior and have less accidents in the future. We also need to know why it happened.

"Blame is almost poisonous in its ability to encourage people to cover up the truth"

If someone covers up the truth, they should be fired immediately. I honestly don't want someone like this working anywhere near the medical field..and neither should you.

Children cover up their mistakes and lie..adults own up to their problems and admit when they made a mistake.

Also, the reason pilots have less mistakes is because a computer pretty much does 99% of the work. This is not the case with surgery and medicine. Humans still do most of the work.

You: Mistakes happen because of people

Article: When a space shuttle crashes or an oil tanker leaks, our instinct is to look for a single, “root” cause. This often leads us to the operator: the person who triggered the disaster by pulling the wrong lever or entering the wrong line of code. But the operator is at the end of a long chain of decisions, some of them taken that day, some taken long in the past, all contributing to the accident; like achievements, accidents are a team effort.

> the reason pilots have less mistakes is because a computer pretty much does 99% of the work

Except in take-off and landing, which is when most accidents happen. Also, automation would not have helped at all in the United 173 situation - that was clear human error, and the reason those kinds of errors happen less are due to the accident investigation and human resource management techniques this article is talking about:

> To the great credit of the aviation industry, it became one of the most influential disasters in history. Galvanised by it and a handful of other crashes from the same era, the industry transformed its training and safety practices, instituting a set of principles and procedures known as CRM: crew resource management.

You want everyone on your team to be focused on quality outcomes, not on avoiding blame.

When someone is made accountable and the first thing they do is establish a process to avoid blame then you need to ask the question: do these people have my best interests at heart or are they avoiding being accountable?

On many occasions the outcome is to avoid a breach rather than to fix a problem.

The decision makers and the senior staff have considerably smaller shifts. With respect to the surgical and anaesthesiologist staffing, most of them (80%+) are contracted out and are private staff working on contract to the NHS now. A surgeon for example will probably spend two days a week on NHS cases, one day being surgical and the other being consultant and that's all you see of them. The remainder of the time is lucrative private care and occasionally teaching (which is well paid as well).

There are a few exceptions to this i.e. StR's and consultants who seem to migrate into a speciality then leave after a few years. They all take the extra shifts because they're paid for the privilege of up to a 50% bonus. They're overworked because they're chasing the cash.

The REAL overworked staff are mainly the support staff i.e nursing, supply, logistics. Most of them are contracted out now as well in the larger PCTs.

Even with all that, no one should be allowed to cover their arses as it turns into a vicious circle of unaccountability. It's a poor excuse which must not be allowed to continue in the profession (or ANY profession to be honest).

Punishing people for making mistakes might motivate the lazy, but it will not fix systemic issues.

>Also, the reason pilots have less mistakes is because a computer pretty much does 99% of the work. This is not the case with surgery and medicine. Humans still do most of the work.

A number of recent high-profile crashes were blamed, in part, on pilots' manual flying skills getting rusty due to the prevalence of automation. If anything, automation is a leading cause of pilot error.

Surgery is a risky thing, no matter how 'minor' you perceive it to be.

When I did the first poat surgery BP obs run (6 hours later - nurses had skipped her in post op, doctors not seen her either!!!) and found she was 70/40 @ 149bpm with 93% o2 saturation then I assure you its juat negligence. Plus they picked the wrong surgical method and placed the ports too close together so the laparoscopy had no chance of success.

(I had another life before EE and software which was a little closer to the ground on this). I prefer to appear as ignorant as I can tell the difference between facts and fuzz.

The statistics outlined above reflect this, it includes the element labeled as 'human error' (assuming they are up-front about this, which is in a litigious society not usually the case).

A friend of mine almost had his wife bleed to death during delivery, she lost pretty much all her blood and was going out when they located a doctor capable of diagnosing the problem on the spot. He's a pretty tough guy and never at a loss for immediate solutions to whatever problems appear in life. He's never been so helpless. If not for one doctor out of a whole pile of people who happened to be there that seemed to be making things worse rather than better she'd have died. And that's 'routine childbirth' in the hospital (which they push you very much here to avoid, do it at home, so much more fun and natural)... Life is fragile, and as soon as you start cutting things or delivering babies or such then you're at risk. Routine tonsil surgery, hernias, gal bladders etc have left people with serious complications or in the morgue.

I've had more than I would like to do with the insides of hospitals in the last couple of years and I try to avoid having procedures done unless they are absolutely necessary. I don't play the lottery either...

When these processes are not followed and no effort is made to even acknowledge their existence then it is no longer within the realms of human error. It is just negligence.

This is the majority case when someone dies. They literally just left someone or skipped something or overlooked something they do every time successfully.

We expect risk but we expect reasonable mitigation strategies rather than none.

When this happens there is no excuse. None at all. Not a statistic, not a human error allocation, nothing. The same as if I can't be bothered to QA a change, roll it out and it screws up a client's financial data.

You make a change, start running tests to make sure nothing is broken. It's the end of Friday and you don't finish the testing. You close everything down and on Monday you forget you had a last few tests still to complete. You pass the change into your build process and it makes it into a release. You didn't follow the process… not because you maliciously decided to skip testing, it was simply a human mistake (slightly different to your comment, but let's be charitable and assume most people aren't purposefully breaking things).

Does tracing the problem and firing you stop it happening again? Does it even stop you from making the same kind of error again in ten years time when memory has dimmed? In the end you've lost your job and your company has lost someone who probably did good work for many years (and would no doubt continue to do so in the future) over a single easily made mistake (however big the fallout from that mistake). You're also much more likely to try and bury your mistake if you know it will cost you your job.

On the other hand what about adding a step to your build process where a second person runs the testing while they review your code? No one gets sacked, everyone learns something and the chances of this happening to anyone else are massively reduced. Is the person who made an understandable mistake any more to blame than the manager who oversees the process as a whole? They allowed flaws in the process that meant your mistake could get into a release. It’s rarely as simple as one person making a mistake.

I'm simplifying of course, if your build process is anything like ours there are multiple levels this would be caught at... which is kind of the point. In our development process there are four places this error would be caught before it made it to an official release. If our release goes wrong the worst that happens is we annoy the people using the software, yet we've built up a (reasonably) robust process in exactly the way described in the article.

It’s as the article says: When a space shuttle crashes or an oil tanker leaks, our instinct is to look for a single, “root” cause. This often leads us to the operator: the person who triggered the disaster by pulling the wrong lever or entering the wrong line of code. But the operator is at the end of a long chain of decisions, some of them taken that day, some taken long in the past, all contributing to the accident; like achievements, accidents are a team effort.

The top comment there starts like this:

    What I object to here is the core principle that
    there's something to fix in software development
    terms here.

A bit like many of the doctors.

If you look at critical software development they're big on analysis and testing and those kinds of things. Not the same.

Boy, it's just as well human biology is so perfectly deteministic and well-understood it can be trivially mapped out, unlike the vastly more complicated field of software development.

Yes, it would be great to have a better way to identify and tackle common mistakes in software development, but it is not nearly as crucial and universal in software development as it may be in other fields. And, after all, each tool/language is so different that few common denominators exist. In a way, it isn't really an issue — yet people dying from malpractice or negligence (either in aviation, or medicine), is.

You can have the best people in the world, but without a safety culture that abstains from blame and works to eliminate sources of error over time, your hospital will kill people that didn't have to die.

Now, I've seen studies with positive results and neutral results from checklists, but never one with a negative (harmful) result, so on balance the checklist approach seems useful or at least not harmful.

I think the major issue medicine lags here is that airline disasters are highly visible and airlines and pilots are forced to respond, whereas medical disasters are usually seen as 'one off' so even with root cause analysis of failures, injuries sustained by 1 patient periodically do not become as visible; systemic failures are harder to identify

http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_...

Its the result of a multi-institution look at applying design thinking to healthcare, in the context of patient safety and risk management. In my opinion its quite well put together and has great graphics.

http://hci.stanford.edu/research/icogaid/

This might also be interesting for creators of user interfaces for rapid decision making under stress.

For the sake of context, I'm in the UK (so med school is a 5 or 6 year degree, not an undergrad and postgrad like the US), my fiancee is an Obstectric Surgeon (aka an Ob/Gyn) who's half way through specialist training (six years out of med school, three years training in her specialty), my father is an Oncologist (clinical, not surgical, though a lot of interaction with surgeons as would be expected) in his mid-sixties (so med school was forty odd years ago, consultant for over thirty years) - he is, however, both research focused and head of his department so is far more aware of current trends in the NHS than someone who's purely a clinician who wouldn't have much reason to keep up to date with the current literature or thinking in the NHS.

It's interesting to compare and contrast the mindsets and training they both have. A large number of the points raised in the article have been, from what I can tell, incorporated in the training that doctors receive these days - there have been changes to how surgical teams work to help break down barriers between the surgeon(s) and everyone else in the room. In the current NHS thinking the idea of surgeon-as-god is long gone, and even little measures like everyone in the operating theatre introducing themselves before the operation starts go along way to breaking down the formal barriers between the staff.

However, medicine is a very traditional discipline and it can be hard change the way people practice when you're trying to fundamentally change something they've been doing for the last twenty years. From the stories I hear from my father, there are more than a few surgeons of his era who do take the view that they can do no wrong and errors happen as a result. Thankfully, people of that mindset are mostly approaching retirement age and the generation of doctors below them seem to be more open to new ideas - something I suspect is an artefact of being trained during the eighties onwards when lots of fields of medicine became a lot more research focused and less interested in doing things because "that's how it's always been done". Quite likely also due to increased collaboration with colleagues around the globe suddenly being possible so new ideas spread faster.

It's also difficult to talk about the NHS as a whole. Every NHS trust (every geographical area in the UK falls under a particular trust that encompasses one or more hospitals) is slightly different, as there's a degree of autonomy within each trusts. Each deanery (the group of hospitals that a doctor does their specialist training in) is different - the quality of training you'd get in a deanery which is recognised as delivering world class training in your specialty is going to be entirely different than if you end up at a random deanery that has little connection to your specialty. Like any other area of academia, the best people tend to clump in certain places.

This means that, to the detriment of the NHS as a whole, just because some hospitals and trusts change the status quo and improve matters, it doesn't mean that it will spread immediately to everywhere else. It's sad, but perhaps not too surprising when you consider the size of an organisation like the NHS - were you to add them all together, it's an organisation with something in the order of 1.4 million employees, so it perhaps should be a surprise that chance is often slow. Doubly so when you consider that it's a profession that likes to see an idea proved before it's implemented.

On the upside, my fiancee will be spending the next year (from Sept) as an education and simulation fellow where she'll be designing and running simulations for junior doctors. Making sure that they can recognise when the blinders start to slip on and empowering them to speak up when they think that something is going wrong, even if it means telling someone several decades their senior to stop what they're doing, is surely one of the most useful things we can do to ensure that problems like the one in the article are kept to a minimum.

The surgeon I was talking to said that it was probably a good thing when Dunlop died when he did, as there were increasing murmurs in the surgical field about the extremely poor quality of his work, some floating the possibility of him being deregistered. The words my friend used were "his techniques were fine for the jungle, but his skills haven't advanced since then" ('this is the way things are done'). The two examples he gave were that when doing an amputation, he wouldn't leave a flap of skin to cover the stump; and that his sutures were "almost thick enough to be twine", leading to huge, ropey scars where other surgeons were routinely making their work invisible. "Old soldiers love those scars, because they can point to them and say 'Dunlop worked on me', but it's simply bad practise". Basically my surgical friend was saying 'can you imagine being the person who stripped a folk hero of the license to do the exact thing he's a folk hero for doing?'

I'm mostly intrigued that someone who has no background in medicine was able to cut through some bureaucracy and effect major change. It's tragic that something like this had to happen in order for someone to do that, but it makes me wonder how many other tragedies have to happen before something like this changes.

This guy experienced a tragedy. Afterward, he insisted upon an investigation. When he received the results, he saw a very fixable flaw in the NHS, assembled experts on that issue, and managed to kickstart a revolution within the system to address the very fixable flaws that he and his experts had identified.

The Flight 173 case is another example of this. A tragedy happens, experts are assembled, and very fixable flaws are, well, fixed.

Contrast that with other tragedies that occur monthly (if not more often) in America. Mass shootings in Aurora, Santa Barbara, Fort Hood. Investigations occur, and maybe experts are assembled, but nothing changes. What's up with that?

Then we have the other side of the coin. Take 9/11, for example. Investigations were done, experts were assembled, and changes were made. Have they helped? Probably not as much as we'd like.

I'm not really sure what the takeaway is, but I'm very interested in seeing if we can really, truly prevent some of these tragedies from happening with a little foresight, a little communication, and a global recognition that things need to change for the greater good.

    http://en.wikipedia.org/wiki/Peter_Pronovost

The article also mentions the airline industry - specifically the the use of checklists to reduce pilot errors in the new complex bombers of WWII.

A great read: http://www.newyorker.com/reporting/2007/12/10/071210fa_fact_...

It has also been discussed here on HN.

And don't say it's because UK population has different needs, one example, two hospitals at London same medical specialty, they recommend different medication doses for exactly the same treatment in their protocols.

No specialty chiefs, no teams, no supervisors, only solo consultants.

Of course, the other part was the the consultants doing the analysis were quite happy to work with this complexity as it meant more billable time for them - they had no incentives to make anything simpler.

NB I'm not implying that medics should be forced to standardize when it isn't relevant - getting the right level of standardization of processes is why implementing these kinds of systems is so difficult.

http://www.slate.com/blogs/thewrongstuff/2010/06/28/risky_bu...

http://www.amazon.co.uk/Human-Factors-Healthcare-Level-One/d...