Book of the month: Inviting Disaster (Lessons from the edge of technology) by James R Chiles

"Inviting Disaster" has a number of praise quotes on the first page; the quote that made this book seem relevant to a simulation and human factors audience was:

"The lessons are clear and disconcerting: a sequence of minor human errors when combined with elementary technical problems can lead to unimaginable catastrophe."

From a human factors perspective the book gets off to a bad start but redeems itself in later chapters. The 'Special Introduction' to the paperback edition deals with the 9/11 terrorist attacks on the Twin Towers in New York.  It diverges from the rest of the book by not focusing on human error and complexity. Chiles could have discussed the attackers' movements and training in the US and considered the lack of communication between federal agencies (human errors) and the difficulty of tracking thousands of potential suspects (complexity). Instead this special introduction looks at the towers' design, construction and their evacuation. With regard to these factors there really wasn't a sequence of minor errors but instead perhaps a lack of comprehension that airplanes might be used as terrorist weapons.

After the disappointment of the special introduction, the book begins to redeem itself with the first chapter. Chiles describes the sinking of the Ocean Ranger oil rig off the Canadian coast in 1982. The specifics are unfamiliar but the types of error lining up to cause the sinking are familiar ones. They include:

1. Equipment design




1. The eight legs which support the oil platform are so big that the designers decided to use the space inside them for storage and other functions, such as the ballast control room (see below). Unfortunately the legs are also closer to the water than the platform, which means they are more likely to be hit by a wave. Also, because the legs are being used for storage, the space inside them could flood during a storm. There is a particular risk of flooding of the four corner legs because they have five-foot holes in them for bringing wire ropes and chains onboard. The conditions required for water to enter these holes (severe tilting of the platform in bad weather) are considered an 'impossibility'. Additionally there is no alarm to warn the crew that water is entering the rig legs.
2. The ballast control room allows the operator to control the pumps which move the seawater ballast between the ballast tanks and the ocean. There are 16 tanks but only three pumps and this means that valves have to be set to certain positions in order to make sure the correct pump is connected to the correct tank. Under crisis conditions it might prove difficult to work out which pump was working which tank.
3. The ballast control panel is not protected against seawater. There is a cutoff switch to stop electrical short-circuits from opening and closing the valves automatically, however this switch also disables all the electrical displays. Once the cutoff switch is active there is no way of knowing the valve positions, pump state or depth of ballast within the tanks.
2. Training


1. In order to qualify for the job of ballast control operator, a worker had to spend "several hours each day hanging about the control room and watching over the ballast operator's shoulder"(p.27). There had been a training programme for new operators but this had ended some time previously. Lack of training means that the ballast control operators do not know how the pumps work, or that the pumps will not work if the tank to be pumped is much lower than the pumping chamber (as occurs when the rig is listing.)
2. There was no training for evacuation during a storm. Evacuation drills would occasionally consist of counting the people on the rig, at other times they would lower the lifeboats in calm conditions.
3. Human error
1. The glass portholes to the ballast control room can be protected by steel storm covers. These were not put in place before the storm
2. The most knowledgeable ballast control operator on the rig found out about an emergency workaround which would allow the pump valves to be controlled without electrical power.  This undocumented and untested emergency procedure would allow the control operator to open specific valves. Unfortunately the ballast control operator thought that the procedure would close them instead.

During the storm on the 14th February 1982 a powerful wave smashes one of the ballast control room portholes and seawater splashes onto the control panel. Because the equipment is short-circuiting the cutoff switch is activated. Although not ideal, the rig is in a safe, 'holding' position. The valves to the tanks are all closed, the anchors holding the rig are secure and the rig is straight in the water. In this condition, the rig would not have sunk.

Unfortunately the crew decide at some stage to reconnect the power to the control panel. Electrical short-circuits and possibly trial and error attempts by the crew allow water to enter into the bow section via open valves. The emergency workaround is used at some stage which leads to additional valves opening. Eventually the rig is tilting to such an extent that the 'impossible' event of water entering the five-foot holes in the corner leg on the port bow occurs. As more water enters the rig, the listing worsens until even small waves can reach the hole.

At 01:05 AM the rig's support vessel is radioed to approach the rig which is now being abandoned. Unfortunately the 80mph winds cause the lifeboats to smash against the side of the rig and crack open. Additionally, once the lifeboats reach the water,  the ropes securing them to the rig can only be released when there is no tension on the rope. This means that the lifeboats cannot get clear of the rig and continue to smash into it. When the support vessel reaches the rig at 02:00AM only one lifeboat is still afloat. The eight men inside must have thought they were saved. Unfortunately as they try to climb onto the supply ship's deck, they tumble from the pitching lifeboat and are swept away. The support vessel has no gear to drag the men onboard and the men are too immobilised by cold to climb into the support vessel's life rafts. A total of 84 people die.

The subsequent chapters of Chiles' book detail events which are varied in time, place and industry, but similar in causation. They include the shutting down of the wrong engine on a Boeing 737, the Three Mile Island nuclear accident, the R.101 airship, the Challenger disaster and more. The in depth analyses of the various incidents are one of the book's great assets but also its major weakness. At times the description of the incident is so long that the reason for it being mentioned in the first place is forgotten. There is a lack of a logical progression from chapter to chapter so that, at times, the story seems to be drifting from one disaster to the next without an underlying structure.

However, Chiles does offer some suggestions for reducing human error. In a reference to unknown unknowns he states:

"When meeting a new system, people need time to know its workings under good conditions and bad. The most dangerous time is when the operators don't know what they don't know" (p.39)

Chiles also describes the benefits of a "fresh pair of eyes" in relation to the Three Mile Island accident (p.61) and the importance of situational awareness and communication.

Most importantly Chiles mentions the characteristics of a high-reliability organisation (HRO) (p.62):

• A priority on safety from top to bottom
• Deep redundancy so the inevitable errors or malfunctions are caught in time
• A structure that allows key decisions at all levels
• A premium on learning lessons from trials and errors 
• Workers who keep their skills sharp with practice and emergency drills

It is in relation to this last bullet point where simulation can help to build a HRO. Simulation allows repeated, coached practice of routine and emergency drills.

In summary, read this book for its interesting stories of disasters and easy prose, but if you are looking for an in-depth discussion of human factors or human error then skip this one.

Human error, boats and tigers.

In the Man Booker Prize-winning "Life of Pi", Yann Martel tells the story of a boy who survives a shipwreck to find himself sharing a boat with the only other survivor, a tiger. This blog is not about that book.

However it is about a boat, a tiger and... human factors.

When the Costa Concordia  ran aground on the 13th January 2012 off the coast of Italy 32 people died. The waters were well-charted and calm. The weather was clear. One had to wonder how this cruiseship with over 4000 passengers and crew managed to hit rocks and become partially submerged. The company which owned the cruise ship issued this statement on the 15th January:

"It seems that the commander made errors of judgement that had serious consequences: the route followed by the ship turned out to be too close to the coast, and it seems that his decision in handling the emergency didn't follow Costa Cruises' procedures which are in line, and in some cases, go beyond, international standards."

The CEO of Costa Cruises on the 16th January told the world media that the captain made an "unapproved, unauthorised" deviation in course. Lloyd's List published an illustration, widely distributed, which showed how far off the "correct" course the ship had been.

The facts seem very clear: The captain of the Costa Concordia deviated from company protocol and somehow managed to ram his ship into rocks. However a later illustration, also from Lloyd's List, starts to raise more questions. This illustration shows that on a previous voyage the Costa Concordia actually deviated further from its route than on the day it sank.

According to this BBC review of the incident, the captain committed a number of errors. However there are other factors to consider:

• Why did the helmsman not question the captain's decision to come so close to shore? What are the power gradients on board large cruise ships? Can any member of the team voice a concern and expect it to be listened to?
• What are the cruise owner's protocols for dealing with violations? According to Richard Meade, the editor of Lloyd's List, the owner would have known about the route taken in August.

Leaving the boat behind we turn to the tiger and the tragic case of Sarah McClay. Ms McClay was a zoo keeper who died on the 24th May this year after being mauled by a tiger. A statement on the zoo's Facebook page on the 25th May explained:

"...from the investigations that have taken place it is clear that this tragedy was caused by a sad error of judgement and breach of protocols, in essence keeper error. This is not blame, it is not anything but defining the facts as they appear. This does not mean Sarah killed herself on purpose it means simply she died from her own tragic mistake."

The BBC on the 25th May reported the zoo owner, David Gill, as saying:

"We have very strict protocols and procedures for working with big cats, but it seems she failed to follow correct procedures. For inexplicable reasons she opened a door and walked into the enclosure. We will never know why she entered without telling anyone. There was no reason for her to go in there."

And on the same webpage Mr Gill is quoted as saying: "It would not do any good to close the park as there is no safety issue."

However, the most recent news report now suggests that Sarah McClay did not walk into the enclosure. The tiger dragged her into it from a staff area, to which the tigers are not supposed to be able to gain access.

The parallels with the Costa Concordia incident are clear. From an initial blaming of the individual the widening investigation starts to show up human and technical errors. As part of an error chain these incidents lead to a sequence of events which result in the loss of life. In the end, whether it's boats, tigers, or healthcare, we need to move beyond the scapegoating of individuals. In a complex system, we need to focus instead on the series of mishaps which culminate in a catastrophic error.

Updates:

On the 11th February 2015 Costa Concordia captain Francesco Schettino was sentenced to 16 years in jail for multiple manslaughter. According to the maritime trade union Nautilus "There has been an absence of meaningful action to improve safety in response to the Costa Concordia accident, and this trial has simply served as a distraction from the important underlying issues." The ship's operator, Costa Cruises, paid a 1-million euro fine in April 2013 to settle potential criminal charges and in the February 2015 court case the company and Schettino were jointly ordered to pay £22,000 to each passenger.

On the 18th September 2014 an inquest into Sarah McClay's death concluded that the tiger was able to reach her through a door which should have been locked with the possibility that the door was faulty.

Decisions, decisions, decisions...

Have I got time for a cup of tea before work?

Do I need a haircut?
Left sock then right sock or right sock then left sock?

The majority of the decisions we make are trivial, having only a small effect on our lives or the lives of others. However, at times we make decisions which do affect others.

Will I drive home after a few beers?
Am I alright to have the last cigarette of the day in bed?
Should I alert somebody to that suspect package?

Although these decisions may have greater and graver consequences, the answers are still straightforward (No. No. Yes.) The decisions that we see being made in the simulation courses are of a different type. They are often time-sensitive, the consequences are poorly understood and the number of possible decisions can be colossal.

In her book "Sitting in the hot seat", Rhona Flin describes the traditional decision making (TDM) process (p.141):

1. Identify the problem
2. Generate a set of options for solving the problem
3. Evaluate these options concurrently using one of a number of strategies
4. Choose and implement the preferred option

Unfortunately TDM is not applicable in the pressured, time-critical, life-threatening environment of a healthcare crisis. It is here that naturalistic decision making (NDM) occurs. Although there are a number of NDM models, they are similar to the extent that there is no step-wise progression from problem to solution. Instead problem-identification, option-appraisal and problem-solving occur almost concurrently and cyclically.

The principal NDM model is recognition-primed decision making (RPD). In RPD the decision-maker is thought to try and recognise the situation, choose the correct response and then implement this response. Flin (p.145) describes the 3 variants of RPD although Wikipedia does a better job of making them easy to remember:

1. If... then... : The standard response as detailed above
2. If??? then... : The situation is unclear/not recognised. The decision-maker knows a number of correct responses but does not know which one to use. The decision-maker must gather further information.
3. If... then???: The situation is clear but a lack of knowledge means the decision-maker does not know which response to choose. The decision-maker must mentally simulate the consquences of a given response and consider whether these are acceptable. If they are not then the decision-maker must choose more responses until he/she finds one with the desired outcome.

RPD relates back to the post about experts and their unknown knowns. Experts, because of their time spent doing the job, are more likely to recognise the situation and are more likely to know what the correct response to a given situation is.

Flin (p.147) tells us that the key features of the RPD model include a "focus on situation assessment" and an "aim to satisfice, not optimize". "Satisfice" may not be a word you see every day but it relays the need to look for a solution that is "good enough" not "brilliant/amazing/superb". And Flin's placement of the need for situation assessment as a key feature is spot on. Without appropriate situation assessment all subsequent decisions will be prone to failure.

The benefit of simulation-based education is that everybody can review the decisions they made by using video-assisted debrief. Alternatives can be discussed and, because the simulator can be "reset", those alternatives can be played out to see what the consequences are on outcome. The goal is to make all of us better decision-makers when time is short and the stakes are high.

Naturalistic decision-making
8 characteristics of NDM: p.52 of stress and human performance:

A NDM environment

1. ill-structured problems
2. Uncertain and dynamic environments
3. Shifting, ill-defined, or competing goals
4. Action/feedback loops
5. Time stress
6. High stakes
7. Multiple players
8. Organizational goals and norms

   

   Time for some NDM

Book of the month: Sitting in the hot seat by Rhona Flin

Before I begin this review, I have to confess something: Dr Flin is one of the greats in human factors research and my writing a review of one of her books may seem hubristic. I therefore appeal to you (kind reader) to take all comment in the spirit in which it is meant, as a subjective appraisal by a neophyte of an expert's work.

Flin's book focuses on the individual (pilot, police officer, manager) who leads the on-scene response to an emergency. She also provides an overview of how different services train and assess these individuals. At the same time, Flin makes suggestions on how to improve the preparedness of these individuals. Flin's book was published in 1996, almost twenty years ago, and although aviation had been making strides with Cockpit (later Crew) Resource Management (CRM), the lessons from the flightdeck had only been taken up by other professionals in a haphazard manner. 

Flin's first chapter starts with a quote from the Cullen report into the Piper Alpha disaster in which 167 people died:

"The failure of the OIMs [offshore installation managers] to cope with the problems they faced on the night of the disaster clearly demonstrates that conventional selection and training of OIMs is no guarantee of ability to cope if the man himself is not able in the end to take critical decisions and lead those under his command in a time of extreme stress."

It is possible that only a few men died in the initial explosion; the majority of the remaining crew followed standard procedures and gathered in the accommodation block to await further instructions from the OIM, which never came.

Later on in chapter one, Flin provides further quotes from the Cullen report which give us insight into the actions of the OIM:

"The OIM had been gone 'a matter of seconds when he came running back' in what appeared... to be a state of panic..."

"One survivor said that at one stage people were shouting at the OIM and asking what was going on and what procedure to follow. He did not know whether the OIM was in shock or not but he did not seem able to come up with any answer."

The Cullen report led to a focus on safety management systems and on the process of OIM training and selection. The report called for regular training of OIMs in emergency exercises which would allow them to practice decision-making in a stressful environment.

The parallels with my own job as an anaesthetic consultant are clear: 99% of my work is straightforward and routine, 1% of the work is crisis management requiring rapid action to prevent patient harm or death. In 1990 Cullen realised that the only way to make sure that the OIMs were prepared for the 1% was to practice and simulate. Although we have made a start in healthcare and in many respects Anaesthesia is ahead of the game, we still today do not practice for rare events frequently enough.

In the remainder of chapter 1, Flin supplies definitions and provides an overview of the incident command and control procedures in the emergency services, hazardous industries (nuclear, chemical, etc.) and the armed forces. She then goes on to look at the lack of training which contributed to other major disasters such as the Scandinavian Star fire, the Bradford fire, Heysel Stadium, Hillsborough Stadium and others.

Fig 1

Chapter 2 explores selection of incident commanders and Chapter 3 describes (in some detail) the training of incident commanders in different workplaces. Chapter 4 looks at the stress of incident command which I discuss a bit later in this post. Chapter 5 explores command decision making, which deserves a whole post for itself (stay tuned). Chapter 6 looks at incident command teams (including high performance "dream teams") and has an excellent diagram of a model of command team performance (Fig 1). Flin mentions the need for a shared mental model, a term I often use in debriefing but importantly contrasts this with "groupthink", where a team clings on to the wrong mental model. Flin also discusses whether everybody needs to know "the big picture" and this is certainly something that is of interest in the operating theatre. As people arrive intermittently to help out with a critical incident who should update them and does every new team member need to know the whole story? Chapter 7 is entitled "Conclusions and future directions". Here Flin concludes that the best leaders can diagnose a situation, have a range of leadership styles that they can adopt and then match the correct style to the situation.

Fig 2

One critique I have of the book is the perhaps unnecessary complexity of some of the diagrams (Fig 2 shows us, amongst other things, the location of the VGA to composite video encoder in a control room) and the over-description of some of the training installations such as: "This has a central control room, a tactical and action room, which can accommodate 24 observers as well as the rescue leader, and three smaller team rooms." (p.78) Flin also provides a lot of information on individual courses which, although interesting, can at times become overly descriptive.

I think the main lesson I derived from Flin's book concerns the role of stress on incident commanders. In the Piper Alpha explosion mentioned at the beginning of this post, the OIM is in a state of panic, people are shouting at him and he's not responding. The Hillsborough chief superintendent "froze" (p.30). One of the two vital attributes of a leader according to World War II Field Marshall Montgomery is "calmness in crisis" (p.40)  Flin refers to a competence assessment of OIM (p.55) which includes an ability to "Deal with stress in self and others". The best chapter in the book is devoted to "The Stress of Incident Command". However, when I look around today at the various rating tools and marking systems there is neither a mention of "coping with stress" nor is there an approach to exploring stress under pressure during simulation or in "real life". It may be that, as with "communication", it is thought that the ability to cope with stress underlies all the other behaviours that we do assess and talk about (e.g. planning, situational awareness, leadership). Having been involved in a few critical incidents I can easily recall the effect of stress on some of the individuals in the team, to the extent that this was the main driver in terms of loss of communication and prioritisation. I would therefore like to see more of a focus on stress in simulation and the exploration by simulation faculty of ways of dealing with stress by candidates. Flin mentions taking a deep breath as one example. 

To steal another quote that Flin has attributed to Montgomery: "One great problem in peace is to select as leaders men whose brains will remain clear when intensely frightened; the yardstick of 'fear' is absent." It is here where simulation can make a difference; the stressful nature of high-fidelity simulation allows us to assess our candidates' responses and behaviour. Much more importantly, it allows us to coach our candidates and promote self-reflection so that they might improve these same responses and behaviour when disaster threatens in the "real world". Flin provides a quote from Charlton (1992):

"Knowledge of the effects of stress enables the individual to take positive steps to avoid the stressors or to reduce them to limit their impact, thereby defusing a potentially dangerous situation."

However, as I mentioned above, simulation is still under-utilised within healthcare. Let's change that.