Nancy Leveson: Safeware CHAPTER 1: Risk in Modern Society

Nancy Leveson: Safeware CHAPTER 1: Risk in Modern Society • Three Mile Island 1979 • Bhopal, 1984 • Space Shuttle, 1986 • Chernobyl, 1986 • Therac-25 1985 - 1987

IS NEW TECHNOLOGY MAKING OUR WORLD RISKIER? In the United States technological hazards account for 15 to 25 % of human mortality and have significantly surpassed natural hazards in impact, cost and general importance. • [Ref 87 - Ned Franklin. The accident at Chernobyl. • The Chemical Engineer, pages 17-22, November, 1986]. p.4

Flood damage in the United States, for example, has increased as expenditures on flood control have increased. [154 - Trevor Kletz. Myths of the Chemical industry. The Institution of Chemical Engineers, Rugby, Warwickshire, United Kingdom, 1984].p.4

In fact, all hazards are affected by complex interactions among technological, ecological, sociopolitical, and cultural systems [30, 174, 3339]. Attempts to control risk by treating it simply as a technical problem or only as a social issue are doomed to fail or to be less effective than possible.

1.1 Changing Attitudes toward Risk • Not Safe, only Safer. • Societies are recognizing human and workers’ rights • “Workers are at the mercy of their employers in terms of safety” • Complete abdication of personal responsibility, however, is not always wise

In some instances --such as the Bhopal accident -- the public has completely trusted others to plan for and respond effectively to an emergency, with tragic results. • Many aspects of Bhopal guaranteed that an accident would occur. • emergency and evacuation planning, training, and equipment were inadequate. • Public not told of simple measures (such as closing eyes with wet cloth over faces) which could have saved their lives.

In writing about the Bhopal tragedy, Bogard expresses this new attitude: We are not safe from the risks posed by hazardous technologies, and any choice of technology carries with it possible worst case scenarios that must take into account in any implementation decision. The public has the right to know precisely what these worst case scenarios are and participate in all decisions that directly or indirectly affect their future health and well-being.

In many cases, we must accept the fact that the result of employing such criteria may be a decision to forego the implementation of a hazardous technology altogether [30, p. 109]. p.6

Is risk increasing in our modern society as a result of new technological achievements, or are we simply experiencing a new and unjustified form of Luddhism? p.6 • [The Luddite disturbances occurred in Yorkshire between 1811 and 1816, when workers in the English woolen industry tried, through violence, to stem the increasing mechanization of the mills. Luddism has become a generic term describing opposition to technological innovation]

1.2 Is Increased Concern Justified? • Is technological risk increasing? Depends on data used and its interpretation [p.6] • “Harris and colleagues argue that technological hazards, in terms or human mortality, were greater in the earlier, less fully managed stages of industrial development [112] • Data cited from National Safety Council shows that occupational death and injury rates have declined steadily since the early part of this century [112].

NSC Concludes that technological hazard mortality is not currently rising. • However warning that: • “The positive effects of technology have for some time reached their maximum effect on human mortality, while the hazards of technology continue partially unchecked, affecting particularly the chronic causes of death currently account for 85 percent of mortality in the U.S.A. “ [112] • On other hand, examination of technological accident rate, rather than the occupational death and injury rate, suggests that the technological risk is increasing • 60% of all major industrial disasters from 1921 to 1989 occurred after 1975 [30].

Bogard (1989) argued that 12 of the 19 major industrial accidents in the twentieth involving 100 or more deaths occurred after 1950. • If small-scale accidents also included (transportation, dams, structural collapses) evidence is more compelling. • Example: Military Aviation; accident rate has slowed; probably due to emphasis on system safety. • Past experience does not allow us to predict the future when the risk factors in the present and future differ from those in the past. Examining these changes will help us understand the problems we face.

1.3 Unique Risk Factors in Industrialized Society • RISK = Likelihood of an accident plus severity of the potential consequences • Factors include: new hazards, increasing complexity, exposure, energy, automation, centralization, scale, and pace of technological change in systems

1.31 The Appearance of New Hazards • Misuse and overuse of antibiotics have created resistant microbes • Children no longer work in mines or as chimney sweeps, but are exposed to man-made chemicals and pesticides in their food or increased environmental pollution [57] • Atomic energy has increased potential for death and injury from radiation exposure

Redundancy (duplication of components to protect against individual failures) • Not effective means of controlling risks • not effective against hazards that arise from interactions among components in today’s increasingly complex and interactive systems. • May in fact increase complexity to the point where the redundancy itself contributes to accidents

1.3.2 Increasing Complexity • Perrow distinguishes between accidents caused by component failures and those which he calls system accidents, that are caused by interactive complexity in the presence of tight coupling [259, Normal Accidents] • High technology systems are often made up of networks of closely related subsystems. Conditions leading to hazards emerge in the interfaces between subsystems, and disturbances progress from one component to another.

Increasing Complexity/Cont.An example of this increasingly common type of complexity, modern petrochemical plants often combine several separate chemical processes into one continuous production, without the intermediate storage that would de-couple the subsystems [274]. • “…analysis of major industrial accidents invariably reveal highly complex sequences of events leading up to accidents, rather than single component failures.” p.8

Increasing Complexity Cont./ In the past component failure was cited as the major factor in accidents, today more accidents result from dangerous design characteristics and interactions among components [108] p8 top.

Not only does functional complexity make the designer’s task more difficult, but the complexity and scope of the projects require numerous people and teams to work together. The anonymity of team projects dilutes individualresponsibility [172] • Paradox that people are willing to spend money on complexity but not on simplicity [158, Kletz] WHY IS THIS THE CASE?

CASE in Point: A British Chemical Plant (p.9) • pump, various pipelines, had several uses including: • transferring methanol from a road tanker to storage • charging it to the plant, • moving methanol back from a road tanker to storage, • charging it to the plant • moving recovered methanol back from the plant

On this particular occasion, a tank truck was being emptied: • The pump had been started from the control panel but had been stopped by means of a local button. The next job was to transfer some methanol from storage to the plant. • the computer set the valves, but as the pump had been stopped manually it had to be started manually.

When the transfer was complete the computer told the pump to stop, but because it had been stated manually it did not stop and a spillage occurred[157, p.225]. P 9 • a simpler design -- independent pipelines for different functions, actually installed after the spill, made errors much less likely and was not more expensive over the lifetime of the equipment.

Computers • may encourage the introduction of unnecessary and dangerous complexity • enable more interactive, tightly coupled, and error-prone designs to be built • Kletz has noted: “Programmable electronic systems have not introduced new forms of error, but by increasing the complexity of the the processes that can be controlled, they have increased the scope of the introduction of conventional errors” [158].

Accepting Perrow’s argument that interactive complexity and coupling are a cause of serious accidents, then the introduction of computers to control dangerous systems, may increase risk unless great care is taken to minimize complexity and coupling.. Conclusion

1.3.3 More People Exposed to Hazards • Larger flight capacities • Dangerous plant facilities closer to population centers • More of workforce in cities or within commuting distance. • Interdependencies and complexity cause ripple effects of hazards magnifying potential consequences. • 1.3.4 High Energy Sources increase risks

1.3.5 Increasing Automation of Manual Operations • Automation does not remove humans but tends to redefine their roles • operators become concerned with maintenance, repair and higher level supervisory control and decision making [270]

Operators relegated to central control roomsCase in Point: 1977 NYC Blackout • Indirect Information • Operator followed prescribed procedures • But electrical system was brought to a complete halt • Operator could not know there were two relay failures: • one leading to a high flow over line normally carrying little or no current (operator would have been alerted) • other blocked flow over line making it appear normal

Operator had no way of knowing that zero reading would appear normal. • Operators become the “scapegoat” of an automated system.

Operators and Embedded Systems • Embedded systems can mask the occurrence and subsequent development of a problem • When malfunction is discovered it may be more difficult to control • Systems may be hidden or distorted • Such design further limits operator options and hinders broad comprehension. p.11

Case: China Airlines, 1985 • 747 suffered slow loss of power in right outer engine • autopilot compensated preventing yaw to the right • when limit reached, crew had no time to determine cause • plane rolled and went into vertical dive of 31,500 ft. • Aircraft severely damaged

Multiple Goals may lead to conflicts • 1970’s attempt to combine energy savings efforts with process plants lead to complications • Safety and Economy conflicting • component interactions make system functions less transparent to designers and operators • Place where trouble first recognized may not be where it started.

1.3.7 Increasing Scale and Centralization • Bupp and Derian [1968] observed that manufacturers were taking orders for nuclear power plants six times the size of those in operation. • Previously 1-2 times extrapolations were considered at the outer boundary of acceptable risk. • Brown’s Ferry Accident, 1975 was 10 times size of plant construction in 1966

Supertankers built without sound design and redundancy • Mostert writes: • The gigantic scale of vessels creates an abstract environment in which crews are far removed from direct experience of the sea’s unforgiving qualities and potentially hostile environment. Heavy automation undermines much of the old-fashioned vigilance and induces engineers to lose their occupational instincts -- qualities that in earlier days of shipping were an invaluable safety factor.

Increasing Pace of Technological Change • Average time of conversion of technical discovery into commercial product was 30 years [1880 -1919] • now 5 years! • number of new products increasing exponentially • dangerous new substances with economic pressures preventing adequate testing.

Pace of Technological Change/ cont. • Learning by trial and error not possible in modern times • Design and testing procedures must be right from the start. • Christopher Hinton (1957) writing about nuclear power pointed out that in other domains, learning from failures was possible. • Progress continues at torrid pace; new standards and regulatory procedures are necessary.

1.4 How Safe Is Safe Enough? • The goal is to understand and manage risk in order to eliminate accidents or to reduce their consequences. • Frola and Miller (88) claim that system safety investment has reduced losses where it has been applied rigorously in military and aerospace programs.

How Safe is Safe/ cont. • Conflicting goals between • safety, performance, and goals. • e.g. Industrial arm which cannot be stopped easily. Slower arm determines that it must hit something. • human interface(s) designed for ease of use are often more troublesome

Risk - Benefit Analysis • Often viewed as only way to make technology and risk analysis. • Must be able to: • measure risk • choose appropriate level for risk • Systems must be designed and built while knowledge of their risk is incomplete or nonexistent.

Risk Assessment / cont. • Impossible to measure before system is built. • Failure rate of 4 (over 10000 years) and use body is missing • Hard to perform and hard to determine acceptable risk. • Case of Ford Pinto gas tank • Optimal Risk involves a tradeoff that minimizes the sum of all undesirable consequences.

Risky Systems and Perrow • Perrow Divides High-Risk systems into 3 categories:

Nancy Leveson: Safeware CHAPTER 1: Risk in Modern Society