The Safety Benefits of LOSA: A Manufacturer’s View

1. R. Curtis Graeber, Ph.D. Senior Technical Fellow Chief Engineer, Human Factors Boeing Commercial Airplanes The Safety Benefits of LOSA: A Manufacturer’s View International LOSA & TEM Workshop, Toulouse, France November 16, 2006

2. Today’s Topics • A Proactive Approach to Safety • TLC-Boeing Collaboration • LOSA’s Unique Contribution to Safety • Learning from LOSA data • Future Opportunities

3. Boeing’s Approach to Safety • Be proactive in research and promoting safety • Learn from incidents as well as accidents • Learn from the experience of our customers • Continuously improve the Airplane Life Cycle process • Seek Global Involvement • Airlines • Other Manufacturers/Sub-contractors • Regulatory agencies • Safety and flight crew organizations

4. Global Aviation Safety Roadmap Industry commits to unify plans through Roadmap

5. ICAO Global Aviation Safety Roadmap Goals and Objectives: • Provide a common frame of reference for all stakeholders • Coordinate and guide safety policies and initiatives worldwide to reduce the accident risk for commercial aviation • Avoid duplication of effort and uncoordinated strategies • Encourage close industry and government cooperation on common safety objectives Timescale: Near (2006-10) and Medium Term (2010-14)

6. Boeing’s Safety Approach: Focus on Human Factors • Collaborate globally on external research • NASA, FAA, NLR • Madrid BRTE • Customer airlines • Universities (Illinois, Michigan, Cranfield, Delft, Massey, etc.) • University of Texas HF Project/The LOSA Collaborative • Create safety tools to support SMS at airlines • Flight Operations Support Program (FOSP) • Procedural Event Analysis Tool (PEAT) • Maintenance & Ramp Error Decision Aids (MEDA, REDA) • Cabin Procedure Investigation Tool (CPIT) • Line Operations Safety Audits (LOSA)

7. The LOSA Collaborative (TLC)Boeing MOU Agreement • Signed May 2002 • Assures airline and crew member confidentiality • Enables TLC to provide customers a two-way channel of communication with Boeing • Establishes a mutually valuable research and safety partnership

8. The Value of Collaboration For The LOSA Collaborative: • Manufacturer’s perspective and expertise in global aviation safety • Scientific and expert technical support • Detailed design knowledge • Comparisons with findings from other sources including FOSPs • Expanded opportunity to benefit all airlines • Enhanced resources (data collection software)

9. The Value of Collaboration For Boeing: • Synergistic research partnership • Invaluable, objective insights about how our products are operated in “normal” situations • Access to information about crew errors during normal operations (ARCHIE) • Insights about how crews recover from errors • Objective crew performance data to guide design of future airplanes and procedures

10. Improving the Crew Interface Potential Data Sources: • Accidents Investigations • Incident Investigations • Crew Reports: ASAP, STEADES, etc • Recorded airplane data: FDM/FOQA • Normal Operations • Airline Feedback • LOSA • Boeing FOSPs, in-flight observations

11. Accident Data • Example: NTSB, AIB, BASI etc. • Extraordinary detail about rare single events • Can be a high value data source when applicable • Very dependent upon expertise and processes of investigating authority • Poor comparability and usually too few worldwide to understand HF issues • Always too late!

12. Incident and Crew Report Data ECCAIRS, ASRS, CHIRP, ASAP, STEADES • Key factors frequently identified • Participant recollection can be suspect • Level of detail varies, especially regarding equipment • Extremely high value data source to corroborate other data • Can support precursor search

13. FDM/FOQA Data • Proprietary and closely protected by operators • Very detailed objective airplane outcome data from normal operations • Uniquely suited for multi-flight statistical analysis • Limited in scope • Requires information about context and crew actions to understand “Why”

14. LOSA Data • Structured format that accurately captures • Facts and events in normal operations • Background that puts facts & events in context • Provides more than single flight context • Can provide better detail than incident reports • Greater accuracy than anecdotal analysis • Can support rate of occurrence analysis • Reveals how crews successfully manage errors • Enables focused observations on issues of interest

15. Boeing’s Use of LOSA Data • Search for threats and errors related to safety of our fleet (e.g., automation related errors) • Close the “loop” on design • Correlate results with other data to gain convergent validity • Capitalize on observations in ARCHIE to discover trends and patterns

16. Other LOSA Benefits • Proactively discover new safety issues/ through manufacturer/airline inquiry • Assist in prioritizing risks and identifying the real safety issues • Help define “Best Practices!” • Validate or better formulate other safety initiatives • Provide design and procedural insights about different Boeing types and models

17. Experienced-Based Airplane Lifetime Safety Cycle New Technology Laboratory Testing Flight Testing • Previous DR&O • FAR’s, • JAR’s • Customer requirements • SR&O’s • Design reviews • Analyses • Testing Analyses Delivery Boeing design and regulatory requirements Validateandcertify LOSA In-Service Operation Design Produce Service bulletins, etc. • DATA • 300,000 telexes per year • 85,000 accidents, incidents to date Lessons Learned Problem Resolution Reports Modification Review Board Test requirements Continuous feedback of information BCA In-Service Safety Process

18. LOSA Data - Influence on Flt Deck Design • LOSA provides valid performance data on flight crew successes and errors • Flight deck designers can learn from both categories • Leverage successes into even better crew-centered designs • Change / enhance control operations to design away from errors • LOSA data provides a framework for future flight deck designs and enhancing existing designs

19. Visits to The LOSA Collaborative • Annual 2-Day Visits, Safety Pilot + Human Factors Specialist • August 2005 • Reviewed approximately 400 sets of ARCHIE data, primarily 747-400 and 777 • Airfield and Dates De-identified • Airline De-identified • October 2006 • Reviewed entire archive of 5609 sets of data

20. 2005: Where Are The Threats? • Threats in the pre-departure phase of flight are highest. • Where English is not the native language • Traffic compression is a growing threat • Pre-flight after major maintenance often do not adequately prepare the airplane for the aircrew

21. An Example: Checklist Errors • LOSA data confirm our previous reports (Boorman 2001; Holder ‘03, ‘04) • Value added by LOSA data • Provided rates of occurrence • Separated occurrences by phases of flight • Added context to better understand each error • Most frequent reasons for checklist errors are: • Late checklist • Checklist not performed to completion • Items on checklist left open • Wrong checklist • Not knowing which checklist to use (training & proficiency)

22. Percent of Flights with Checklist Errors

23. 777 Checklist • We knew electronic checklist could provide a great advancement in safety • Operator experience confirms we succeeded • We designed a lot of the traditional checklists problems out • Anecdotal and FOSP information has identified a few potential enhancements • Seeking more information, such as LOSA, to focus our enhancement efforts

24. Hitting The Target On Errors • There seems to be a correlation between error and delays. • Most errors are inconsequential! • Callouts for un-stabilized approaches and procedural callouts are not solving handling errors. • When deviations occur, crew members identify it with a callout only ½ the time!

25. Phase 1 – Discovering Errors • 165 B777 LOSA flights • 11 of 165 flights had checklist errors (7%) • 13 checklist errors in 11 flights • 3 errors reported as intentional • 1 error led to additional error • No undesired aircraft states

26. Phase 2 – Classifying errors • Omitted checklist • Forgot checklist • Did not call for checklist • Self-initiated checklist • Checklist performed late or at wrong time • Checklist not performed to completion (e.g., open item that gets forgotten) • Checklist performed from memory

27. Phase 3 – Learning the Context • Pilot forgot to ask for checklist • PNF forgot he left an item open on CKL • PF did not call for “After-Landing” CKL • Landing CKL initiated at 2000 ft (late) • Late completion of landing CKL • FO ran “Engine Start”, “Pushback” CKL from memory • Most errors occur on descent/approach (7/13) and pre-departure (4/13) phases

28. Phase 4 – Design Solution • New checklist alerting will be integrated into new airplane designs to prevent: • omission • forgetting, and • late execution of critical normal checklists • Plus other data-driven enhancements to 777 and 787 baseline checklist design

29. Altitude Deviations Are Common • Many occur due to distractions and high workload situations • Deviation around weather • Fuel in center tank operational requirements • ATC • Deviation around traffic • Missed Approaches • Traffic compression

30. Altitude Deviation Solution • Possibly Design • More emphasis in the Flight Crew Training Manuals to bring about awareness • More emphasis during LOFT and CRM training to bring about awareness • More structured/defined cross check process

31. 2006 ARCHIE Review: Unstabilized Approaches (5609 total observations as of September 2006) 245 Unstable Approaches - 4 % • 182 Exceeded Company Established Parameters • (Landed within touchdown zone – 74%) • 59 Landed Outside Touchdown Zone • (Landed long, short, firm or off-center – 24%) • 4 Executed Missed Approach (no further error - 5%)

32. 2006 ARCHIE– Unstabilized Approaches 179 LOSA OBSERVATIONS – BOEING AIRPLANES 78% UNSTABLIZED APPROACHES – VISUAL APPOACH • 56% CAPTAIN FLOWN (BASE RATE) • 4 Executed Missed Approach (179 Boeing – 2%)

33. 78% UNSTABLIZED APPROACHES – VISUAL APPOACH 2006 ARCHIE – Unstabilized Approaches NOTE: CALLOUTS ABSENT PM – NO COMMUNICATION OR CALLOUTS PF – NO ALERTING TO DEVIATIONS

34. LOSA Methodology as Research Tool • University of South Australia Qantas-CASA-AIPA Long-Haul Fatigue Study • Full-Mission Simulation • Fresh crews • Recently returned crews from trip • Structured LOSA observations at key decision points in scenario Note: See Thomas et al., FSF-IATA-IFA International Air Safety Seminar Oct 2006 for reports

35. Research Overview International Trip (US or Europe) Simulator Session (Sydney) 4 Days Recovery (Home) or 5 Days Rest (Home) sleep/wake/fatigue ratings

36. Three Hour Protocol Simulator Session Decision-making Event Subjective Ratings Performance Tasks Dispatch Simulator Scenario Performance Tasks Subjective Ratings Interview 10 10 30 120 10 10 30 PRE-SIM SIMULATOR POST-SIM

38. Cruise CDE Q T6 T5 Descent - Approach - Landing T7 Take-Off Climb T8 T1 T2 T3 T4 Pre-Flight T1 Dispatch with Thrust Reverser “Locked Out” CDE Critical Decision Event T6 High Speed Descent & Track Shortening T2 Change in Loadsheet T3 Change in Duty RWY T7 ATC QNH Error T4 Clearance to Lower Altitude on Departure T8 Failure to Provide Clearance to Land T5 Higher Cruise Level & Expedite Climb Scenario Structure high low workload

39. Situation Awareness COA Development Decision Evaluation Decision Variables • Obtain Melbourne WX/TTF • Time to obtain Melbourne WX/TTF • Below alternate criteria • Cross-check figures • Number of options considered • Runway 16 available • Diversion to another airport • Time to decision • Time to execute decision • Decision review

40. Future Questions For LOSA • How is TCAS being used in an operational context? • Are crews proactive or reactive with regards to potential TA/RA’s? • Is there any correlation between threat rates versus performance errors? • What are “The Best Crew Practices”? • Attributes of crews who make few errors? • Attributes of crews who do an excellent job in managing or trapping errors? • Why do some crews commit more errors than others?

41. Conclusions • LOSA is a unique tool that provides a “window” into crew performance during normal ops • By cooperating, we can greatly multiply LOSA’s potential for improving global aviation safety • Consistent implementation is the key • Will learn together what we don’t know • Will help us to anticipate future system risks • More questions are being raised than answered • Success is critically dependent on the protection of safety datasuch as LOSA • Boeing is committed to strongly supporting LOSA as a key safety tool

42. Thank You For Your Attention!