Flight Test and Statistics

1. Flight Test and Statistics PRESENTED BY Richard Duprey Director, FAA Certification Programs National Test Pilot School Mojave, California

2. Flight Test and Statistics“If you want to be absolutely certain you are right, you can’t say you know anything.”

3. Flight Test and Statistics Overview • Background on National Test Pilot School • Coverage of Statistics • Scope - six hours of academics • Detail • Use of statistics in flight test • Types of questions we try to answer

4. NTPS Background • Private non-profit • Grants Master Science • Only civilian school of its kind • SETP equivalent to USAF and Navy Test Pilot Schools • Offers variety of courses (Fixed Wing and Helicopters) • Professional - 1 year • Introductory • Performance and Flying Qualities Testing • Systems Testing • Operational Test and Evaluation • NVG • FAA Test Pilot / FTE initial and recurrent training

5. Data Analysis 0 z

6. “Tunnel in the Sky”

7. Data Analysis - Hour 1 • Types of Errors • Types of Data • Elementary Probability • Classical Probability • Experimental Probability • Axioms • Examples

8. Introduction • Flight testing involves data collection • time to climb • fuel flow for range estimates • qualitative flying qualities ratings • INS drift rate • Landing and Take-off data • Weapon effectiveness • All of these experimental observations have inaccuracies • Understanding these errors, their sources, and developing methods to minimize their effect is crucial to good flight testing

9. Types of Errors • There are two very different types of errors • systemic errors and random errors • Systemic errors • repeatable errors • caused by flawed measuring process • ex: measuring with an 11 inch ruler or airspeed indicator corrections • Random errors • not repeatable and usually small • caused by unobserved changes in the experimental situation • errors by observer - reading airspeed indicator • unpredictable variations - small voltage fluctuations causing fuel counter errors • can’t be eliminated but typically distributed about a well defined distribution

10. Types of Data • There are four types of numerical data: • NOMINAL DATA • numerical in name only - say an aircraft configuration • 1 = gear down, 2 = gear up, 3 = slats extended • normal arithmetic processes not applicable • 3 >1 or 3-1=2 are not valid relationships • ORDINAL DATA • contains information about rank order only • #1 = C-150, #2 = B-1, #3 = F-15 • in terms of max speed: 3>1 is valid, but not 3-1=2

11. Types of Data • There are four types of numerical data (continued) • INTERVAL DATA • contains rank and difference information - ex: temperature in degrees Fahrenheit • 30, 45, 60 at different times, 15 deg. difference • zero point arbitrary, so 60o F is not twice 30oF • RATIO DATA • all arithmetic processes apply • most flight test data falls into this category • Can say that a 1000 pound per hour fuel flow is 4 times greater than 250 PPH

12. Probability and Flight Test • Quantitative analysis of random errors of measurement in flight testing must rely on probability theory • Goal • Student to understand what technique is appropriate and limitations on the results

13. Elementary Probability • The probability of event A occurring is the fraction of the total times that we expect A to occur - • Where: - P(A) is the probability of A occurring • - na is the number of times we expect A to occur • - N is the total number of attempts or trials

14. Elementary Probability • From this definition, P(A) must always be between 0 and 1 • if A always happens, na = N and P(A) = 1 • if A never happens, na= 0 and P(A) = 0 • In order to determine P(A) we can take two different approaches • make predictions based on foreknowledge (“a priori”) • conduct experiments (“a posteriori”)

15. Classical (‘a priori’) Probability • If it is true that • every single trial leads to one of a finite number of outcomes • and, every possible outcome is equally likely • Then, • na is the number of ways that A can happen • N is the total number of possible outcomes • For example: • six-sided die implies six possible outcomes: N = 6 • if A is getting a 6 on one roll, na = 1 • P(A) = 1/6 = 0.1667

16. Second Example • What is the probability of getting two heads when we toss two fair coins? • There are four possible outcomes (N = 4) • (H,H) (H,T) (T,H) (T,T) • na = 1 since only one of the possible outcomes results in two heads (H,H) • Thus P(A) = 1/4 = 0.25

17. Classical (‘a priori’) Probability • Approach instructive • Generally not applicable to flight test where: • Possible outcomes infinite • Each possible outcome not equally likely • Leads us to second approach

18. Experimental (‘a posteriori’) Probability • Experimental probability is defined as • Where - nA obs is the number of times we observe A Versus . number of times we expect A to occur - Nobs is the number of trials

19. Experimental Example • If the probability of getting heads on a single toss of a coin is determined experimentally, we might get 1.0 Porb (heads) 0.5 0 norb 1000 100 10 1

20. Probability Axioms • Probability Theory can be used to describe relationships between events

21. Probability Axioms • Three probability axioms are easily justified as opposed to proven • P(not A) = 1 - P(A) • Probability of something happening has to be one • P(A or B) = P(A) + P(B) • P(H or T) = 0.5 + 0.5 =1 for a single coin • P(A and B) = P(A) x P(B) • P(T and T) = 0.5 x 0.5 = 0.25 for two coins • same answer we got when examining all possible outcomes • The last two axioms require that • each outcome is independent • A occurring doesn’t affect probability of A or B occurring • each outcome is mutually exclusive • Only one can occur in a single trial

22. Example • Problem: • Based on test data, 95% of the time an F-4 will successfully make an approach-end barrier engagement on an icy runway • what is the probability that at least one of a flight of four F-4’s will miss? • Solution: • P (1 or more miss) = 1 - P(all engage) • Probability that at least one will miss is the complement of the probability that all will engage • P (all engage) = P(1st success) × P(2nd ) × P(3rd) × P(4th) = 0.95 × 0.95 × 0.95 × 0.95 = 0.954 = 0.81 Thus, • P (1 or more miss) = 1 - 0.81 = 0.19

23. Example • Problem: • What is the probability of getting 7 or 11 on a single roll of a pair of dice? • Solution: • Since getting 7 or 11 are independent, mutually exclusive events, we can say • P (7 or 11) = P (7) + P (11) • N = 62 = 36 • n7 = 6 • (6, 1) (1, 6) (5, 2) (2, 5) (4, 3) (3, 4) • n11 = 2 • (6, 5) (5, 6) • Thus, • P (7) = 6/36, P (11) = 2/36 • P (7 or 11) = 6/36 + 2/36 = 0.222

24. Data Analysis - Hour 2 • Populations and Samples • Measures of Central Tendency • Dispersion • Probability Distributions • Discrete • Continuous • Cumulative

25. Population & Samples • A population is all possible observations • Many populations are infinite • A pair of dice can be rolled indefinitely • Population of F-117 weapons deliveries is all the possible drops it could make in its lifetime • Some populations are limited • Votes by registered Republicans • A sample is any subset of a population • For example • 100 rolls of a pair of dice • Bomb scores for 100 weapon delivery sorties

26. Population Constructs • Constructing a population • Must impose assumptions • Homogenous • Independent • Random

27. Sample Requirements • Homogeneous • the data must come from one population only • DC-10 take-off data shouldn’t be used with MD-11 • Independent • selecting one data point must not affect subsequent probabilities • selecting and removing a heart from a deck of cards changes the probability of drawing another heart • DC-10 landing 75 feet past touchdown aim point on one landing doesn’t change probability that next landing will miss by same distance (or any distance) • Random • equal probability of selecting any member of population • using a member of a population with a bias would be non-random • F-16 with boresight error would cause a bias in downrange miss distance

28. Measures of Central Tendency • Given homogenous, independent, random sample, need to describe the contents of that sample • Measure steel rod diameter with a micrometer - would get several different answers • Tighten the micrometer • Dust particles on the rod • Reading scale on micrometer • What to do with answers that are different?

29. Measures of Central Tendency • There are three common measures of central tendency: • Mean (arithmetic average) - most commonly used • Mode • most common value in the sample • there may be more than one mode • Median • middle value • for an even-numbered sample, average the two middle values • Dangers ........

30. Dispersion • Just reporting the mean as the answer can be very misleading • Consider the following two samples, both with a mean of 100 (and same median as well) • Sample 1: 99.9, 100, 100.1 • Sample 2: 0.1, 100, 199.9 • We also need to report how much the data generally differs from the mean value

31. Deviation • We define deviation as the difference between the ith data point and the mean: • Averaging the deviations does not help:

32. Mean Deviation • Since there as many deviations above and below the mean, we could average the absolute values of deviations:

33. Standard Deviation • While the mean deviation can be used, the standard deviation s is a more common measure of dispersion: • versus • The square of the standard deviation, s2, is called the variance

34. Notation • Normally, we use Greek letters to denote statistics for populations: m for population mean s2 for population variance • And we use Roman letters for sample statistics: for sample mean s2 for sample variance

35. Sample Standard Deviation • One other difference exists between s and s • The sample standard deviation has the sum of the squares divided by N - 1 versus N • Mathematically, this is due to a loss of one degree of freedom • The effect is to increase the standard deviation slightly • Difference decreases as sample gets larger

36. Flight Test Example - PA28 Takeoff Distance • Two data points eliminated - wrong configuration, improper technique • Data adjusted for standard weight (2150 lbs.), runway slope (GPS), temperature, pressure, airspeed/altimeter corrections • Technique, rotate at 65, liftoff at 70, maintain 75 until 50 feet AGL

37. Probability Distributions • Statistical applications requires understanding of the characteristics of the data obtained • Probability distributions gives us such understanding

38. Probability Distributions • To understand probability distributions, consider the problem of tossing 2 coins • Let n represent the number of heads for a single toss of both coins • Then the probabilities of getting n = 0, 1, or 2 can be calculated: • for n = 0, P(0) = 0.25 • for n = 1, P(1) = 0.5 • for n = 2, P(2) = 0.25

39. Discrete Distributions • We can present the data as a bar graph

40. Empirical Distributions • In flight test, we are concerned with empirical distributions versus theoretical in the coin example • If we collect data on landing errors:

41. Continuous Distributions • If we get more and more data, and make the intervals smaller, our histogram approaches a continuous curve: Continuous Probability Distribution of Touchdown Miss Distance • Can’t be interpreted same way as the previous discrete distribution

42. Continuous Distributions • Height of curve above a point is not the probability of “x” having that point value • Any one point on the x-axis represents a non-zero point on the curve • But the probability associated with that single point must be zero, since there are an infinite number of points on the x-axis • We can meaningfully talk only about the probability of being between two points a and b on the x-axis

43. Probability as Area Under Curve • The probability of getting a result between a and b is rep-resented by the area under the probability distribution curve between a and b f (x) P(a £ x £ b) x

44. Cumulative Probability Distribution • A cumulative probability distribution gives the probability that x is less than or equal to some value, a • Relative probability of aircraft landing miss distances could be displayed in the following cumulative distribution 1.0 0.95 f (x) 0.5 x xT

45. Data Analysis - Hour 3 • Special Probability Distributions: • Binomial • Normal • Student’s t • Chi squared

46. Binomial Distribution • The binomial is a discrete distribution • It tells us the probability of getting n successes in N trials given the probability (p) of a single success • Limiting cases • if n = N, then obviously P(N) = pN • if n = 0, then P(0) = (1 -p)N • or, letting q = 1 - p, P(0) = qN • For 0 < n < N, the possible number of combinations of success and failure gives

47. Binomial Distribution -flight test ex. • Two flight control systems are equally desirable • What is probability that 6 out of 8 pilots would prefer system A over B? • If A and B are truly equally good, probability of pilot picking A over B is 0.5 (P=q =0.5) • Probability of 6 pilots picking A over B is: = 0.109 • There is only a 11% probability that this would happen. If it did, it would mean that your initial assumptions about the two flight control systems was in error

48. Binomial Flt. Test Example • If p = q = 0.5, then for N = 8, the binomial distribution would be and from the figure, P(2) is about 11%

49. Normal Distribution • The normal distribution is a continuous probability distribution based on the binomial • SINGLE MOST IMPORTANT DISTRIBUTION IN FLIGHT TEST ANALYSIS • Any deviation from a mean value is assumed to be composed of multiples of elemental errors evenly distributed • The mathematical derivation is left as an exercise

50. Normal Distribution • Graphically, it can be seen that x = m gives the maximum value and x = m ± s are the two points of inflection on the curve f (x) x m m+s m-s