Assessment of Probability Distributions

1. Assessment of Probability Distributions Analytica User Group Webinar Lonnie Chrisman Lumina Decision Systems 6 Mar 2008 While you are waiting….Download “Probability assessment.ana” from talk abstract on User Group wiki page. http://lumina.com/wiki/images/6/6b/Probability_assessment.ana Do not start the assessment yet.

2. Explicit Uncertainty • Models bottom-out in parametersthat must be estimated. • Point estimates • Probability distributions • Benefits of encoding distributions: • Awareness of forecast precision • Better decisions (decisions based on uncertainty) • Makes risks explicit • Estimate can be “correct”, even if imprecise.

3. Subjective Probability • Assessed distribution reflects available knowledge (not an inherent property of the world). • There is no “correct distribution”. • Quality (e.g., “calibration”) is determined by the elicitation process (not the final result). Knowledge Probability Distribution Available Data ElicitationProcess Available Information Literature

4. Assessment Quality • Calibration • Discrimination Ability to adjust assessment for different conditions. More expertise  less entropy These are measured across many assessments using a fixed elicitation process. Hence, they grade the elicitation process, not an individual probability or distribution assessment.

5. Scoring Rules(binary outcome) • Assess one number: p = Pr(X=1) • Observe actual outcome, x • Scoring rule: • S(p,x) = quality of prediction (a number) • Rewards both calibration and discrimination. • “Proper scoring rule” • E[S(p,x)|x~q] is maximal when p=q • Cannot be “gamed” • Examples: • Log score: ln p(x) • Scaled log score: 1 + log2 p(x) • Quadratic score p(x)2 • Improper rule: linear score: p(x)

6. Probability Assessment Game Download “Probability assessment.ana” from talk abstract on User Group page. • http://lumina.com/wiki/images/6/6b/Probability_assessment.ana • These are not intended to be trick questions. • Do not look up the answer before you’ve answered.

7. Cognitive Biases Common biases include: • Over-confidence • Anchoring / insufficient adjustment • Motivational bias • Management bias • Expert bias • Denial of uncertainty

8. De-biasing… Probability that Hillary Clinton will be the Democratic Presidential Nominee this year? Discuss: • Relevant facts. (Clinton: has 1462 needs 563 more. Obama: has 1567 needs 458 more. Voters: 611 remaining. Undecided superdelegates: 346 [SJ Merc. News 6 Mar 2008]). • Describe possible scenarios that could prevent this from happening. • What events could transpire to cause it to happen? • Have you separated what you want from your estimate? Are you clear that your estimate does not influence the outcome?

9. Quartile assessment(real-valued quantities) (oops – its quartile, not quantile) • Scoring: • Scaled log score: 1+log2 αp(x / α), α=|x| adjusts for scale • Interval: 50% of your response should be inside quartiles, 50% should be outside your quartiles

10. Combating Cognitive Biases

11. Identify meaningful variables • People are less susceptible to biases in their own areas of expertise, and when the question is meaningful to them. • Make sure the measurement scale is meaningful to the person. (Units and linear/log scale). • Most of the time, the selection of variables and influence structure is far more important than numeric uncertainty!

12. Defining the Quantity • “Clairvoyant test”: Would an oracle know the value without needing further clarification? • Probability is not a measure of fuzziness. • Don’t worry if you don’t know, or can never know the value, as long as it is unambiguous and meaningful. • In principle, you can assess any quantity. • As a modeler, you can chose to decompose a quantity into simpler quantities for assessment.

13. Decompose to meaningful variables • If you feel the variable is conditional on some other variable, incorporate the conditionality into the model. • If you can decompose into variables that are easier to assess, make that explicit in the model.

14. Pre-assessment Before making any estimate of a particular variable, do the following: • List all relevant knowledge that you can. • Describe scenarios of how the extremes may occur – especially the least likely extreme. • Assess any background frequencies before taking the situational specifics into account. These help you to avoid anchoring and can help reduce over-confidence.

15. Design the assessment • What “shape” is the distribution? • Usually consists of identifying the distribution type, e.g.: Normal, uniform, LogNormal, etc. • What parameters will you estimate? • Quartiles: 25%, 50%, 75%, etc. • Min, max • Mean, Std.dev. • Probability (or cum prob) of selected values • Indirect: Betting scenarios, comparisons, etc.

16. Assess • Consider different approaches to assessing the same quantity. Do you get consistent estimates? If not, don’t just split the difference – dig deeper and explain why. • Take into account take you are likely to be over-confident, and that the less you know, the more over-confident you are likely you are likely to be.

17. Sensitivity Analysis • Use “Make Importance” and other sensitivity analyses to determine the sensitivity of your results to your assessments. • Revisit critical assessments.

18. Practice • Practice on almanac problems (as we did today).

19. Fast Assessment • Your (numeric) subjective assessment can depend on how long you think about it. • Less-thought  higher entropy, less discrimination • Calibration should not (ideally) suffer. • Fast distribution assessments can take less time than point assessments. (Variable definitions, model structure, should never be done hastily.)

20. Selecting a Distribution(Discrete numeric) Binary (yes/no) events: • Bernoulli(p) Tallies/counts (number of events): • Poisson(p) • Binomial(n,p) • Geometric(p), HyperGeometric(…) Integer values: • Uniform(min,max,integer:true)

21. Selecting a Distribution(categorical) • ChanceDist • ProbTable In both, you assess the probability of each possible outcome. Note also the task of identifying the outcomes. Neglecting possible outcomes can also be a source of bias.

22. Selecting a Distribution(Bounded Continuous) Distribution over a probability (0,1)-interval: • Beta(x,y), Beta_m_sd(m,sd) • Assess mean = x/(x+y) • Spread (σ) best judged graphically, plot and re-plot. Bounded on both ends. • Uniform(min,max) • Triangular(min,mode,max) Bounded from below (usually by 0): • Exponential(mean) • LogNormal(median:m,stdDev:sd) • Gamma(mean) • Weibull(shape,scale) – failure times

23. Selecting a Distribution(unbounded continuous) General: • Normal(m,sd) • ProbDist, CumDist – custom distributions Describing deviation from mean: • StudentT(dof) • ChiSquared(dof) Describing growth: • Logistic(mean,scale)

24. Final words • Most critical step: Identifying the variables. Don't let yourself be deterred by the fact that you do not know, nor can you know, some of the quantities in your model. Build the model as if you could know everything you would like. • Subjective uncertainty describes your state of knowledge – not a property of the world. • Humans exhibit cognitive biases when assessing probabilities, most notably over-confidence.