Forecasters

1. Forecasters’ view ofclimate-change:Methodology also counts J. Scott Armstrong The Wharton School University of Pennsylvania Kesten C. Green Business and Economic Forecasting Unit Monash University March 9, 2009 File: Heartland 09-JSA v36

2. Role of methodology Inputs Data Problems with the data Lack of disclosure Changes in measurement Relationships Which variables Direction and effects of variables Forecasting methodology How to integrate inputs to produce forecasts and prediction intervals for policy analysis 2

3. Climate change forecasting problem • Structuring the forecasting problem (JSA) • Auditing the forecasting process (JSA) • Selecting a proper forecasting method (JSA + KCG) • Testing forecast validity (KCG) 3

4. The forecasting problems Rational public policy requires scientific forecasts of the direction and magnitude—and confidence intervals—for: 1. Climate change 2. Effects of climate changes (costs & benefits) 3. Effects of alternative policies (ditto) We have been unable to find any scientific forecasts for #1, climate change: warming or cooling. (Green, Armstrong & Soon 2009) Our search for #2 & #3 continues, also without success to date.

5. Handouts for Armstrong & Green talks References for papers cited in our talks / WWF solicitation based on false advertising? Boxer vs.Armstrong Wager Gore vs. Armstrong Prediction Market. Suggestions on what to do? • Analogies study • WWF action steps • CO2 policy forecasting “Climate Change Forecasts are Useless for Policy Making”

6. Forecasting methodology • Structuring the forecasting problem • Auditing the forecasting process • Selecting a forecasting method • Validation tests 6

7. Auditing IPCC forecasts g IPCC “projections” of global temperature change used improper procedures. Green & Armstrong audit* showed: • IPCC authors violated 72 forecasting principles. 2. Forecasts by scientists, not scientific forecasts. 3. No proper evidence on predictive validity * Green, K. C. & J. S. Armstrong (2007) 7

8. Audit of Polar bear (PB)population forecasting Polar bear population has grown in recent decades due to restrictions on hunting. Government forecast global warming assumed to reduce sea ice. Analysis of 5 years of data (with “outlier” removed) related PB population to ice. PB experts used “a” and “b” to forecast rapid decline in PB population over 50 years. Government assumed that an “endangered” classification would boost the PB population. No forecasts for this, or for alternative policies.

9. Only 12% of the principles were properly applied*. Government researchers declined to provide peer review or to answer questions we had about their paper. Offered to act as reviewers for the journal. Government researchers declined invitation to review or to provide commentary. * Armstrong, Green, & Soon (2008) Polar bear audit results

10. Example of a principle: Full disclosure Government researchers used complex mathematics to forecast PB populations. We requested the data they used on the PB population.

11. Here are the data provided to us in our requests for our audit of the U.S. Fish and Wildlife Services forecasts:

12. Current polar bear situation Lacking scientific forecasts*, the U.S. Fish and Wildlife Services classified the PB as “threatened.” * Armstrong, Green & Soon (2008)

13. Announcing the Boxer-Armstrong charity wager Senator Barbara Boxer supports government forecasts that the polar bear population will decrease by about 23% over the next ten years. With an upward trend over recent decades and high uncertainty, Armstrong, Green & Soon forecast no decline in the PB population. Professor Armstrong challenges Senator Boxer to a $5,000 charity wager on the polar bear population for the next ten years. An independent two-person panel would determine which forecast was closest to the actual population at the end of 2018.

14. Senator Boxer’s response as of March 8

15. Policy based on GW forecasts:World Wildlife Fund WWF TV commercials: In an appeal for members and donations, the ads claim that due to global warming, the number of polar bears is decreasing rapidly, thereby putting the species at risk of extinction. WWF website says: “The general status of polar bears is currently stable, though there are differences between the populations. Some are stable, some seem to be increasing, and some are decreasing due to various pressures. The status of several populations is not well documented.”

16. False advertising by the WWF? Since Dec. 12, 2008, I have made attempts to contact the CEO and various Trustees of the WWF to raise this issue of what appears to be false advertising I also offered to report on their position at this conference.

17. WWF response as of March 8 17

18. Prevalence of false advertising Are many organizations engaged in false advertising with respect to global warming? If so, what might be done to eliminate such false advertising?

19. WWF action steps? Please put your suggestions on the sheets that we have passed out. I will summarize these on theclimatebet.com

20. Forecasting methodology • Structuring the forecasting problem • Auditing the forecasting process • Selecting a forecasting method • Validation tests 20

21. Selecting a proper forecasting method Evidence-based forecasting methods* • Expert judgment • Extrapolation • Causal models * Forecasting methods that have been experimentally tested for efficacy in given situations 21

22. Expert judgment Unaided expert judgment not useful given high complexity and high uncertainty. (Tetlock 2005) Applies with respect to • Accuracy • Experts’ statements about confidence • Agreement among experts 22

23. Evidence-based expert judgment methods (use structure) Delphi Prediction markets Structured analogies Causal models

24. Delphi Delphi: A multi-round survey of unbiased experts who make anonymous forecast with feedback on predictions and reasons given between rounds. Assumes that experts can make useful forecasts. It also helps to assess expert opinions in an objective manner and provides decision makers with reasons. 24

25. Prediction markets • We are currently examining the use of prediction markets 1) assess expert opinion 2) forecast climate change • Potential problems: a. If individuals can’t forecast, does combining their forecasts help? b. Is a market for long-term bets feasible c. Would a series of short-term bets be useful? d. Is play money adequate? e. Might people manipulate the market? 25

26. Prediction markets can also summarize prevailing opinions in an objective manner When will “Manmade Global Climate Change” next be disputed by a major media outlet? (HubDub.com) 26

27. Gore vs. Armstrong conducted via a prediction market To stimulate work on validation of alternative methods to forecast climate change. I have informed Mr. Gore of this prediction market and asked for his comments and suggestions.

28. Gore’s response as of March 8 28

29. Progress on prediction market Assume Armstrong and Gore made a ten-year bet starting January 1, 2008. Armstrong forecast “no change” in global mean temperature. Gore implied that global mean temperatures would increase at least as rapidly as the IPCC’s 1992 projection of 0.03oC per year Criterion: mean absolute errors of annual forecasts for the ten-year period using UAH global mean temperature record. 29

30. Current betting on a play-money market (hub-dub) Armstrong by 62% as of March 4, 2009 (100 bets) http://tinyurl.com/gore-armstrong-bet

31. Plans on prediction markets • Alternative criteria (e.g., Hadley temperatures) • Real money sites • Add information via comments • Vary time horizon (e.g., one-year ahead?) • Compare with Delphi expert panel • Graefe, A. J., J. S. Armstrong, K.C. Green (2009) 31

32. Handouts for Armstrong & Green talks References for papers cited in our talks / WWF solicitation based on false advertising? Boxer vs.Armstrong Wager Gore vs. Armstrong Prediction Market. Suggestions on what to do? • Analogies study • WWF action steps • CO2 policy forecasting “Climate Change Forecasts are Useless for Policy Making”

33. References for the Armstrong and Green talks [Papers available at http://publicpolicyforecasting.com unless otherwise indicated] Armstrong, J. S., Green, K. C., & Soon, W. (2008). Polar Bear Population Forecasts: A Public-Policy Forecasting Audit. Interfaces, 38, 5, 382–405. (includes commentary & response). Graefe, A., Armstrong, J. S., & Green, K. C. (2009). Using Prediction Markets to Solve Complex Problems: An Application to the ‘Climate Bet’. Working paper available at http://kestencgreen.com/cbpm.pdf Green, K.C., & Armstrong, J. S. (2007). Global Warming: Forecasts by Scientists versus Scientific Forecasts. Energy and Environment, 18, No. 7+8, 995-1019. Green, K.C., & Armstrong, J. S. (2007). Structured Analogies in Forecasting. International Journal of Forecasting, 23, 365-376. Green, K.C., Armstrong, J. S., & Soon, W. (2009). Validity of Climate Change Forecasting for Public Policy Decision Making. International Journal of Forecasting, Forthcoming. Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press, Princeton, NJ. Gore-Armstrong prediction market at http://tinyurl.com/gore-armstrong-bet Details on Gore-Armstrong bet at http://theclimatebet.com