computational aggregation

1. computational aggregation scott e page university of michigan santa fe institute

4. Evolving Interpretations and the Possibility of Collective Wisdom Jenna Bednar Lu Hong Ken Kollman PJ Lamberson Scott Page Maria Riolo Nathan Seegert

5. Outline The Wisdom of Crowds - anecdote - undergraduate - graduate Generated vs Interpreted Signals - foundations - statistical properties

8. UCSC

9. Iowa Electronic Markets IEM Prices Obama 0.535 McCain 0.464 Final Gallup Poll Obama 0.55 McCain 0.44 Actual Outcome Obama 0.531 McCain 0.469 Oxford 5-1- 07

10. Interpreted vs Generated Signals Generated Signal: truth plus a disturbance or interference Interpreted Signal: prediction from a model

11. Generated Signal noise Outcome Signal

12. UCSC

13. L -  L + 

14. L

15. Generated Signals Independent Common correlation Common bias Produced in any number by some generating process UCSC

16. Voting vs Markets Voting: all models get equal weight Markets: models can get differential weight

17. Signals as Predictions Outcome  in  N individuals indexed by i Signal received by i: si Vector of signals: s = (s1,s2,…sn ) f(si si | ) = distribution of signals si conditional on . The signals are often assumed to be independent

18. Measures Squared Error for individual i: SqE(si) = (si - )2 Average Squared Error: SqE(s) = Collective Prediction: c = Collective Squared Error: SqE(c) = (c - )2 Predictive Diversity: PDiv(s) =

19. Diversity Prediction Theorem SqE(c) = SqE(s) - PDiv(s)

20. Crowd Error = Average Error - Diversity 0.6 = 2,956.0 - 2955.4

21. Signals as Random Variables Mean of i’s signal: i() Bias of i’s signal: bi =(i() - ) Variance of i’s signal: vi =E[(i() - )]2 Average Bias B = Average Variance V = Average Covariance C =

22. Bias Variance DecompositionEnsemble Learning Theory E[SqE(c)] =

23. Resolving the Paradox Diversity Prediction Theorem: Predictive Diversity is realized diversity, which improves accuracy. Bias Variance Decomposition: Variance corresponds to noisier signals, which reduces accuracy. Negative covariance implies diverse realized diversity and improves expected accuracy of collective predictions.

24. Large Population Accuracy If the signals are independent, unbiased, and with bounded variance, then as n approaches infinity the crowd’s error goes to zero E[SqE(c)] =

25. Interpretive Signal model Attributes Prediction

26. Interpretations Reality consists of many variables or attributes. People cannot include them all. Therefore, we consider only some attributes or lump things together into categories. (Reed, Rosch, Mullinathun, Jackson and Fryer, Collins)

27. West Virginia • Congressional District • District 1 • District 2 • District 3

28. West Virginia Slaw available on request Slaw standard on hot dogs Slaw not available No data available

29. Interpretations: Pile Sort Place the following food items in piles Broccoli Carrots Canned Beets Fresh Salmon Arugula Fennel Spam Ahi Tuna Canned Posole Niman Pork Sea Bass Canned Salmon

30. BOBO Sort Veggie Organic Canned Broccoli Fresh Salmon Canned Beets Arugula Sea Bass Spam Carrots Niman Pork Canned Salmon Fennel Ahi Tuna Canned Posole

31. Airstream Sort Veggie Meat/Fish Weird? Broccoli Fresh Salmon Canned Posole Fennel Spam Sea Bass Carrots Niman Pork Arugula Canned Beets Canned Salmon Ahi Tuna

32. Interpretive Signal model Attributes Prediction

33. Interpretive Signal Example Charisma Experience H MH ML L H G G G B MH G G G B G ML G B B B L B G B B

34. Experience Interpretation 75 % Correct H G G G B B G B MH G G G B G B G ML G B B B B L B G B B B

35. Interpreted Signals Statistical assumptions depend on diversity and sophistication. Independent? Positive correlation and bias result from similar models Amount of potential diversity and sophistication depend on complexity and dimensionality of outcome function

36. Binary Interpreted Signals Model Set of objects |X|=N Set of outcomes S = {G,B} Interpretation: Ij = {mj,1,mj,2…mj,nj} is a partition of X P(mj,i) = probability mj,i arises

37. Independent Interpretations P(mji and mkl) = P(mji)P(mkl) Probability j says “i” and k say “el”equals the product of the probability that j says “i” times the probability k says “el”.

38. Why Independence? We’re interested in independent interpretations because that’s the best people or agents could do in the binary setting. It’s the most diverse two predictions could be. Captures a world in which agents or people look at distinct pieces of information.

39. Claim: Independent, informative interpreted signals that predict good and bad outcomes with equal likelihood must be negatively correlated in their correctness.

40. Proof g b G = X B = 1-X G = Y B = 1-Y g b G = W B = 1-W G = Z B = 1-Z Prob row correct: (X+Y+2-(W+Z))/4 Prob column correct: (X+Z+2-(W+Y))/4 Prob both correct: (X+1-W)/4 (X+Y+2-(W+Z))(X+Z+2-(W+Y)) - 4(X+1-W) = (X-W)2 - (Y-Z)2 > 0

41. Claim: Independent, informative interpreted signals that predict good and bad outcomes with equal likelihood that are correct with probability p exhibit negative correlation equal to Note: p is bounded above by 0.75

42. The following assumptions are inconsistent with independent interpreted signals States = {G,B} equally likely Signals = {g,b} conditionally independent on the state.

43. Independent Interpreted Signals Interpreted signal: sj(mji) prediction by j given in set I Interpreted signals are independent iff sj(mji) and sk(mkl) are independent random variables.

45. Claim: Any outcome function that produces conditionally independent interpreted signals is isomorphic to this example.

46. ! Claim: For signals based on independent interpretations, the difficulty of the outcome function does not alter correlation other than through the accuracy of the interpreted signals

47. Proof g b g b

48. Proof g b g b Probability that each is correct and that both are correct is unchanged

49. Interpretive Signal Example Charisma Experience H MH ML L H G G G B MH G G G B G ML G B B B L B G B B

50. Experience Interpretation 75 % Correct H G G G B B G B MH G G G B G B G ML G B B B B L B G B B B