Wisdom of Crowds and Rank Aggregation

1. Wisdom of Crowds and Rank Aggregation Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Brent Miller, Pernille Hemmer, Mike Yi, Michael Lee

2. Wisdom of crowds phenomenon • Aggregating over individuals in a group often leads to an estimate that is better than any of the individual estimates

3. Examples of wisdom of crowds phenomenon Galton’s Ox (1907): Median of individual weight estimates came close to true answer Prediction markets

4. Our research: ranking problems What is the correct chronological order? Abraham Lincoln Ulysses S. Grant time Ulysses S. Grant Rutherford B. Hayes Rutherford B. Hayes James Garfield Abraham Lincoln Andrew Johnson James Garfield Andrew Johnson

5. Aggregating ranking data ground truth group answer ? A B C D = A B C D Aggregation Algorithm A D B C D A B C B A D C A C B D A B D C

6. Task constraints • No communication between individuals • There is always a true answer (ground truth) • Unsupervisedalgorithms • no feedback is available • ground truth only used for evaluation

7. Unsupervised models for ranking data • Classic models: • Thurstone (1927) • Mallows (1957); Fligner and Verducci, 1986 • Diaconis(1989) • Voting methods: e.g. Borda count (1770) • Machine learning applications • Information retrieval and meta-search • e.g. Klementiev, Roth et al. (2008; 2009), Lebanon & Mao (2008); Dwork et al. (2001) • multi-object tracking • e.g. Huan, Guestrin, Guibas (2009); Kondor, Howard, Jebara (2007) Many models were developed for preference rankings and voting situations  no known ground truth

8. Unsupervised Approach latent ground truth ? ? ? ? Incorporate individual differences Generative Model A D B C D A B C B A D C A C B D A B D C

9. Overview of talk • Reconstruct the order of US presidents • Effect of group size and expertise • Reconstruct the order of events • Traveling Salesman Problem

10. Experiment: 26 individuals order all 44 US presidents

11. Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps = 1 = 1+1 = 2 Ordering by Individual A B E C D A B E CD E C D A B C D E A B True Order A B C D E

12. Empirical Results (random guessing) t

13. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson Each item has a true coordinate on some dimension

14. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson … but there is noise because of encoding errors

15. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A B C Each person’s mental encoding is based on a single sample from each distribution

16. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A A < C < B B C The observed ordering is based on the ordering of the samples

17. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A A < B < C B C The observed ordering is based on the ordering of the samples

18. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson Important assumption: across individuals, standard deviation can vary but not the means

19. Graphical Model of Extended Thurstonian Model Latent group means Individual noise level Mental representation Observed ordering j individuals

20. Inferred Distributions for 44 US Presidents George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) error bars = median and minimumsigma

21. Calibration of individuals t individual t distance to ground truth s inferred noise level for each individual

22. Wisdom of crowds effect t

23. Alternative Heuristic Models • Many heuristic methods from voting theory • E.g., Borda count method • Suppose we have 10 items • assign a count of 10 to first item, 9 for second item, etc • add counts over individuals • order items by the Borda count • i.e., rank by average rank across people

24. Model Comparison t Borda

25. Overview of talk • Reconstruct the order of US presidents • Effect of group size and expertise • Reconstruct the order of events • Traveling Salesman Problem

26. Experiment • 78 participants • 17 ordering problems each with 10 items • Chronological Events • Physical Measures • Purely ordinal problems, e.g. • Ten Amendments • Ten commandments

27. Ordering states west-east Oregon (1) Utah (2) Nebraska (3) Iowa (4) Alabama (6) Ohio (5) Virginia (7) Delaware (8) Connecticut (9) Maine (10)

28. Ordering Ten Amendments Freedom of speech & religion (1) Right to bear arms (2) No quartering of soldiers (4) No unreasonable searches (3) Due process (5) Trial by Jury (6) Civil Trial by Jury (7) No cruel punishment (8) Right to non-specified rights (10) Power for the States & People (9)

29. Ordering Ten Commandments

30. Effect of Group Size: random subgroups t

31. How effective are small groups of experts? • Want to find experts endogenously – without feedback • Approach: select individuals with the smallest estimated noise levels based on previous tasks • We are identifying general expertise (“Pearson’s g”)

32. Group Composition based on prior performance # previous tasks T = 0 T = 2 T = 8 t Group size (best individuals first)

33. Endogenous no feedback required Exogenousselecting people based on actual performance t t

34. Overview of talk • Reconstruct the order of US presidents • Effect of group size and expertise • Reconstruct the order of events • Traveling Salesman Problem

35. Recollecting Order from Episodic Memory Study this sequence of images

36. Place the images in correct sequence (serial recall) A B C D E F G H I J

37. Average results across 6 problems t Mean

38. Calibration of individuals t individual distance to ground truth s inferred noise level (pizza sequence; perturbation model)

39. Overview of talk • Reconstruct the order of US presidents • Effect of group size and expertise • Reconstruct the order of events • Traveling Salesman Problem

40. Find the shortest route between cities Individual 5 Individual 83 Optimal Individual 60 B30-21

41. Dataset Vickers, Bovet, Lee, & Hughes (2003) • 83 participants • 7 problems of 30 cities

42. TSP Aggregation Problem • Data consists of city order only • No access to city locations

43. Heuristic Approach • Idea: find tours with edges for which many individuals agree • Calculate agreement matrix A • A = n × n matrix, where n is the number of cities • aij indicates the number of participants that connect cities i and j. • Find tour that maximizes (this itself is a non-Euclidian TSP problem)

44. Line thickness = agreement

45. Blue = Aggregate Tour

46. Results averaged across 7 problems aggregate

47. Summary • Combine ordering / ranking data • going beyond numerical estimates or multiple choice questions • Incorporate individual differences • assume some individuals might be “experts” • going beyond models that treat every vote equally • Applications • combine multiple eyewitness accounts • combine solutions in complex problem-solving situations • fantasy football

48. That’s all Do the experiments yourself: http://psiexp.ss.uci.edu/

49. Predictive Rankings: fantasy football Australian Football League (29 people rank 16 teams) South Australian Football League (32 people rank 9 teams)

50. Predicting problem difficulty city size rankings t t distance of group answer to ground truth ordering states geographically std( s ) dispersion of noise levels across individual