Evidence-Based Public Policy New Tools from Experimental Economics

1. From design to adoption of matching-market clearinghouses: The role of experimentsAlvin E. RothHarvard University Evidence-Based Public Policy New Tools from Experimental Economics October 24, Chateau Cartier, Quebec

2. Experiments play a role • In diagnosing and understanding market failures, and successes • In designing new markets • In communicating results to policy makers

3. Matching markets • Labor markets: match workers and jobs • School choice systems: match students and schools • Organ transplant rules and procedures: match patients and organs/donors My colleagues and I have had the opportunity to take a number of such design projects from initial conception to adoption and operation.

4. Kidney Exchange • Exchange between incompatible patient-donor pairs • Donor D1 may not be able to donate to Patient P1, but maybe D1 can give to P2 and D2 can give to P1. • New England Program for Kidney Exchange—approved in 2004, started 2005. • Organizes kidney exchanges among the 14 transplant centers in New England • Ohio Paired Kidney Donation Consortium • National (U.S.) kidney exchange?

5. Canadian kidney statistics (from Canadian Organ Replacement Registry) • There’s a world-wide shortage of transplantable kidneys: • In 2004, there were 1,013 kidney transplants in Canada, 600 with deceased donor organs, and 413 living donors. • Patients waiting for a kidney transplant comprise 72% of the waiting list (n=2,872). • During 2004, 55 people died while waiting for a kidney. • In 2002, there were 4,839 new dialysis patients. The number of patients on dialysis followed in the CORR database as of December 31st, 2002 was 17,116. • In the U.S. 60,000 people on the waiting list, approx 8,500 deceased organ transplants, 6500 live donor transplants, 3500 annual deaths on waiting list

6. Matching students to schools • New school choice mechanism for New York City high schools • Approx. 100,000 students per year matched since 2003 (for entry in Sept. 2004) • First two years of operation very successful • Redesign of the Boston Public Schools choice mechanism • Replace the existing mechanism in 2006 (for entry into grades K, 1, 6, 9 in 2007).

7. Matching doctors to first positions in U.S. and Canada • The redesign in 1995 of the • U.S. National Resident Matching Program (NRMP) (approx. 23,000 positions) • Canadian Resident Matching Service (CaRMS) (1,400 Canadian medical grads, including 41 couples, 1,500 positions in 2005) • The two markets are coordinated: CaRMS operates two ‘iterations’ typically right before and after the NRMP • The redesign in 2005 of the fellowship matches for more senior physicians • The Gastroenterology fellowship match

8. Centralized clearinghouses • The NRMP and the CaRMS use a recently redesigned centralized clearinghouse • Roth, A.E. and E. Peranson, "The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design, American Economic Review, 89, 4, September, 1999, 748-780. • In the clearinghouse, doctors and residency programs each fill out Rank Order Lists of preferences, and are matched by a computerized algorithm.

9. Background to redesign of the medical clearinghouses • 1900-1945 UNRAVELLING OF APPOINTMENT DATES • 1945-1950 CHAOTIC RECONTRACTING--Congestion • 1950-197x HIGH RATES OF ORDERLY PARTICIPATION ( 95%) in centralized clearinghouse • 197x-198x DECLINING RATES OF PARTICIPATION (85%) particularly among the growing number of MARRIED COUPLES • 1995-98 Market experienced a crisis of confidence with fears of substantial decline in orderly participation; • Design effort commissioned—to design and compare alternative matching algorithms capable of handling modern requirements: couples, specialty positions, etc. • Roth-Peranson clearinghouse algorithm adopted, and employed

10. What makes a clearinghouse successful or unsuccessful? • A matching is “stable” if there aren’t a doctor and residency program, not matched to each other, who would both prefer to be. • Hypothesis: successful clearinghouses produce stable matchings. • How to test this?

11. Market Stable Still in use (halted unraveling) • NRMP yes yes (new design in ’98) • Edinburgh ('69) yes yes • Cardiffyes yes • Cambridge no yes • London Hospital no yes • Birmingham no no • Edinburgh ('67) no no • Newcastleno no • Sheffield no no • Medical Specialties yes yes (~30 markets, 1 failure) • Canadian Lawyers yes yes (Alberta, no BC, Ontario) • Dental Residencies yes yes (5 ) (no 2) • Osteopaths (< '94) no no • Osteopaths (> '94) yes yes • Pharmacists yes yes • Reform rabbis yes (first used in ‘97-98) yes • Clinical psych yes (first used in ‘99) yes So stability looks like an important feature of a centralized labor market clearinghouse.

12. The need for experiments • How to know if the difference between stable and unstable matching mechanisms is the key to success? • There are other differences between e.g. Edinburgh and Newcastle • The policy question is whether the new clearinghouse needs to produce stable matchings (along with all the other things it needs to do like handle couples, etc. ) • E.g. rural hospital question…

13. A matching experiment(Kagel and Roth, QJE 2000) • 6 firms, 6 workers (half "High productivity" half "low productivity") • It is worth $15 plus or minus at most 1 to match to a high • It is worth $5 plus or minus at most 1 to match to a low • There are three periods in which matches can be made:-2, -1, 0. • Your payoff is the value of your match, minus $2 if made in • period -2, minus $1 if made in period -1 • Decentralized match technology: firms may make one offer at any period if they are not already matched. Workers may accept at most one offer. Each participant learns only of his own offers and responses until the end of period 0. • After experiencing ten decentralized games, a centralized matching technology was introduced for period 0 (periods -2 and -1 were organized as before). • Centralized matching technology: participants who are still unmatched at period 0 submit rank order preference lists, and are matched by a centralized matching algorithm. • Experimental variable: Newcastle (unstable) or Edinburgh (stable) algorithm.

15. The Roth-Peranson algorithm is also used in other Canadian and American market clearinghouses • Postdoctoral Dental Residencies in the United States • Psychology Internships in the United States and Canada • Neuropsychology Residencies in the United States and Canada • Osteopathic Internships in the United States • Pharmacy Practice Residencies in the United States • Articling Positions with Law Firms in Alberta, Canada (Peranson runs National Matching Services, Inc. in Toronto)

16. The market for gastroenterologists Why did the market for gastroenterologists collapse? With what consequences? How to get it back on its feet? Science/persuasion/politics… Experiments play a role in each of these, and complement other kinds of economic investigation (in this case theory, surveys, analysis of historical data...)

17. Niederle, Muriel and Alvin E. Roth, “Relationship Between Wages and Presence of a Match in Medical Fellowships,” JAMA. Journal of the American Medical Association, vol. 290, No. 9, September 3, 2003, 1153-1154. Niederle, Muriel and Alvin E. Roth, “Unraveling reduces mobility in a labor market: Gastroenterology with and without a centralized match,” Journal of Political Economy, vol. 111, no. 6, December 2003, 1342-1352. Niederle, Muriel and Alvin E. Roth, “The Gastroenterology Fellowship Market: Should there be a Match?,” American Economic Review, Papers and Proceedings, 95,2, May, 2005, 372-375. C. Nicholas McKinney, Muriel Niederle, and Alvin E. Roth, “The collapse of a medical labor clearinghouse (and why such failures are rare),” American Economic Review, 95, 3, June, 2005,878-889. Niederle, Muriel and Alvin E. Roth, “Market Culture: How Norms Governing Exploding Offers Affect Market Performance,” working paper. Niederle, Muriel and Alvin E. Roth, “The Gastroenterology Fellowship Match: How it failed, and why it could succeed once again,” Gastroenterology, 127, 2 August 2004, 658-666. Niederle, Muriel, and Alvin E. Roth, “Re-starting the Gastroenterology Match,” letter, American Journal of Gastroenterology, vol. 100, no. 5, May, 2005, 1202-1203. Niederle, Muriel, Deborah D. Proctor, and Alvin E. Roth, “What will be needed for the new GI fellowship match to succeed?” Gastroenterology, forthcoming, 2005.

18. Early History of the Gastro Market: Unraveling and Congestion Gastroenterology (2-3 years): subspecialty of Internal Medicine (3 years). Before 1986: Decentralized Process. • Residents apply for fellowship positions, receive interviews, and offers from hospitals. • However, they experienced market failure in terms of unraveling (early, dispersed, exploding offers). • Interim attempts to solve the problem: “Setting guidelines for interviewing candidates and negotiating positions was tried, and it was unsuccessful.” (Congestion)

19. 1986-~1996: The Match (MSMP) Applicants (and hospitals) submit rank order lists over hospitals (and applicants). The MSMP uses a version of a hospital proposing deferred acceptance algorithm, which yields stable outcomes (since ’98 they use the Roth-Peranson algorithm). When such a match has a high participation rate, it simultaneously solves the timing, congestion, and incentives problems.

20. Gastro and health care reform Gastroenterology, in ’93-94 (the midst of health care reform): Manpower analysis. Meyer et al 1996: JAMA: • 25-50% reduction of GI fellows (over the next 5 years) endorsed as goal by GI leadership council. • Starting with summer 1996: 3 years of GI fellowship required for board certification eligibility

21. The demise of the Match Within 4 years the Match collapsed. After 1997 the match was effectively abandoned, and in 2000 it was formally abandoned

22. What have been the consequences of the demise of the match? • The market has become • Less thick in time and space • Earlier and more dispersed in time • Less mobility • No changes in wages

23. Timing of Interviews: Unraveling

24. Dates during which fellowship programs were making offers. Each program is represented by one of the horizontal lines, indicating the (maximal) dates during which it could have had outstanding offers (2005 survey data, n=44). (As of November 15, 11 (27%) programs had already finished making offers, 12 (25%) had not yet started, and 21 (48%) were in the midst.)

25. Effects of a Match: Mobility In the absence of a match, the market was more local, less national than when the match was operating. The decentralized market unraveled so that offers were earlier and more dispersed than during the years of the match. No effect on wages (Niederle and Roth, JAMA 2003.)

26. Why did the Gastro Match break down?And why are such failures so rare? Some Hypotheses • A centralized Match only works when there are more applicants than positions. • There may not be enough “high quality” applicants to fill high quality positions. • The match failed because of the shock that reduced the demand of positions below the supply. • The match failed because, when this shock occurred, fellowship programs were aware of it, but applicants were not. • Programs could update their priors when they saw how many applications they received.

27. How to sort among these hypotheses? • Each of them is consistent with the historical data. • And, since stable match failures are rare, there isn’t a good possibility of a cross-market comparison. • But in the laboratory, we can shock a market in different ways, and try to make a stable matching mechanism fail.

28. A simple experimental environment • 2 types of firms and workers; "High" and "low” productivity • Matching to a High type is worth 150 points + private value: in [-10,10]). • To a low type is worth 50 (+ private value) • There are three periods in which to match: -2, -1, 0. • Your payoff is the value of your match, minus 20 points if made in period -2, minus 10 points if made in period -1 • Decentralized match technology: firms may make one offer at any period if they are not already matched. Workers may accept at most one offer. Each participant learns only of his own offers and responses until the end of period 0. (no period 0 eq.) • After experiencing ten decentralized games, a centralized matching technology was introduced for period 0 (periods -2 and -1 were organized as before). • Centralized matching technology: participants who are still unmatched at period 0 submit rank order preference lists, and are matched by a centralized (stable) matching algorithm.

29. Experimental conditions • Different supply and demand (between subjects variable) • Shocks that change the long side of the market (within subject variable) • Decentralized and Centralized matching (within subject) • Different information conditions (symmetric or asymmetric between firms and workers) (between subject variable)

30. Experimental Results • Simple demand/supply imbalances have no effect. • Shocks disturb the match, especially when applicants are not aware of them. • Gradual adoption of the match…

31. Full Information

32. After the shock firms start making more early offers to workers. Workers: • Before the shock: high type workers eagerly accept early (-1) high offers. • After the shock: firms are in excess supply, workers have no big incentive to accept early offers. Workers only know this when they are informed of the shock, and indeed in that case accept early offers with much lower propensity. The ability of applicants to reject early offers that prevents unraveling. No other specialty experienced such shocks: Cardiovascular: from 1990 to 1998 the ratio of applicants to positions offered varied from a high of 1.6 to a low of 1.3. Pulmonary disease those ratios varied from a high of 1.5 to a low of 1.1, For Infectious disease (from 1994 to 1998) those ratios vary from a low of .68 to a high of .92.

33. Gastro market—preliminary advice to the Gastroenterologists • It appears that the collapse of the match was due to a shock, and the match could succeed if re-started. • This would increase the mobility of GI fellows. • Not all GI programs would do better: some small programs enjoy having a captive market.

34. Transition to a later market • In May 2005, the American Gastroenterological Association (AGA), the American College of Gastroenterology (ACG), the American Society for Gastrointestinal Endoscopy (ASGE) and the American Association for the Study of Liver Diseases (AASLD) decided to reintroduce a GI fellowship match, starting in 2006, for positions beginning in July 2007. • How to manage the transition? • Rates of participation • Concern among programs about whether their chief competitors will participate. • Change of dates to June 2006 (from as early as July 2005) • How to reassure programs that other programs will wait for Match?

35. Market Culture(Niederle and Roth 2005) • Different markets have different cultures: e.g. rules, norms, and expectations about how and when offers will be made, accepted, rejected. • In some markets, an exploding offer is easy to enforce and rely on, in other markets, making such offers would be frowned on, and other market participants wouldn’t help in enforcing the resulting contract. • Many markets have organizations that try to change and influence these norms in order to promote more orderly markets and control the timing of the markets.

36. Organizations concerned with how and when offers are made, accepted, rejected • Council of Graduate Schools (CGS): graduate admissions, • National Association for College Admission Counseling (NCAC): undergraduate admissions, • National Resident Matching Program (NRMP): entry level medical residencies, (also Canadian Resident Matching Service – CaRMS – and various regional matches in Britain) • Specialty Matching Services (SMS): advanced medical residencies and fellowships, • Association of Psychology Postdoctoral and Internship Centers (APPIC): clinical psychology positions, • National Association for Law Placement (NALP) for positions in law firms, • Judicial Conference of the United States and various ad hoc committees of judges for federal judicial clerkships, • Provincial Law Societies in Canada. • (see e.g. the Law Society for Upper Canada’s Procedures Governing the Recruitment of Students for Summer 2005 Positions in the City of Toronto • National Association of Colleges and Employers (NACE) for US college undergraduates, • NCAA: formerly for postseason college football bowls, now regulated by the Bowl Championship series (BCS), • National Panhellenic Conference for sorority matching • The Japan Federation of Employers’ Associations (Nikkeiren) for japanese university graduates • National Collegiate Athletic Association (NCAA) for recruitment of college athletes, and various drafts…

37. Which of these markets are unraveled? • Of these markets, it appears that markets in which transactions are made at early, uncoordinated times are markets in which there are both • Exploding offers • Binding commitments • Example: Graduate admissions has a nicely designed set of market rules, sent to all admitted students, that suppresses exploding offers (in part by facilitating “reneges” of premature acceptances).

38. “Students are under no obligation to respond to offers of financial support prior to April 15; earlier deadlines for acceptance of such offers violate the intent of this Resolution. In those instances in which a student accepts an offer before April 15, and subsequently desires to withdraw that acceptance, the student may submit in writing a resignation of the appointment at any time through April 15.

39. An experiment allows us to view different offer regimes in a controlled environment • 5 firms, 6 applicants, 9 periods. • In each period, a firm may make an offer to at most one applicant. Firms make offers, applicants decide upon the offers they receive. • Firms and applicants are assigned “qualities.” • If firm of quality x hires an applicant of quality y, both firm and applicant will receive a payoff of xy points each. • Firms’ qualities are simply their assigned participant number, 1,2,3,4,5. • Uncertainty about applicants’ quality is only resolved over time: • In periods 1, 4 and 7, each applicant receives a “signal,” an integer between 1 and 10, each equally likely. • In period 7, the relative ranking of the sum of the 3 signals determines the applicants’ quality. The applicant with the highest sum of 3 signals has a quality of 6… Ties are broken randomly. • Firms see all signals, applicants see only their own signals (as they become available over time) and their ranking in period 7. (Note that in this experiment the cost of early matching is bad matches due to uncertainty about quality.)

40. Experimental treatments: 3 “market cultures” Exploding offers: Firms can make exploding offers and acceptances are binding. Renege: Firms can make exploding offers, but applicants can renege on their acceptance, for a small fee (1 point). Open offers: Firms can only make open offers. (Many equilibria: One in which all matches are agreed upon inefficiently early. All environments have a perfect equilibrium with efficient late matching. But the late matching equilibrium is more fragile when offers are exploding and acceptances are binding.)

41. Timing of final offers (offers that were accepted and not reneged upon for the renege treatment) in terms of the number of signals that were observed.

42. In last five periods…

43. Not everyone loses…(so it can be hard to get consensus on reversing unraveling:)

44. So…How should Gastro transition from early to late? • October meeting of AGA Executive Committee • Agenda: adopt a resolution on the conduct of the market prior to the match, regarding early offers.

45. Proposed AGA resolution • 2. Acceptance of Offers - Before, Through and After the Match: Gastroenterology fellowship applicants are under no obligation to accept an offer prior to the Match. Furthermore, applicants are not bound by the acceptance of an offer prior to the Match. … • 3. Eligibility for the Match: All applicants are eligible to participate in the Match, whether or not they have accepted an offer prior to the Match.When an applicant is pressured by a program into accepting an offer prior to the Match, he may accept that offer and still remain eligible to participate in the Match. Since any agreement reached through the Match is binding, applicants should only submit to the Match a Rank Order List of programs they definitely prefer to any offer accepted before the Match. Applicants who have accepted an offer before the Match, and received a position in the Match, must announce their resignation to the program that made the early offer on the same day that they learn the Match result. … Applicants who accepted an early offer, but were not matched, must honor their original agreement. • Applicants who receive several early (pre-Match) offers to which they are pressed to reply before the Match, can at any time accept the one they prefer, and should contact the programs whose offers they reject, whether they have accepted them beforehand, or were simply considering the offer. • While applicants maintain their eligibility for the Match, even if they are pressured to accept offers before the match, such early offers violate the intent of this resolution. When applicants and programs are each others’ first choice, the Match will automatically match them when they submit those preferences on their Rank Order Lists. When all participants comply with the spirit of this resolution, all positions will be offered through the Match, or after it.

46. AGA Resolution adapted to current NRMP policies • 3. Eligibility for the Match and Participation in the Match: All fellowship positions and applicants are strongly encouraged to participate in the Match.Offers prior to, or outside of, the Match, whether made by a program participating in the Match or otherwise, are a violation of this resolution and are not condoned. All applicants are eligible to participate in thematching process by registering for the Match to interview and consider match-participating programs; however, an applicant who accepts a position prior to, or outside of, the Match must either resign that position if he/she wishes to submit a rank order list of programs or withdraw from the Match prior to the rank order list certification deadline. In other words, an applicant may submit a rank order list of programs as long as he/she does not hold any other positions outside of the Match at the time of the rank order list certification deadline, which is the first week in June. No program may withdraw a position after the quota change deadline in order to offer that position outside the matching process. Any agreement reached through the Match is contractually binding. • Applicants who receive several offers prior to, or outside of the Match, to which they are persuaded to reply before the rank order list certification deadline can at any time accept the one they prefer and should contact the programs whose offers they reject, whether they have accepted them beforehand or were simply considering the offer.

47. Conclusions: • Markets don’t always spring up like weeds, some of them are hothouse orchids that need care and attention.

48. Experiments have multiple roles to play • They allow us to investigate questions that the field data cannot answer • E.g. why did the GI match fail in ’96? • They allow us to investigate hypotheses suggested by the field data • E.g. is stability the key to understanding the successful and failed British medical matches? • E.g. does the ability to renege on early acceptances reduce exploding offers and unraveling? • They play a role in the considerable amounts of discovery, demonstration, and persuasion that are necessary to coordinate market participants to move from one equilibrium to a better one.