Inside graduate admissions: Merit, diversity, and FACULTY GATEKEEPING

1. Inside graduate admissions:Merit, diversity, and FACULTY GATEKEEPING Julie Posselt, Ph.D. University of Southern California posselt@usc.edu @JuliePosselt

2. Doctoral Degree Attainment BY RACE, ACROSS FIELDS BY GENDER, SELECTED FIELDS Source: National Science Foundation, 2013, Doctorate Recipients from US Universities.

3. Empirical research suggested basic tensions. There is strong evidence... Institutions profess the importance of racial & gender diversity in... • Undergraduate admissions & other types of selection1 • Mission statements & websites2 1 Grodsky, 2007; Lamont, 2009; Karabel, 2005; Stevens, 2008 2 Morphew & Hartley, 2006; Osei-Kofi, et al., 2013

5. Empirical research suggested basic tensions. There is strong evidence... At the same time.... Institutions profess the importance of racial & gender diversity in... • Undergraduate admissions & other types of selection1 • Mission statements & websites2 • Work of graduate diversity officers is often stymied.3 • 2 of the 3 best predictors of admission are high GRE scores and selective college attendance.4 • Unconscious bias in responses to emails from prospective students.5 1 Grodsky, 2007; Lamont, 2009; Karabel, 2005; Stevens, 2008 2 Morphew & Hartley, 2006; Osei-Kofi, et al., 2013 3 Griffin & Muñiz, 2011 4 Garces, 2012; Attiyeh & Attiyeh, 1997 5 Milkman, et al., 2014

6. Why?

7. Research Questions: -How do faculty individually judge & collectively select applicants to highly ranked Ph.D. programs?-What assumptions about merit guide faculty judgment?-How do disciplinary norms shape faculty judgment? • Comparative ethnographic case study • 10 programs in 3 public & private universities • 85 interviews with professors & a few graduate students • 22 hours of admissions meeting observations in six of the programs Harvard University Press, 2016

10. Evaluative cultures explain apparent tensions between (definitions of) merit & (valuing) diversity. • Preference for specific criteria was rooted in beliefs about what they signal. Those beliefs were tightly linked to their own identities as scholars in highly ranked programs. • Preference for a process that is efficient and collegial. They wanted to quantify quality & minimize conflict. • In high-consensus fields, defended decisions to themselves and one another using disciplinary logics (def: shared disciplinary norms about epistemology/methods, theories & metaphors, and practical priorities) • In low-consensus fields, the strong role of individual preferences included four clear patterns of homophily (pedigree, cool, social mobility, international). • Ambivalence about organizational change, especially reforms related to diversity and equity. Faculty mindsets toward diversity, equity, and inclusion must be addressed.

11. Two-tiered review is used in most places.

12. Two-tiered review is used in most places.

13. PROBLEMS WITH THE FIRST CUT

14. Scripts about GRE SCORES • Theory of cultural & evaluative scripts (Goffman, 1959; Lamont, 2009) • GRE scores and Grades | Institutional prestige  Intelligence  • Belonging in an elite intellectual community • Risk profile

15. GRE SCORES & INTELLIGENCE • In interviews, 50% of the sample volunteered some idea about intelligence when asked what GRE scores signal (e.g., “sheer intellectual horsepower”, “native intelligence”) • In meetings, >50% of GRE mentions were what I classified as smart talk.

16. Risk aversion • Risk aversion as obligation & luxury • Examples of challenging risk aversion Prof. Bob: “Her GREs [of 690, 740, & 4.5] present a risk for her not succeeding” particularly because she “didn’t attend a top-rated university.” Prof. Lynn: “She may have undershot…This is an area that can be gendered…We have to be very careful here.” Prof. Bob: “All in all, it gives me doubt.” [Student ultimately waitlisted] PHILOSOPHY

17. Prof. Denise: “She might be a bet, but it could be a good bet… If we are going to increase diversity, these are the students we need to take seriously.” Prof. Jack: (Tentatively) “What’s the diversity?” Dept. Chair Nancy: “Family financial hardship.” [Committee agrees to move her forward, but discussion continues.] Dept. Chair Nancy: “It will be good for the whole faculty to take a look at her file. It seems pretty clear that she’s a risk, but if we’re going to increase diversity, we have to take risks.” Prof. Denise: “And she seems like a good bet.” [Student ultimately rejected after being waitlisted and attending recruitment weekend] LINGUISTICS

18. BLINDSPOTS ABOUT RISK • Informal assumptions about risk might not be entirely accurate. • Untested • Difficult to reliably predict PhD completion for populations who rarely enroll (i.e., problem of small n’s) • Validity of graduate entrance exams varies by test and graduate school outcome, but is consistently strongest for first year grad school GPA.1 • Student outcomes result from what they bring to the table AND from the educational experience we provide. • Women who did not complete the PhD had higher mean GPA than men who didn’t complete, but left programs in higher numbers.2 • 1 Kuncel & Hezlett, 2007 • 2 Lovitts & Nelson, 2000

19. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

20. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

21. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

22. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

23. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

24. GRE Quantitative Score (2006-2007) SOURCE: ETS, "Factors that can influence performance on the GRE general test 2006-2007”

25. GRE Test Disparities Are… • Technically not “bias” • Nearly independent of intended graduate major • Qualitatively unchanged when controlling for undergraduate GPA • Qualitatively the same for • GRE Subject test • SAT Math • 8th grade math achievement tests • 4th grade math achievement tests • A feature of standardized testing Miller, C., & Stassun, K. (2014). A test that fails. Nature, 510 (7504), 303-304.

26. Pop Quiz With all else equal, which folder do you admit? Folder A Folder B GRE-Q: 800 (perfect) GRE-Q: 740 (80%)

27. From ETS Guide to Use of Scores: “It is an inexact measure; only score differences that exceed the standard error of measurement of a given score can serve as a reliable indication of real differences in applicants' academic knowledge and developed abilities.” S.E.M. ~60 points (on old GRE scale, 200-800) 740 = 800 Source: http://www.ets.org/gre/institutions/scores/guidelines/

28. What does the literature say about THE GRE & predicting student success? Meta-analyses come to differing conclusions. • Morrison & Morrison, 1995; • Kuncel, et al., 2001; • Kuncel & Hezlett, 2010 • Orlando, 2005 Why? Studies draw upon different methods, different disciplinary and institutional contexts, and different populations. Only a few correct for attenuation bias; ETS continues to revise the test. What do we know? There has never been a true validity study: Denied students aren’t studied. Correlations vary by exam and by graduate school outcome (Kuncel & Hezlett, 2007). The longer the time between the test and the outcome, the weaker the validity. A flurry of discipline-specific studies: some find relationships with first year graduate school GPA, none with later outcomes, race or gender (despite score gaps)

29. r = 0.24 [ETS 0.27]; N = 1686 r = 0.33; N = 1743 r = 0.33; N = 1743 r = 0.02 [ETS 0.04]; N = 2133 r = 0.15; N = 2133 r = 0.15 [ETS 0.18]; N = 2133 Miller et al., submitted

30. r = 0.24 [ETS 0.27]; N = 1686 r = 0.33; N = 1743 r = 0.33; N = 1743 r = 0.02 [ETS 0.04]; N = 2133 r = 0.15; N = 2133 r = 0.15 [ETS 0.18]; N = 2133 Miller et al., submitted

31. Practical Significance?

33. UNDISCIPLINED ”HOLISTIC” REVIEW

34. Astrophysics committee:BLINDSPOTS in Holistic review

35. Juan: Is it enough to be a woman in science? (Discussion of how different perspectives might affect the community.) Prabhat: Lisa said she wants to be a role model because she never received explicit encouragement until recently. She wrote about the importance of providing active support, not just the absence of discrimination. Wayne: Shawna says she needs to develop self-confidence and overcome self-doubt. Juan: And then there’s Amy, who claimed to experience teasing and bigotry from her peers and a high school science teacher. She went to an all-women’s college so she could still study science. Chris: I’m less persuaded by that story. Maybe the teacher was young and inexperienced in handling high school boys. She might come to the program with an axe to grind. Juan: Either way, now she’s taking action, organizing a lecture series on women in science... We need to read between the lines on these things. (Person eventually nominated had started an astronomy outreach program and had letters of recommendation from familiar names.)

36. Prabhat: He grew up in a yurt in the Himalayas, was raised by his mom and grandma after his father died at an early age, and the next neighbors were two mountains over. He then found his way to a major U.S. public research university and has since started the only organization for the discipline in the Himalayan region. Jeff: But do we think he can succeed? [long pause] Prabhat: He’s the most amazing case we’ve ever seen. George: He would bring some personality to the department. I commit to look after him and fund him through the prelims…. He presents himself as quite intelligent. Chris: Excellent idea to give him a chance. [Student ultimately admitted and enrolled.]

37. IMPROVING HOLISTIC REVIEW

38. ”Beware the tyranny of best practices.”-Paul Courant

39. What is holistic review? • “…the consideration of a broad range of candidate qualities including ’noncognitive’ or personal attributes” (CGS, 2016, p. iii) • Three types: • Whole file: Considers all parts of the application. • Whole person: Considers many facets of the applicant. • Whole context: Considers the context in which the applicant achieved what they did.(Bastedo, Bowman, & Glasener, 2018) http://cgsnet.org/ckfinder/userfiles/files/CGS_HolisticReview_final_web.pdf

40. Contextualized Comprehensive Systematic

41. Comprehensive • Numerous and diverse criteria • Consider the whole person and the sum of their potential • Note that diverse perspectives improve scholarly work • Consider that socio-emotional skills are necessary for outstanding professional performance

42. Non-Cognitive Competencies • Social and emotional skills that we use to navigate life. • Measurable! • Results from decades of psychology research (developmental, social, and industrial-organizational) • Predict academic/job performance • Little, if any, group differences by gender and race • Orthogonal to cognitive measures (e.g., GPA, SAT/GRE) • Domain specific. Some will be specific to academia, grad school, and/or fields of study.

43. Correlating professional performance with admissions criteria and non-cognitive competencies Self-Management competencies correlate with clinical grade. “Cognitive ability and knowledge are threshold aspects of professional work, necessary but not sufficient for outstanding professional performance.” Victoroff and Boyatzis, J. Dent. Ed 77, 416 (2013) Achievement Orientation Adaptability Initiative Emotional Self-Control Trustworthiness Conscientiousness Optimism

44. Self Awareness Self Management Self-Confidence: A strong sense of one’s self-worth and capabilities. Accurate Self-Assessment: Knowing one’s strengths and limits. Emotional Awareness: Recognizing one’s emotions and their effects. Optimism: Persistence in pursuing goals despite obstacles and setbacks. Trustworthiness: Maintaining integrity. Achievement Orientation: Striving to improve or meeting a standard of excellence. Conscientiousness: Taking responsibility for personal performance. Adaptability: Flexibility in handling change. Emotional Self-Control: Keeping disruptive emotions/impulses in check. Initiative: Readiness to act on opportunities. Social Awareness Relationship Management Cultural Awareness: Respecting and relating well to people from varied backgrounds. Organizational Awareness: Reading a group’s emotional currents and power relationships. Empathy: Sensing others’ feelings and perspectives, and taking an active interest in their concerns. Service Orientation: Anticipating, recognizing, and meeting customers’ needs. Teamwork and Collaboration: Working with others toward shared goals and creating group synergy in pursuing collective goals. Communication: Listening openly and sending convincing messages. Building Bonds: Nurturing instrumental relationships. Conflict Management: Negotiating and resolving disagreements. Influence: Wielding effective tactics for persuasion. Change Catalyst: Initiating or managing change. Inspirational Leadership: Inspiring and guiding individuals and groups. Developing Others: Sensing others’ development needs, bolstering their abilities.

45. SYSTEMATIC Contextualized • Metrics in context • Note intrinsic error • Note societal patterns • Achievements in context • Distributions of opportunities relative to societal patterns • Achievements don’t always signal aptitude or ability • Admissions in context • How students aid your program’s identity/mission and broader goals • Base review on shared, predefined criteria with structured protocols, for efficiency & consistency. • Build in safeguards & checks to promote equity and limit biases. • Carefully select & train gatekeepers • Coordinate evaluation with recruitment and yield efforts

46. Rubrics: Comprehensive, Contextualized, & Systematic

47. Rubrics offer benefits that redress common drawbacks in many programs’ process. • EFFICIENCY is enhanced by expediting review, reducing faculty load. • STRUCTURE for a process in which many applicants are compared on multiple dimensions. • SPECIFICITY about what reviewers should be looking for may reduce implicit bias and prevent unseemly considerations from creeping in. • TRANSPARENCY about evaluation criteria is good for decision makers, their colleagues, and applicants themselves. • RELIABILITY across raters can be assessed. • ACCOUNTABILITY heads off charges that the process is unfair.

49. Developing a rubric:Identify qualities on which everyone should be evaluated. • Here, knowing your program mission can be very helpful. • Qualities can be broad if you want to leave room for individual interpretation & multiple ways for people to fulfill them • Or, qualities, can be narrowly defined if you want a highly structured process. • Examples: Research experience, Academic preparation, Clearly defined goals align with program expertise • Recommended: If you choose to require GRE scores, fold GRE scores and grades into a single judgment of academic preparation, to prevent anchoring bias and/or attributing small differences in scores/grades into large differences in overall quality.

50. Developing a rubric:Define how you will measure/ operationalize the qualities named above. • What does it means for an applicant to be outstanding, strong, acceptable, or weak on each of these? • The more concrete your definitions, the more consistent you can expect your judgments to be. • Recommended: Create space for comments to justify assessments; Leave open the possibility of naming unique strengths that merit special consideration. • Optional: Weight some qualities more than others.