Enhancing Postdoctoral Success: Evaluating the Impact of Working Conditions on Outcomes

1. The Productive Postdoc:Do Working Conditions Affect Outcomes? Geoff Davis Visiting Scholar and Survey Principal Investigator Sigma Xi, The Scientific Research Society gdavis@sigmaxi.org

2. Improving the Postdoctoral Experience • Many calls for changes to the postdoc • National Academies, AAU, NPA, etc • Big question: What, if anything, works?

3. What Works? • Changes have costs (money, time) • Do benefits justify investments? • What should priorities be? • What gives the biggest bang for the buck? • These are empirical questions

4. Our “Experiment” • Postdoc administration takes place largely at the level of the PI • Tremendous variability in conditions from lab to lab • Recent, limited introduction of new practices • Natural experiment • Ask postdocs about their working conditions • Ask about how well they are doing • Find conditions associated with positive outcomes

5. Sigma Xi Postdoc Survey • Ran a big web survey • Contacted 22,400 postdocs at 47 institutions • ~40% of all postdocs in US • Overall response rate: 38%* • (*See tech report for details)

6. Our Sponsor The Alfred P. Sloan Foundation Alfred P. Sloan Michael Teitelbaum

7. Additional Support Werthheim Fellowship, Harvard University

8. Partner Organizations • National Postdoc Association • Science’s Next Wave • NBER/Sloan Scientific Workforce Group

9. Sketch of Our Analysis • Create measures of inputs (working conditions, demographics, etc) and outcomes • Build linear models to test hypothesis that inputs have an impact, gauge magnitude of impact (if any)

10. How Do We Determine Success? • Ideal: track people down in 10 years, see what they are doing / have done • Problems: • Very expensive • Takes 10 years to learn anything • Driving via the rear view mirror • Instead, look at immediate proxies for longitudinal data

11. Outcomes • What makes for a “good” experience? • No single “best” measure • Different people want different things • Create collection of outcome measures • Look at impact of inputs on each

12. Subjective Outcome Measures • Subjective success measure • Overall satisfaction, preparation for independent research, quality of training in research / teaching / management • Advisor relations measure • How is your advisor doing? Is s/he a mentor? How would s/he say you are doing? • Generate numerical scores by summing Likert scored answers

13. Objective Outcome Measures • Absence of Conflict/Misconduct • Has postdoc had a conflict with advisor? Has s/he seen misconduct in the lab? • Productivity • Rate at which papers submitted to peer reviewed journals

14. Outcome Measure Distributions

15. Outcome Measure Details • Correlations all fairly low • Subjective success and advisor relations ~0.45 • Other pairwise correlations all < 0.2

16. Our Explanatory Variables • Model outcomes as function of explanatory variables • Field of research • Institution • Basic demographic variables • Sex • Citizenship • Minority/Majority Status • Type of degree (MD vs PhD) • Total time as a postdoc • “Working Conditions”

17. “Working Conditions” • How do we measure working conditions? • Inspiration comes from various calls for changes • Look at rate of implementation

18. Recommended Changes • 5 broad classes of recommended changes • Pay people more • Fellowships rather than assistantships • Better benefits • More structured oversight • Transferable skills training

19. Measures of Working Conditions • Salary measure • log(annual salary), full-time people only • Independent Funding measure • Dummy variable, 1 if fellowship, 0 otherwise • Benefits measure • Count of different benefits received (health insurance, retirement plan, etc)

20. Structured Oversight • Structured Oversight measure • Count of administrative measures in place • Individual development plans • Formal reviews • Policies (authorship / misconduct / IP / etc) • Letters of appointment • High values = lots of structure, low = little

21. Training • Transferable Skills Training measure • Count of areas in which postdoc reports receiving training • Grant writing, project/lab management, exposure to non-academic careers, negotiation, conflict resolution, English language, etc • High values = training in lots of areas • Low values = no training in lots of areas

22. Working Conditions Distributions

23. Working Conditions Details • Again, correlations all fairly low • Structured oversight and skills training ~0.30 • Other pairwise correlations all < 0.15

24. What Has Biggest Impact? • Who is most satisfied, most productive, etc? • People with • Independent funding? • High salaries? • Lots of benefits? • Lots of structured oversight? • Lots of types of transferable training?

25. Simple Analysis • Crude analysis: compare satisfaction, productivity, etc for people in appointments with • Fellowships / other funding • High / low salaries • High / low benefits • High / low structure • High / low training

26. Independent Funding

27. Salary

28. Benefits

29. Structured Oversight

30. Transferable Skills Training

31. Regression Coefficients

32. Take Home Message #1 • Structured oversight and transferable skills training make a big difference

33. Causality? • We have correlation. Is there causation? • Psych literature gives reasons to believe in causation • Alternative explanations • Structure and training attract people who are intrinsically more satisfied / productive / successful • Structure / training correlate with some other unobserved factor • Advisors are effective managers / have more resources • Postdocs take more initiative / are better organized / etc

34. Causality? • 2 classes of explanation • Structure/training attract intrinsically more productive people • Structure/training directly cause productivity or are indicators for some causal mechanism (Some combination of 1 & 2 also possible) • Should be able to differentiate between 1 & 2 by looking at people with multiple appointments

35. Intrinsic vs. Time-Localized

36. Causality? • Add in terms that allow for change in slope of papers(t) curve starting at beginning of most recent postdoc • Equivalent to adding interactions with ratio (months in current postdoc / total months as postdoc) to regression model • Training appears to have a time-localized effect • Other inputs ambiguous

37. Don’t Pay Postdocs? • Not saying postdocs shouldn’t be paid! • Hard to attract US students to science if you don’t pay them • Maslow’s hierarchy of needs • Must meet basic physical security needs first • Living wage, basic benefits • More nuanced interpretation of data: beyond a certain threshold, structure and training matter more than compensation • Institutional “postdoc tax” to support service provision?

38. More Details • Look at individual components of structure and training measure • What specific measures have the greatest impact?

39. Impact • One measure appears to have significant impact all 4 outcomes: • Research / career plans • Written plans • Plans that spell out what both postdoc and PI will do • Advocated by FASEB, National Academies

40. Plans • Compare those with such a plan to those without: • Much less likely (~40%) to be dissatisfied • Much less likely (~30%) to have conflicts • After controlling for field, institution, demographics: • Submitted ~14% more papers for publication

41. Why? • Plans: • Expectation setting device • Postdocs without plans were much more likely to report PI had not lived up to expectations • Contract • Research shows that people are more likely to live up to explicit (esp. written) commitments • Forces postdocs to take responsibility for their careers early • More time to take advantage of training opportunities • Time management device • Mechanism for focusing effort

42. Take Home Message #2 • Individual development plans make a big difference

43. Additional Measures • Several other measures show concrete benefits: • Teaching experience • Exposure to non-academic careers • Training in proposal writing • Training in project management • Training in ethics

44. Policy Implications • For postdocs, more effective to invest additional dollars in management than in salaries • Management at all levels: • Infrastructure for institutional oversight / training • Management training for PIs • Management training for postdocs

45. Further information • More information at http://postdoc.sigmaxi.org • Workshop (with NPA) in January 2006 • Contacts • Geoff Davis, PI, gdavis@sigmaxi.org • Jenny Zilaro, Project Manager, jzilaro@sigmaxi.org

46. Extra Material

47. End Products • Sigma Xi: • Highlights in May/June issue of American Scientist • Tech reports (2 out now, more to come) • Scholarly paper this fall • NPA: Analyses of various topics • NBER SEWP • Workshop in January 2006

48. Aside: Postdoc Definition • Half a dozen different definitions • AAMC, AAU, FASEB, NAS, NSF • BUT if you read and compare them, they all say the same thing • Only substantive difference is that FASEB includes narrow subset of clinical fellows • (We excluded them from this analysis) • Most people don’t fully satisfy definition anyway

49. Postdoc Definition • The appointee has a PhD or equivalent degree, • the degree was received recently, • the appointment is temporary, • the purpose of the appointment is training for a research career, • the appointment involves substantially full-time research or scholarship, • the appointee is expected to publish the results of his or her research, and • the appointee works under the supervision of a senior scholar or a department in a university or research institution.

50. Survey Non-Response • 30-second summary of non-response analysis: • Non-citizens and African Americans appear to be slightly under-represented • No evidence of bias based on level of satisfaction (respondents not overly disgruntled)