Findings from a National Longitudinal Study of the RISE Program

1. Findings from a National Longitudinal Study of the RISE Program P. Wesley Schultz California State University, San Marcos Presented at the 2009 NIGMS Council Meeting, January 23, 2009, Washington DC.

2. Findings from a National Longitudinal Study of the RISE Program Research Team: Mica Estrada-Hollenbeck Richard Serpe Anna Woodcock David Morolla Randie Chance Paul Hernandez Maria Aguilar Perla Rivas Victor Rocha Brian McDonald

3. Minority Training Programs NIH values diversity Underrepresentation is an important issue STEM and biomedical sciences Women in science Racial and ethnic representation (Black, Native American, Hispanic/Latino) Many NIH-funded programs promote research careers among members of underrepresented groups Primarily through NIGMS-MORE

4. Do These Programs Work? 1. National-level Data 80% increase in minority PhDs since 1977 62% increase in minority population during same period 2. Site-specific Data Each NIH program submits yearly progress reports, and longer-term evaluation results Typically track their student success, but what about a control group?

5. Do These Programs Work? 3. Large-Scale Evaluation Assessment of NIH Minority Research and Training Programs (Phase III). Science, 2006. Surveyed students from 47 programs (1970 -99) Substantial limitations in finding former students 5,371 identified participants, of which data was obtained from 739 (only 83 trainees funded as undergraduates--19 U*STAR, 31 Bridges, 33 COR)

6. Problems with Existing Data 1. Programs can “cherry pick” students who are likely to succeed - These students are likely to succeed without the program (growing new talent versus harvesting abilities) 2. No control group 3. Long-term evaluations outside of funding scope 4. No way to examine the “mechanisms” of success 5. Retrospective accounts can be biased

7. Overview: The Science Study Prospective, propensity matched control Longitudinal study of R.I.S.E. and MARC students Participants from 25 programs nationwide Twice yearly surveys from students Matched control group For each R.I.S.E. or MARC student, we find a similar student who does not go through the program Matching variables: ethnicity, gender, major, GPA, intention to become a scientist, enrollment level (LD, UD, Grad) Secondary matching: age, parental education, community college transfer, English as first language

8. Overview: The Science Study

9. Longitudinal Panel • Entering Panel N=1,228 (Fall 2005) • 457 R.I.S.E./MARC • 457 Matched Control • 157 Other funded (and 157 matched)

10. Longitudinal Panel

11. Survey Data Collection Data collected through web interface www.TheScienceStudy.com

12. Survey Measures • Psychological constructs • self-efficacy, scientific identity, task-goal orientation, value orientation, ethnic identity, integrative identity, satisfaction, self- esteem, etc. • Accomplishments/milestones • publications, presentations, research participation, conference attendance, degree completion, etc. • Perceptions of the academic environment • mentorship, belonging, financial support and MTP experience Response rates Survey coverage: 93.1% W0 Fall ‘05 100% W1 Spring ‘06 84.7% W2 Fall ‘06 78.9% W3 Spring ’07 78.0% W4 Fall ‘07 75.4% W5 Spring ’08 76.0%

13. Questions we can (ultimately) answer 1. Does participating in the RISE/MARC program cause an increase in the likelihood that a minority student will pursue a career in the biomedical sciences? (summative) 2. What are the types of activities that students involved in RISE/MARC programs are exposed to? 3. Are there some types of students who benefit more from the RISE/MARC program than others? 4. Are there elements of the RISE/MARC program that are linked with student success?(Research experience, faculty mentoring, financial support, motivated peers, resume of accomplishments)

14. Preliminary Findings • Basic effects (summative outcome) • Intention to become a scientist • Graduation rates • Applications and acceptance to doctoral programs • 2. Program elements (mediators) • Financial support, research experience, faculty mentorship • 3. Reasons for success (process) • Self-efficacy, identity, values

15. Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10) Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

16. C=1.58 Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10) Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

17. Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

18. Baccalaureate Graduation • W0 Juniors and Seniors. • MTP = Minority Training Program

19. Graduate School: Applications * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

20. Graduate & Medical School: Enrollment 1% 13% 10% * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

21. 2. Program Elements (Mediators) Invited Speakers & Workshops Paid Tuition & Stipends Tutoring Research Experience Support to attend professional conferences Faculty Mentorship

22. 2. Program Elements (Mediators) Note: Table reported at the 2007 NIH progress report meeting. Results based on survey responses from 25 RISE directors, and 457 RISE students.

23. 2. Program Elements (Mediators) • Mediation analyses of programs • Focus on three program elements • Research Experience • Faculty Mentorship • Financial Support (hours spent in employment off campus)

24. Mediation: Research Experience Research Experience a=.36 ** b=2.38 ** Intention to Pursue a Career in the Biomedical Sciences c=1.58** RISE Status c’= .68* Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001 Bootstrapped Indirect Effect: Mean = 0.68, CI99% = 1.31 to 0.45 Sobel: Z = 4.67, p < .001

25. Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded, and MATCH = students never funded by any program and enrolled on a RISE campus. Research is any research experience ever during undergraduate education. Intention to pursue career as biomedical scientist.

26. Mediation: Faculty Mentor FacultyMentor a=.32** b=.78* Intention to Pursue a Career in the Biomedical Sciences c=1.58** RISE Status c’= 1.33** Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001 Sobel: Z = 2.08, p < .05

27. Mediation: Financial Support FinancialSupport a=-.26** b=-.42ns Intention to Pursue a Career in the Biomedical Sciences c=1.60** RISE Status c’= 1.49** Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01 Sobel: Z = .85, ns Financial support is hours spent working off campus

28. PROCESS MODEL FOR RISE OUTCOMES Program Constructs Outcomes • Efficacy to Achieve • Scientific Self Efficacy • Self Esteem Intention to pursue graduate education in the biomedical sciences Research Experience Faculty Mentorship (Student-Faculty Contact) • Behavior • GPA • GRE/Other • Schools applied to/accepted • Degree(s) attained • Professional career choice • Identity as a Scientist • Salience • Commitment • Integrative (with ethnicity) Financial Support Value of Scientific Community Objectives and Lifestyle Cohort (peer support) • Motivation to Become a Scientist • Learning motivation • Performance motivation (approach) • Performance motivation (avoid) Conference (attending) • Accomplishments • Presentations • Publications

29. 3. Process of Influence

30. 3. Reasons for Success (process) Scientific Self-Efficacy (rule) b=.98** a=.28* Intention to Pursue a Career in the Biomedical Sciences c=2.55 ** Research Experience c’=2.28** Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01 Bootstrapped indirect effect: mean = .27, CI99% .06 to .56 Sobel: Z = 2.57, p<.01

31. 3. Reasons for Success (process) Scientific Identity (role) b=1.34** a=.56** Intention to Pursue a Career in the Biomedical Sciences c=2.55 ** Research Experience c’=1.80** Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01 Bootstrapped indirect effect: mean = .75, CI99% .33 to 1.23 Sobel: Z = 4.37, p<.001

32. 3. Reasons for Success (process) Scientific Values (value) b=1.08** a=.24** Intention to Pursue a Career in the Biomedical Sciences c=2.51** Research Experience c’=2.24** Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01 Bootstrapped indirect effect: mean = .27, CI99% .01 to .60 Sobel: Z = 2.43, p<.01

33. Key Points 1. NIH programs for undergraduates (RISE and MARC) have a strong effect on student intentions to pursue a career in the sciences. 2. RISE and MARC undergraduates have higher graduation rates, applications to doctoral programs, and enrollment in graduate programs (preliminary). 3. RISE works primarily because it provides students with research experience. 4.