College Ambition Program Navigating the College-Going Process

1. College Ambition Program Navigating the College-Going Process Improving Postsecondary Outcomes for Low Income Students: The College Ambition Program Barbara Schneider Christopher Khawand Justina Judy Michigan State University www.collegeambition.orgFunded by the National Science Foundation

2. The College Ambition Program (CAP) • Program Goals: • Increase attendance at 2-year and 4-year postsecondary institutions • Improve postsecondary persistence and completion • Reduce student under-matching • Increase student success in Science, Technology, Engineering, and Mathematics (STEM) careers • Program Components: • Mentoring • Course planning • Financial aid assistance • College visits

3. Motivation for Intervention • The CAP model is based on: • fifteen years of empirical work that has followed over 1,000 adolescents from middle school into adulthood • a four year randomized trial • extant literature on determinants of college enrollment • Most students aspire to attend college but lack an understanding of the educational requirements for a given career path • 90% of 9th graders expect to attend college (Aud et al., 2010), but matriculation rates (and even high school completion rates) are not on par

4. Human Capital Investment Framework • Economic theory posits an optimal level of postsecondary education for each student • In reality, we expect students to mismatch in postsecondary choices • Credit access constraints • Imperfect foresight • Gap between ability and motivation • Misalignment between ambitions and concrete plans • Most of these are problems of information that can be solved by one-on-one contact with an expert

5. Intervention and Research Design • Treatment schools: • “CAP Center” • Site coordinator • Volunteer mentors and tutors (STEM emphasis, honors students) • Sign-in sheets and mentor/tutor contact logs • Surveys • Computer-based student survey • Teacher survey • Parent survey (forthcoming) • Senior Exit Survey • College visits • Parent events (e.g. Financial Aid night) • Control schools: • Surveys & administrative data collection

6. Data Collection

7. Preliminary Results • First year of data available (2 treatment schools, 2 matched control schools) • Dependent variable: college matriculation conditional on college ambitions • Propensity score-based sample selection with students in ELS:2002 combined with regression approach finds positive effect (+5%-11%)

8. Methodology

9. Detecting Effect Sizes • Quasi-experimental study • Do not get the full benefits of a randomized controlled trial • Schools and students may self-select for participation, biasing estimated effects • Sample size small • Funding only allows at most 24-36 schools to be surveyed, with half of those being treatment schools • Substantial heterogeneity between schools • Have to use appropriate regression and matching techniques to reduce bias and improve power

10. Sample Selection Framework • Strategic sample selection using matching technique (Stuart, Cole, Bradshaw, & Leaf, 2011; Hedges & O’Muircheartaigh, 2011; and Tipton, 2011) • Schools matched through Mahalanobis distance on important covariates: • District-level: • % over 25 holding BA • Per capita income • School-level: • Graduation and dropout rate • % Free and Reduced Price Lunch • Student ethnic composition • Performance on state assessment (Michigan Merit Exam) • Outcome variable: % enrolling in postsecondary schools

11. Preliminary School Matches • Mahalanobis matching • Provides measure of “distance” between schools on covariates, weighted by correlation between the covariates • Could also weight by partial effect from regression to “prioritize” factors • Direct covariate matching is robust to functional form • If matching is done perfectly with no omitted variables, a simple difference-in-means test would yield an unbiased estimate of the Local Average Treatment Effect on Treated Schools • Realistically: • Residual post-match differences in covariates must be corrected for in regression, which is less favorable • There will be omitted variables, but we will be able to make inferences about the magnitude of bias

12. School Match Examples

13. Advantage of Mahalanobis Metric

14. Identification Strategy #1: School-Level Panel Data Approach • School-level analysis #1 • Construct panel data set based on NCES Common Core Data and other sources (MDE, etc.) • Estimate generalized linear models with school and district fixed effects • Use between- and within-school variation determine intervention’s causal effect on postsecondary matriculation, persistence, and STEM participation • Takes treatment high schools and compares them with all high schools in Michigan over 3-5 time periods

15. Appraising Bias in Estimates of Effect Size • Can run school-level regressions on percent of students going to college in pre-treatment years 2007-2008 and 2008-2009 • In a cross-section of 600+ schools, without including a lagged outcome variable, R2> 0.7 • Selection on observables can inform us about selection on unobservables (Altonji, Elder, & Taber, 2005) • We expect that selection into CAP on unobservables is no larger than selection on observables, because our estimation already includes the most important covariates • This allows us to bound the potential bias • R2 of linear regression serves as a measure of precision for the AET method

16. Identification Strategy #2: School-level Matching Approach • School-level analysis #2 • Use panel data set and match treatment to control schools across time periods (one treatment matched to many controls) • If covariates are individually balanced between treatment and control, can estimate LATT • Better: look at difference in outcome between treated units and matched control groups, conditional on difference in covariate values • Alternatively, can use synthetic controls approach • Abadie & Gardeazabal (2003) • Construct a synthetic school using linear combination of other schools’ covariates • Creates a counterfactual match for each treated school – ideally identical, but untreated

17. Identification Strategy #3: Multi-Level Analysis • Use aforementioned school matches • Match students within treatment/control on their propensity to participate in the program (to reduce heterogeneity of school-level effects) • Student-level analysis allows us to estimate heterogeneous treatment effects • Can factor in intensity of student dosage • Contact and activity logs recorded at each treatment site • Number of visits, reason for visit, duration from • AET bias analysis also applicable

18. Comments • Too early in study for conclusive results • Initial findings are positive, especially from qualitative case studies • Effect of intervention is expected to grow over time • Implementation will improve • Students will get exposure to CAP in earlier grades up through their senior years • Success would set a good precedent for identifying effect sizes in similar funding-constrained interventions

19. Potential Policy Implications • Students may not be reaching their optimal level of (educational) human capital investment because of information limitations • Intervention can address this inefficiency • Improve financial aid access • Inform students of the benefits of college education • Create structured plan for meeting postsecondary attainment hurdles • Provide direct assistance in the college application process • Findings from the intervention can be used to create low-cost, sustainable solutions that are individually implementable in schools

20. www.collegeambition.org • Information for educators, mentors, parents and students • Interactive College Checklist • Links to scholarships and college information sites • Links to specific school subject help sites • The CAP research design • Benefits of pursuing college