Meeting Data Collection Challenges of the Future

1. Meeting Data Collection Challenges of the Future James Griffith Ted Socha Thomas Weko Postsecondary Studies Division National Center for Education Statistics

2. Purpose of the Presentation To show how our “technological innovations” have responded to challenges facing sample survey and longitudinal study data collections.

3. Overview • Provide overview of postsecondary sample survey and longitudinal studies as prelude to … • Major challenge – Encourage participation of institutions and students • Technological innovations to meet this challenge • Future challenges, concluding remarks

4. Postsecondary Longitudinal and Sample Survey Studies • National Postsecondary Student Aid Study (NPSAS) • Beginning Postsecondary Students Longitudinal Study (BPS) • Baccalaureate and Beyond Longitudinal Study (B&B) • National Study of Postsecondary Faculty (NSOPF)

5. Primary Purposes NPSAS • NPSAS:08, includes 125,000 undergraduate and 13,000 graduate/first-professional students enrolled in 1,963 institutions • Describes how students and their families pay for postsecondary education and the role of federal student aid • Describes undergraduate and graduate student populations, including topics such as civic participation, community service, educational aspirations, life goals, disabilities • Provides rich database for postsecondary research and policy analysis, data gathered on over 1,100 variables • Responds to emerging federal policy interests, such as debt level, use of private loans, training in STEM majors, and state-level financial aid issues

6. Primary Purposes BPS • Includes about 18,500 first-time beginning students identified in NPSAS, followed 2 years and 5 years later • Provides data on student “flow” in and out of postsecondary education, such as persistence, transfer behavior, attainment • Gathers data on initial work experiences of those who obtain certificates or 2-year degrees B&B • Includes about 23,500 bachelor degree completers identified in NPSAS, followed 1 year, 5 years, and 10 years later • Provides important information on graduate education, work, and career through retrospective information on: -- postsecondary experiences -- paths taken to the degree award, time to degree completion • Focuses on “teacher pipeline,” interest in teaching, preparation, and job search

7. Chronology of PLSSS Studies BPS = Beginning Postsecondary Students Longitudinal Study B&B = Baccalaureate and Beyond Longitudinal Study

8. Major Data Collections NPSAS occurs in staged sampling periods requiring data collection at several levels: • Institution data collection • Student interview data • Other administrative data

9. Data Collections by Study

10. The Challenge

11. Declining participation of institutions and students • Missing data at the case-level and item-level • Complicated by … -- Diversity in access and use of technology across institutions (about 6,700) and students (about 19 million) -- Data privacy/security concerns Government = Misuse of study data Institutions = Risks of releasing student information Families = Privacy, identity theft -- Tight schedule for data collection, processing, and delivery – from lists of students gathered in May to data delivery in January -- Students as transient population GOAL – To maximize the usefulness and quality of the data vis-à-vis the challenge and complicating factors …

12. Institutional Data Collection Challenge: Maximize institution participation by acquiring enrollment lists for sampled institutions. Innovations: • Contacted early (fall contact for spring collection) • Established “Study Coordinator” (IRP as initial commitment) • Provided Help Desk (10-12 staff, Mon-Fri work hours) • Provided easy access through Website (information, secure login, CADE) • Ensured confidentiality / security (FERPA documents, IRBs) • Offered multiple options for participation (secure website, encrypted fax, hard copy)

13. Institutional Data Collection Challenge: Minimize institution nonresponse. Innovations: • Developed real-time monitoring of characteristics of institutions (IMS) • Performed bias analysis for < 85% participation overall and/or within strata, NCES standard • Made weight adjustment to reduce bias at the institution/unit level

14. Real-time Monitoring System We can see our response statistics in real-time. We then target “under-responding” institutions for participation in near real-time.

15. Institutional Data Collection Challenge: Maximize student record collection. Innovations: • Provided accessible Web-based instrument (CADE) • Provided Help Desk (Mon-Fri during normal work hours) • Negotiated reimbursement to “incentivize” participation

16. Example – Web CADE Student Record Abstraction

17. Student Record Abstraction Method – NPSAS:04 Web CADE preferred choice 1 The total represents the number of students sampled from institutions that completed computer-assisted data entry (CADE) and may include students who were classified as study nonrespondents. 2 Percentage of total number of eligible institutions/students. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

18. Institutional Data Collection Challenge: Maximize student transcript collection. Innovations: • Addressed confidentiality/privacy concerns through secure fax, encrypted email, FTP transfer • Negotiated reimbursement to “incentivize” participation

19. Example – Web Transcript Collection

20. Student Interview Data Collection Challenge: Maximize student interview collection. Innovations: • Redefined completed case • Employed multiple tracing vendors, e.g., CPS, NCOA, Telematch, Equifax, Experian,TransUnion • Offered response mode options fitting to the population (self-administered via Web, telephone, face-to-face) • Provided accessible Web-based instrument • “Incentivized” for early response • Provided Help Desk (7 days a week)

21. Cases Have Multiple Data Sources … Primary sources Institution records (CADE) 95% Student interviews 70% Federal aid applications (CPS) 60% Combinations of source All three primary sources 40% Two sources 50% One source 10% Additional sources Federal loans & Pell Grants (NSLDS) 50%

22. Allowing for Redefining a Case • Student case is a “complete” if valid data exist for: -- Student type -- Birth date or age -- Gender, -- And at least 8 of the 15 variables: • Dependent status, marital status, any dependents, income, expected family contribution, degree program, class level, first- time beginner, months enrolled, tuition, received financial aid, received non-federal aid, student budget, race, and parent education.

23. Difficult-to-Locate Students

24. Even so, Some Are Locatable

25. Example –B&B:08/09 Self-Administered Web Interview

26. Student Mode Choice for Interview Almost ½ respond via web. 1/3 receive incentive. Growing trend. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

27. NPSAS:04 Experience with Incentives

28. “Incentivized” Respondents $20 vs. $30 Web rises in preferred choice

29. Student Interview Data Collection Challenge: Minimize bias from case-level and item-level missing data. Innovations: • Made weight adjustment at case or student-level • Imputed item-level missing data by: -- Logical imputation or -- Statistical imputation with nearest similar neighbor donor

30. Who Responds and Who Doesn’t

31. Nonresponse Procedure • Nonresponse bias was estimated using statistical modeling. • Dependent variable = Responded or Not responded. • Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.: -- institution type; region; institution enrollment from IPEDS file (categorical); student type; FTB status; etc. • Weight adjustments = Coefficients for predictor variables used for weighting.

32. Example Weight Adjustments Already seen -- hard to locate and under- respond. So, “double-up” responding case.

33. Imputation Procedure • Logical – use other data sources to determine missing variable values for given case (e.g., substitute FAFSA for institution CADE). • Statistical –separate cases into dissimilar groups (using a defined set of variables) such that respective group members are alike. • Membership in final group determines donor candidates or “nearest neighbor.” • For cases having missing variable values, such values are “borrowed” from the “nearest neighbor.”

34. Item-Level Missing Data

35. Future Challenges to Data Collections • Increased concerns about security / confidentiality -- both institutions and students • Reduced access to students -- Increased liability concerns of institutions to release student-level contact information -- Greater cell phone usage • Raised respondent expectations– offering $ incentives

36. Products • Data for analysis Data analysis system, http://nces.ed.gov/dasol Restricted data file, http://nces.ed.gov/pubsearch/licenses.asp • Reports On methods … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006180 On content … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006186

37. How to be Informed of Products http://ies.ed.gov/newsflash

38. Contact Information Tom.Weko@ed.gov 202-502-7643 James.Griffith@ed.gov 202-502-7387 Ted.Socha@ed.gov 202-502-7383