RTI International is a trade name of Research Triangle Institute

1. Challenges of Designing and ImplementingMultimode InstrumentsJennifer Wine, Melissa Cominole, Ruth Heuer, and John RiccobonoMarch 15, 2006 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, North Carolina, USA 27709 Phone 919-990-8456 Fax 919-541-7014 e-mail mcominole@rti.org

2. Why Multimode? • NCES postsecondary studies – primarily CATI in the past, with some CAPI • NCES required the addition of the web self-administered option to help offset the decline in survey response rates • Characteristics of our postsecondary populations • Technologically savvy • Unconventional schedules • Cell phones replacing land lines • Time and money

3. Data Collection Methodologies Used for the Postsecondary Studies

4. Send data collection March 2006 announcement mailing Begin web , self - administered March 2006 data collection Begin telephone interviewing April 2006 ( CATI ) Begin field interviewing June 2006 ( CAPI ) End data collection October 2006 Typical Timeline for Data Collection Source: BPS FS

5. Multimode Design Challenges and Adaptations

6. Interview Specifications -- All Modes Source: NPSAS FS

7. Selected Interview Screen – Web Mode Source: NPSAS FS

8. Selected Interview Screen – CATI Mode Source: NPSAS FS

9. Multimode Design Challenges and Adaptations

10. Interviews Completed during Early Response Incentive Period* *When evaluated in the FT studies, early response incentives were shown to increase the likelihood of a response (p<.05).

11. Design for Telephone Prompting Experiment Source: BPS FT

12. Interview Participation, by Prompting Status *p<.01 Source: BPS FT

13. Prompting and Base Year Response Status *p<.05 Source: BPS FT

14. Average Call Count per Completed Interview, by Mode p<.0001 Source: BPS FT

15. Multimode Design Adaptations

16. Proportion of Time in the Interview and in Transit for Web and CATI Respondents WebCATI Total 34.4 34.8 On-screen 24.4 29.1* Transit 10.0 5.7* • Time on-screen • Time in transit *p<.0001 Source: B&B FS

17. Web Connection Speed

18. Multimode Design Adaptations

19. Help Desk Calls

20. On-Line Help Text Source: BPS FT

21. Rate of Help Text Access by Mode p<.0001 Source: NPSAS FS

22. Multimode Design Adaptations

23. Major/Field of Study Coder

24. Multimode Design Adaptations

25. Mode Differences in Item Reliability • Very few items with significant differences between modes, with no patterns evident • Some studies – NSOPF FT and BPS FT – showed no mode differences on any items

26. Multimode Design Adaptations

27. Percent of Indeterminate Responses, by Mode 5% • 38 interview questions, and over 26 percent of those in the finance section, had nonresponse rates of at least 10 percent • CATI/CAPI respondents provided an indeterminate response for nearly 2.5 percent of all responses. • Web respondents provided an indeterminate response for nearly 5 percent of all responses (p<.05). 2.5% Source: B&B FT

28. Response Conversion Text for Selected Items Source: B&B FS

29. Use of Conversion Text • Percent converted to an explicit response ranged from 0 percent to 100 percent in the B&B FS; comparable rates (18 to 100 percent) were observed in the BPS FT • Items less likely to convert: dollar amounts related to income, monthly expenses, and overall debt (28 to 73 percent) • Web respondents in the BPS FT were more likely than interview respondents to convert on two items – race and undergraduate loan debt; no other mode differences were observed

30. Next Steps? • Different incentive amounts at different points in data collection • Prompting difficult subgroups • Evaluate the use of help text relative to item-level nonresponse and reliability • Continue efforts to assess differences in data quality across modes

31. Contact Information Melissa B. Cominole RTI International 919-990-8456 mcominole@rti.org Slides and paper available at: www.rti.org/fedcasic