Enhancing Understanding and Usability of the GSS through a Response Behavior Survey

1. Enhancing Understanding and Usability of the GSS through a Response Behavior Survey Presented to: Reg Baker, Market Strategies Presented by: Douvan Consulting Group (Julie de Jong, Erin Ferrell, Geon Lee, and Julie Sweetman) November 14, 2005

2. Outline • Introduction, procedures & concerns of the GSS • Introduction, procedures & concerns of the RBS • Recommendations • Improvements to the GSS • Sampling procedures and frequency of administration for the RBS • Improvements to the RBS design and questionnaire • Other avenues to collect feedback on usability of GSS • Conclusions • References

3. Introduction to the GSS and RBS Studies

4. Introduction to the GSS study • Funded by National Science Foundation and National Institutes of Health • Census of all science, engineering, and health-related master’s and doctorate-granting institutions in the US • Purpose is to collect numbers, funding information, and demographics of US graduate students and postdoctorates • 2004 survey: 12,000 departments in over 600 institutions • Field period: 15 months beginning every November • Few major changes since 1980

5. GSS Respondents • Understanding current respondents • Who are they? • Institutional coordinators • Department heads • Support staff • How are they identified? • Through the institutional coordinator for each reporting unit

6. Current Concerns in the GSS • Measurement Error in Numerous Forms • NSF/NIH may not know who all the respondents are • Some respondents may not be the best people for the job • Respondent records may not match the information requested by the GSS • Paper/electronic • Centralized/decentralized • Individual/aggregate

7. Response Behavior Survey (RBS) • Follow-up to the postdoctorate portion of the 2004 GSS • n = 1,500 • Evaluate how data is collected for the GSS • Delegation of responsibility for GSS • Respondent knowledge • Record-keeping practices • Usability of web instrument • Goal is to use RBS data to reduce measurement error in the GSS

8. Response Behavior Survey Concerns • Is the current RBS the most efficient way of evaluating the GSS? • Is the current RBS reaching the correct respondents? • Is the current RBS disseminated in the most efficient manner?

9. Recommendations

10. Circular Improvement Process(Deming, 1986 – 14 points) Improve GSS to improve to improve GSS RBS Improve RBS

11. Recommendations • Improvements to the GSS • Sampling procedures and frequency of administration for the RBS • Improvements to the RBS design and questionnaire • Other avenues to collect feedback on usability of GSS

12. Improvements to the GSS • Respondent identification Maria Johnson Institutional Coordinator Diane Smith Chair of Engineering Dept. Steve Hanna Chair of Biology Dept. Jim Lepkowski Chair of Survey Methods Dept. Jill Esau Program Coordinator

13. Respondent Identification, cont. • Collect name, e-mail address, and title from both the institutional coordinator and each individualrespondent responsible for completing the department portion

14. Respondent Identification at the Department Level • Sample GSS survey question to be completed before data can be submitted at the department level: Please provide the name, email address, and title of everyone in this department who contributed to the completion of this survey.

15. Respondent Identification at the Department Level • The department fills in the required information for each person involved in the completion of the survey: Please provide the name, email address, and title of everyone in this department who contributed to the completion of this survey.

16. Respondent Identification at the Department Level • After the names are filled out, the respondent must answer the final survey question in order for the data to be submitted: Please check the appropriate box next to the name, of the person who contributed most to the completion of this survey.

17. Recommended Sampling Frame for the RBS • Include only the respondent who completed most of the survey for their department • Advantages: • Allows selection of the people who had the most hands-on interaction with the GSS • Still permits flexibility in sampling other names if so desired • Disadvantages: • Does not necessarily capture the perspectives of people who had different types of involvement with GSS

18. Possible Solution for Sampling Procedure for RBS • Compile list of all GSS respondents and information about them (i.e. title, department, etc.) • After GSS field period ends, select sample of respondents from those who indicated that they completed most of the survey • All selected respondents receive e-mail notification at the same time • Follow-up with mailed letter to improve response rates as necessary (Kapolowitz et al, 2004)

19. Possible Solution for Sampling Procedure for RBS (cont.) • Advantages • Allows for straightforward stratification of respondents (i.e. by institution size, respondent title, department, etc.) • Guaranteed to get a GSS respondent • Inexpensive programming of web instrument • Disadvantages • Much time may pass between GSS and RBS administration • Respondents may forget about the GSS • Respondents may change jobs and/or email addresses

20. Recommended Solution for Sampling Procedure for RBS • As each respondent submits his/her data, add the respondent to the RBS sampling frame • Systematic cluster sampling of the RBS frame, on a rolling basis, throughout the entire GSS field period • For example, select every fifth respondent added to the frame • Selected respondents would receive an e-mail invitation to participate in the RBS • Follow-up with mailed letter to improve response rates as necessary (Kapolowitz et al, 2004)

21. Sampling for RBS: Stratification • Both Possible and Recommended Solutions: • Possible solution • Stratify by the number of respondents within institutions or by the number of departments within institutions • Recommended solution • Stratify on number of students in the university • Greatest indicator of variability in how respondents will complete survey

22. Illustration of Recommended Stratified Sampling Procedure Less than 2,000 students Every xth respondent 2,000 – 10,000 students Every yth respondent 10,000+ students Every zth respondent

23. Considerations while Implementing Recommended Stratified Sampling Procedure • Large and small universities will complete the survey in different ways • Aim is to stratify by homogenous groups • May want to sample universities and then people • Individuals will have differing context effects both between and within universities

24. Recommended Solution for Sampling Procedure for RBS (cont.) • Advantages • GSS experiences will still be fresh in the respondents’ minds • Systematic and stratified sampling • Ability to use weights in the analysis stage to account for any over-sampling in a rolling list • Guaranteed to get a GSS respondent • Disadvantages • May need advanced programming techniques to get a representative and stratified sample • May lead to increased cost

25. Sampling for RBS • Sample size considerations • Respondent burden (Phipps et al. 1995) • The RBS is not a short survey • Sample size dependent on stratification procedure

26. Sampling for RBS • Requirements for calculating the sample size • Compute an appropriate n for Simple Random Sampling (SRS) and adjust the nSRS by the design effect (deff) once a clustering and/or stratification design is chosen. • Obtain estimates for these values from previous RBS data • Statistics needed to compute the sampling size: • Number of departments within each university • Number of respondents within each department • Numbers may not be obtainable until after the modified RBS is administered • Number of students in each university • See Kish (1965) for additional guidance

27. Frequency of RBS Administration • Frequency of conducting RBS • Recommended frequency • Once every 2 years unless significant GSS or technical changes occur • Could depend on amount of changes made to GSS, and changes in technology and record-keeping over time • Cost-efficient way to obtain data for improvements, particularly due to the overlapping nature of the GSS from year to year

28. Improvements to the RBS • Additional questions to ask on RBS • Obtain more details regarding respondent record-keeping practices • Include question to determine the frequency with which records are updated • In the future, could use this information to tailor the GSS to the format of records for each institution, or for different types of institutions (e.g., large and small)

29. Adapt / Improve RBS Questions • Expert review of RBS instrument for both questionnaire design problems and interface usability • Laboratory experiments to guide questionnaire development and to evaluate and improve interface usability

30. Expert Review • A group of “experts” in different fields are needed to review the questionnaire • Review the survey itself, as well as usability of the instrument • Comments in open-ended form • Presser & Blair (1994) concluded that overall, expert review identified more problems than cognitive interviewing • Inexpensive solution for efficient questionnaire design

31. Laboratory Evaluations for Questionnaire Development • “Think aloud” interview • Combination of respondent’s thinking aloud and interviewer’s nondirect probing • Respondent Debriefing • Investigate whether respondents understand questions in the way that was intended by the survey designer • Behavior Coding • Occurs as the respondent complete the questionnaire • Identifies the location of problems in questionnaire • Uses a “Coder” and coding sheets De Maio et al. 1998, Willis et al. 1999

32. Analyses with RBS Data to Improve GSS • Examine responses to certain RBS questions and match to actual GSS data to measure reliability • “Did your institution have any post-docs?” • Repeat other survey questions to measure reliability • If GSS ≠ RBS, use regression models to understand predictors of reliability • E.g., size of institution or the title of the respondent who completed most of the survey

33. Analyses with RBS data, (cont.) • Examine responses to questions about usefulness of paper survey and modify paper version of GSS as necessary • Possibly provide links to PDF files for GSS for easy downloading • Analyze responses from open-ended questions to see if useful data is gained, use to develop coding categories to lessen respondent burden by creating closed-ended questions • Example: 2004 RBS asked open-ended question “Where do you get the CIP codes for your department”; responses could be used to develop pre-coded categories for the next RBS

34. Other Avenues to Collect Feedback on Usability of GSS • Lab experiments and expert review with the GSS questionnaire, both in questionnaire design and usability of web interface • Paradata (Couper 1998, 2005) • It is possible to learn a lot about respondent behavior without even asking respondents! • Current advances in paradata analysis are furthering usability • Compare respondent information to keystroke records, use of help screens, time of GSS completion, etc. • E.g., use paradata to examine which help screens are used most and to further improve upon those

35. Conclusions

36. Conclusions We see this process as circular, with continuous improvement as the goal Improvements to the RBS will improve the GSS, which will improve the RBS, etc. W. Deming’s cycle of continuous improvement http://www.asq.org

37. Conclusions, cont.Summary of Recommendations • Improve respondent identification in the GSS • Use improved identification to sample for RBS on a rolling, stratified basis, every 2 years • Use lab experiments and expert reviews to improve RBS questionnaire • Perform analysis with RBS data to further improve GSS • Use other avenues to collect feedback on GSS

38. Conclusions (cont.) • Implementing these recommendations will increase costs because of advanced programming techniques, questionnaire reviews, and analyses of RBS data • However, these increased costs are not unreasonable given the current budget • Web surveys are relatively inexpensive compared to other modes, allowing for the reasonable implementation of our suggestions (Schaeffer, 2001)

39. References

40. References • Couper, M.P. (2005). Technology trends in survey data collection. Social Science Computer Review, 23(4), 486-501. • Couper, M.P. (1998). Measuring survey quality in a CASIC environment. Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 41-49. • DeMaio, T., Rothgeb, J., & Hess, J. (1998). Improving survey quality through pretesting. Statistical Research Division Working Papers in Survey Methodology #98-03. Washington, DC: U.S. Bureau of the Census. (Download from http://www.census.gov/srd/papers/pdf/sm98-03.pdf). • Deming, W.E. (1986). Out of the Crisis. Cambridge, MA: MIT Center for Advanced Engineering Study. • Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004) A comparison of web and mail survey response rates. Public Opinion Quarterly,68, 94-101. • Kish, L. (1965). Survey Sampling. New York: Wiley. • Phipps, P.A., Butani, S.J., and Chun, Y.I. (1995). Research on establishment survey questionnaire design. Journal 0f Business and Economic Statistics, July, 337-346. • Presser, J. & Blair, J. (1994). Survey pretesting: Do different methods produce different results? In P.V. Marsden (Ed.), Sociological Methodology, 24, 73-104. Washington, DC: American Sociological Association. • Schaeffer, E. (2001). Web surveying: How to collect important assessment data without any paper. Office of Information & Institutional Research. Illinois Institute of Technology. • Willis, G., Schechter S., & Whitaker, K. (1999). A comparison of cognitive interviewing, expert review, and behavior coding: What do they tell us? Proceedings of the AmericanStatistical Association (Survey Research Methods Section). Alexandria, VA: American Statistical Association, 28-37.

41. Final Recommendations • Improve respondent identification in the GSS • Use improved identification to sample for RBS on a rolling, stratified basis, every 2 years • Use lab experiments and expert reviews to improve RBS questionnaire • Perform analysis with RBS data to further improve GSS • Use other avenues to collect feedback on GSS