Survey nonresponse - uncomfortable questions (and some answers) for management researchers

1. Survey nonresponse - uncomfortable questions (and some answers) for management researchers Christiane Spitzmüller, Ph.D.Associate Professor University of Houston, Houston, TX, USA Visiting Professor Lagos Business School, Lagos, Nigeria

2. Survey nonresponse • Organizational survey research and nonresponse threat, overview of studies • Methods for nonresponse analysis • Intentions/Population profiling methodology • Demographics, attitudes and personalities of nonrespondents • Survey response – complaint behavior or good citizenship? • A social exchange framework for studying nonresponse. • Stressors, strain and survey nonresponse • Web survey nonresponse and technology acceptance • Meaningfulness of response rates • Future directions

3. So, how much of an issue is survey nonresponse really? • Do you take response rates into account when evaluating study quality as a reader/reviewer/editor? • What do you consider a threshold for an acceptable response rate? • Meta-analytic work (Ansel et al., 2010): 52% average, numerous moderators at work (consumer/employee, university sponsored, incentive use) • Response rates have dropped in comparison to prior meta-analytic studies (reporting 57%)

4. Who are nonrespondents? Why do they fail to respond? How do we find them?

5. Overview of studies – understanding nonresponse • Online survey response (PPsych, 2006, unpublished manuscript) • Survey response as social exchange (JOB, 2006) • What do response rates mean? • Survey response as complaint or OCB (IJSA,2008) Organizational and survey characteristics • Stressors and survey response behavior (JOHP, 2008) • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

6. Overview of studies – understanding nonresponse Organizational and survey characteristics • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

7. Method Limitations • Archival Approach • Wave analysis • Follow-up approach • Intentions-methodology (Rogelberg et al. 2000) • Population Profiling • Limited to study of variables in database, mostly demographics • Late respondents are not really nonrespondents • Problematic: Complete follow-up sample of nonrespondents hard to obtain • Discrepancy between percentage of individuals with intentions to comply, and those who actually comply • Requires near 100% initial response rate Methods of examining characteristics of nonrespondents (Rogelberg et al., 1998)

8. Population profiling Time 1 Time 2 • Surveys a “captive audience” in an organizational setting • Survey contains variables for nonresponse analysis • Intentions to complete future surveys additional survey component • Contact information is collected • Participants from Time 1 are contacted with follow-up survey Active Nonrespondents Passive Nonrespondents Respondents • Intentions to respond at Time 1 • Actual Response at Time 2 • No intentions to respond at Time 1 • No response at Time 2 • Intentions to respond at Time 1 • No response at Time 2

9. Who are nonrespondents? Demographic and attitudinal profiles • Majority of nonresponse research has been conducted on demographic variables (Rogelberg & Luong, 1998) • Respondents have higher education levels than nonrespondents (Gannon, Northern & Carroll, 1971) • Females are more likely to respond than males (Pucel, Nelson & Wheeler, 1971). • Older employees are more likely to respond (Rogelberg, 2000) • Findings about attitudinal variables are inconsistent, clarification through the use of population profiling (Rogelberg et al, 2003)

10. Personality differences between respondents and nonrespondents • Inconsistent results in earlier studies (Rogelberg & Luong, 1998) • Better balanced? (Bebbington, 1970) • More gregarious? (Ognibene, 1970) • Higher in aggression, dependency and need for order? (Lubin, Levitt, & Zuckerman, 1962) • More cooperative conformist, tolerant, intellectual? (Vincent, 1964)

11. Passive Nonrespondents Active Nonrespondents Respondents • Highly conscientious • Highly agreeable • Highly agreeable • Less conscientious than respondents • Less conscientious and agreeable than passive nonrespondents and respondents Big Five and nonresponse • Rogelberg et al., 2003

12. Overview of studies – understanding nonresponse • Survey response as complaint or OCB (IJSA, 2008) Organizational and survey characteristics • Methodology and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

13. Survey response behavior as a helping or venting behavior? • “…taken together, it appears that completing an employee attitude survey can be thought of as an OCB if one believes that data collected from the survey, in the aggregate, work to improve organizational functioning.” (Rogelberg, Luong, Sederburg & Cristol, 1999) • Majority of theoretical models explaining nonresponse rely on OCB models • Survey nonrespondents use organizational surveys to vent or complain about their working environment • Models on survey-based consumer complaint: dissatisfaction positively related to complaint • Spitzmueller, Glenn, Barr & Tunstall, IJSA, 2008

14. Similarities between respondents and good organizational citizens? Variables Citizenship behavior Nonresponse research • Nonresponse relates to conscientiousness and agreeableness • Personality • OCB relates to higher conscientiousness and agreeableness • Older, more educated, and females more likely to respond to surveys • Older, more educated, and females more likely to engage in OCBs • Demographics • Satisfaction, commitment relate to OCB • Satisfaction, commitment relate to nonresponse • Attitudes

15. Nonresponse and OCB facets • Five-dimensional OCB framework (Organ, 1988, 1997) consists of • Altruism • Conscientiousness • Courtesy • Civic Virtue • Sportsmanship (no hypothesis) • Hypotheses: OCB facets relate to survey response behavior, active nonrespondents report lower levels of OCBs than passive nonrespondents and respondents

16. Method • 622 junior and senior undergraduate business students participated during class time, provided contact information • Collaboration with UH Institutional Research Division • Near 100% response rate • Average age = 23 (sd=4.52) • 55% work 20 hours or more per week

17. Measures • Step 1 survey: • Step 2 survey: Survey designed by Institutional Research asks for participation and students’ contribution to development of university resources.

18. OCBs and survey response behavior • Conducted one-way MANOVA to examine relationships between OCB facets and status as respondent, passive or active nonrespondent status. • Significant multivariate effects were obtained (Wilk’s Lambda = .95, F (8, 1226) = 4.24, p < .01, 2 = .03 • Significant differences between respondents and active nonrespondents on altruism, civic virtue, courtesy, but not on conscientiousness • No differences between passives nonrespondents and respondents.

19. Overview of studies – understanding nonresponse • Survey response as social exchange (JOB, 2006) • Survey response as complaint or OCB (IJSA, 2008) Organizational and survey characteristics • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

20. An exchange framework for studying nonresponse • What factors influence if an employee will help others or the organization? Personality: Reciprocation wariness Social exchange • Spitzmueller et al, JOB 2006

21. Economic exchange Social exchange Economic versus social exchange (Blau, 1964) • Formal contracts • Contractual obligations to deliver in exchange for resource • Unspecified obligations • One person does another a favor • Expectations for return are vague, but exist

22. Reciprocation wariness • Reciprocity norm relevant for helping behavior in social contexts (Gouldner, 1960) • Individual differences in inclinations to “accept aid, return aid, or contribute a great deal to social relationships” (Cotterell, Eisenberger, & Speicher, 1992) • Reciprocation wariness is equivalent to “general fear of exploitation in interpersonal relationships” (Cotterell et al., 1992) • High-reciprocation wary individuals were less likely to engage in cooperative and social exchange with peers. Hypothesis 1: Organizational members with high reciprocation wariness are more likely to be active nonrespondents than passive nonrespondents or respondents.

23. Social exchange – survey response as reciprocation behavior • Employees favor balanced exchange of resources (Rousseau & McLean Parks,1993) • Balanced exchange relates to low intentions to leave, and willingness to support the organization (Shore & Barksdale, 1998) • Low employee contributions can be interpreted as attempts to restore balance (Coyle-Shapiro & Kessler, 2002). • Employee survey as an opportunity to either reciprocate or restore balance Hypothesis 2: Positive perceptions of the balance of social exchange, procedural justice, and perceived organizational support are more likely in passive nonrespondents and respondents than in active nonrespondents.

24. Measures

25. Results – social exchange, reciprocation wariness and survey response • MANOVA for three exchange variables (Pillai’s Trace = .08, F (6, 1196) = 7.90, p < .01, 2 = .04) • ANOVA for reciprocation wariness • Analyses of gender as a covariate

26. Limitations • Students are organizational members, but their exchange relationships differ from those of actual employees • Self-report measures of OCB are likely to differ from peer reports or supervisor ratings

27. Discussion • The degree to which individuals are inclined to feel exploited by others predicts whether they are willing to engage in helping their organization through survey completion. • Past experiences that influence organizational members’ perceptions of balance between input and output impact their intentions to complete surveys. • Procedurally fair processes may enhance response behaviors. • Perceptions of organizational support relate to response behavior. Why do passive nonrespondents fail to participate? Do job characteristics and strain relate to survey response behavior?

28. Overview of studies – understanding nonresponse • Survey response as social exchange (JOB, 2006) • Survey response as complaint or OCB (IJSA, 2008) Organizational and survey characteristics • Stressors and survey response behavior (JOHP, 2008) • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

29. Stressors, job characteristics and strain as predictors of survey response behavior • Passive nonrespondents fail to participate despite their reported intentions to comply. • Largest group of nonrespondents in our studies. • Little knowledge about their characteristics and experiences • What role do stressors, job characteristics and strain play in active and passive nonrespondents decisions to not complete surveys?

30. Stressors, strain and survey nonresponse Survey response behavior Role overload Role conflict Role ambiguity Strain Job autonomy • Barr, Spitzmueller & Stuebing, 2008, JOHP

31. Method • Population profiling study in a university hospital setting • Nurse interns completed wave I survey. • Surveys administered during training classes. • Follow-up survey about employment relationship with the university hospital. • 279 wave I respondents - response rate of only 85% • 78 participants (28%) to the follow-up survey, 28 active NR (10%), and 173 (62%) passive NR.

32. Wave I Measures

33. Stressors, strain and survey response behavior • Conducted simulation study to examine statistical power. • Use polytomous logistic regression to predict group membership. • Significant results for overload: Both active (odds ratio = 3.4) and passive nonrespondents (odds ratio 1.8) are more likely to experience overload. • Significant result for role ambiguity – but counter our hypotheses. • As role ambiguity increased, individuals were more likely to be respondents than passive nonrespondents, no differences for active nonrespondents (odds ratio =.54) • No other significant effects

34. Discussion • Role overload and ambiguity distinguish respondents and passive nonrespondents. • First study using population profiling in employment setting • Use of population profiling in employment settings difficult – response rate of 85% is not “near 100%”. • Concerns for job stress research? • Web survey follow up comments – can we assume response is the same?

35. Overview of studies – understanding nonresponse • Online survey response (PPsych, 2006, unpublished manuscript) • Survey response as social exchange (JOB, 2006) • Survey response as complaint or OCB (IJSA 2008) Organizational and survey characteristics • Stressors and survey response behavior (JOHP 2008) • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

36. Application of Unified Theory of Acceptance and Usage of Technology (Venkatesh, 2003) Web survey response behavior Internet self-efficacy Intentions to use internet Trust in internet and internet providers Social influence and norms Performance expectancy • Rogelberg et al., PPsych 2006, Spitzmueller et al., unpublished manuscript)

37. Method • Population profiling study using 804 undergraduate business majors • Study to examine technology acceptance on campus • Again, classification as respondents (n=102), active (n=82) and passive nonrespondents (n=620). • Average of 11 years of computer usage. • Measures based on Venkatesh et al. 2003 study items.

38. Results • Logistic regression analyses and mediated logistic regression. • Passive NR have lower internet usage intentions than respondents, no other differences. • Active NR with lower internet usage intentions, performance expectancy, social influence and trust, no differences in internet self efficacy. • Relationship between performance expectancy, social influence with active nonrespondent - respondent status is mediated by intentions to use the internet.

39. Discussion • UTAUT viable model. • Technology attitudes and intentions to use the internet are critical variables in active and passive nonresponse. • Results for non-student populations may be more pronounced due to lower levels of technology experience and comfort.

40. Organizational, social and individual difference predictors • Online survey response (PPsych, 2006, unpublished paper) • Survey response as social exchange (JOB, 2006) • Survey response as complaint or OCB (IJSA 2008) Organizational and survey characteristics • Stressors and survey response behavior (JOHP 2008) • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

41. So, what do response rates really mean? • Online survey response (PPsych, 2006, unpublished paper) • Survey response as social exchange (JOB, 2006) • What do response rates mean? • Survey response as complaint or OCB (IJSA 2008) Organizational and survey characteristics • Stressors and survey response behavior (JOHP 2008) • Methodology, attitudes and personality of active and passive nonrespondents (JAP, 2003) Individual differences and attitudes

42. So, what do response rates really mean? • Issues with response rate reporting rarely discussed • Concerns about inconsistent reporting of incomplete data, “non-delivered” requests for survey completion, rejected emails etc. • Proposition to use RR5 and RR6 identified in Public Opinion Research and recommend application of RR5 and RR6 in survey research across disciplines • Review of two years of some of the top journals in Org. Behavior show drastic inconsistencies in response rate reporting

43. Implications for Management Research • Expectations regarding equivalent response rates across different populations may need to be reevaluated • Drivers of response behavior among different organizational stakeholders need to be examined • Specific techniques for increasing response rates and drivers of response behavior need to be examined • Bias based on nonresponse should be examined, potential focus on mean biases and biases in relationship estimates

44. Thank you…. • Graduate students: Cristina Rubino, Chris Barr, Kathryn Keeton, Kayo Sady, Alex Milam • Undergraduate lab assistants: Rena, Lucia, Amanda, Lindsay, Hung… • Academic mentors: Milt Hakel, Jeff Stanton, Julia Hannay, Steven Rogelberg, Dieter Zapf • Research participants – respondents and nonrespondents.