Quantitative Research Methods: Questionnaires

1. Quantitative Research Methods: Questionnaires Presented by Stacie Petter and Adriane Davis October 9, 2002

2. Outline • Definition • Criticisms of Survey Methods • Advantages & Disadvantages • Common Problems Found in MIS • Steps for Designing & Conducting a Survey • Methods of Distribution • Examples • Summary

3. Definition • Type of survey method (questionnaire vs. interview) • In usability glossary, [Brehob, 2001] defines a questionnaire to be "a form that people fill out, used to obtain demographic information and views and interests of those questioned". • [Kirakowski, 1998] defines a questionnaire in a more structural way as "a method for the elicitation, and recording and collecting information".

4. Criticisms of Survey Method • No theoretical value can be obtained from surveys, even though they amass a great amount of data • Researchers must be clear about the data that is to be gathered before the survey begins. One cannot use a questionnaire or survey to determine possible variables. • Research performed via surveys can be too confined because of structured nature of questionnaires • There are certain limitations, but there are tradeoffs that must occur in any research technique. • Surveys yield too high of measurement errors to be used reliably • It is important to have a good survey design to ensure data quality. • One cannot infer a causal relationship through surveys and questionnaires. • One cannot prove a causal relationship via a cross-sectional study and statistics; however, this method can support relationships that are suggested within a theory.

5. Advantages • Broad scope • Answers the questions of what, how many, how much, how, and why • Ability to observe a phenomena in a natural setting • Ability to control variables • Respondents may be more honest in their responses because of the anonymous nature • Low marginal costs (reusable) • Easy to administer, score, and code • High external validity • Accurate results due to the large sample size • Method to test theoretical propositions objectively • Confirms and quantifies the findings of qualitative research

6. Disadvantages • Provides little depth and understanding of the context and history of a phenomena • Can only provide a snapshot of behavior at one place and time • Structure of questionnaires may cause people to poorly represent their opinion • Or subjects may elicit no response on some questions • Results may differ if using the same questionnaire in a different context or culture • Cannot prove causal relationships • High administrative costs to set-up and score • Typically low response rates

7. Problems Commonly Found in MIS • Single-method designs where multiple methods are needed • Multiple methods aid in providing the context for the phenomenon; this usually requires fieldwork. • Unsystematic and inadequate sampling procedures • Many researchers either used a convenience sample or did not describe or their sampling procedure. It is critical to have a proper sample for generalizations of the results. • Low response rates • By having low response rates, it can become more difficult to generalize to a specific population. If performing a questionnaire within an organization, one should use the hierarchy and gain top management support to increase the response rate. Good questionnaire design is also necessary to improve the rate of response. • Over-reliance on cross-sectional surveys when longitudinal surveys are needed • Many IT issues have a dimension of time, which cannot be represented appropriately in a cross-sectional survey.

8. Steps for Designing and Conduct a Survey • Set the goals - What do you want to capture? • Decide on the target population and sample size - Who will you ask? • Determine the questions- What will you ask?KISS method • Pre-test the survey - Test the questions • Conduct the survey - Ask the questions • Analyze the data collected - Produce the report

9. Methods of Distribution • Most popular methods include: mail, fax, and web-based/email questionnaires • Contrast speed, response rate, and cost • Consider context for distribution (i.e., university setting vs. business setting) • Consider method for capturing and analyzing the data

10. Methods of Distribution (cont.)

11. Mail • Speed: Slow • Response Rate: Medium-low; generally less than 5% • Cost: Most expensive option • Data Collection: Scan or code by hand • Key Points: • Sensitive to literacy level • Has the highest coverage area

12. Fax • Speed: Fast • Response Rate: Low • Cost: Relatively inexpensive • Data Collection: Scan or code by hand • Key Points: Sensitive to machine-accessibility

13. Web-based/Email • Speed: Fast • Response Rate: High • Cost: Least expensive option • Data Collection: Already in electronic format for easier manipulation; data may directly link to a database • Key Points: Internet usage and poor web-page design

14. Sample Results Context: 300 hospitality professors polled in April 2000 Speed (quickest to slowest response time): • Fax – 4 days • Web-based/email – 5.97 days (Note: 60% came back within the first 1-5 days) • Mail – 16.97 days Response Rate (highest to lowest rate): • Web-based/email – 44.21% • Mail – 26.27% • Fax – 17.0% Cost (least to most costly): • Web-based/email • Fax • Mail

15. Examples of Questionnaires Used in IS Research • Questionnaires are designed to assess aspects of usability, the validity and/or reliability of human-computer interfaces. Most well-known surveys include: • Questionnaire for User Interface Satisfaction (QUIS), Univ. of Maryland , 1988 • Perceived Usefulness and Ease of Use (PUEU), IBM, 1989 • Nielsen's Attributes of Usability (NAU), Bellcore, 1993 • Nielsen's Heuristic Evaluation (NHE), Bellcore, 1993 • Computer System Usability Questionnaire (CSUQ), IBM, 1995 • After Scenario Questionnaire (ASQ), IBM, 1995 • Practical Heuristics for Usability Evaluation (PHUE), OSU, 1997 • Purdue Usability Testing Questionnaire (PUTQ),  Purdue Univ. , 1997 • Graphic, Visualization, & Usability Center (GVU) at Georgia Institute of Technology produces user surveys to evaluate trends in Internet usage and commercial usage of the web.

16. Summary • Consider mixed-modes to increase response rate BUT cover the breadth of the population (i.e., allow those without ready accessibility to the Internet to participate) • Consider cash and non-cash incentives to increase response rates • Consider most effective method for large samples (time and cost)

17. References • Brehob, K., et al. “Usability Glossary.” http://www.usabilityfirst.com • Provides methods for creating easy-to-use website and software design. • “Charm Survey Methods.” http://www.otal.umd.edu/hci-rm/survey.html • This website is a good starting point and overview of conducting a survey. • Cobanoglu, Cihan, Bill Warde, & Patrick J Moreo. “A comparison of mail, fax and web-based survey methods,” International Journal of Market Research; Henley-on-Thames; Fourth Quarter 2001, Volume 43, Issue 4, p.441-452; • This article compares and contrasts three methods for issuing surveys: mail, fax, and web-based. It is significant because it contrasts older methods of surveying with those brought on by the Internet and new technology. • Jenkins, A.M. (1985). “Research Methodologies and MIS Research,” in Research Methods In Information Systems, E. Mumford et al. (eds.), North Holland, pp.103-117. • Compares various research techniques and provides assistance in choosing the best technique for the research problem. • Kirakowski, J., “Questionnaires in Usability engineering.” http://www.ucc.ie/hfrg/resources/qfaq1.html • Presents FAQ regarding questionnaires; in particular, this site looks at questionnaires that measure usability. • Newsted, P. R., W. Chin, O. Ngwenyama, and A. Lee, (1997). "Resolved: Surveys have Outlived their Usefulness in IS Research," Panel presented at the 1996 International Conference on Information Systems, December 17, 1997 ,Cleveland, Ohio. • Presentation that gives various perspectives regarding the usefulness of surveys in MIS research.

18. References • Newsted, Peter, Sid Huff, & Malcolm Munro. “Survey Instruments in MIS.”http://www.ucalgary.ca/~newsted/list.htm • Website that provides information regarding using surveys as a research methodology in IS. This site contains a database of published works within the MIS field that use surveys as a methodology as well as provides information regarding survey instruments in general. • Pinsonneault, Alain & Kraemer, Kenneth L. (1993). “Survey Research Methods Methodology in Management Information Systems: An Assessment.” Journal of Management Information Systems 10(2):75-105. • The authors perform a study within the field of MIS to determine problems that are commonly found in the survey design and methodology. Five critical problems are found within the MIS field along with suggestions for improvement of survey research. • Roberts, E. S. (1999). "In defence of the survey method: An illustration from a study of user information satisfaction." Accounting & Finance 39(1): 53-77. • This paper provides arguments regarding why survey methods are a valid research method. • Stone, Eugene F. (1978). Research Methods in Organizational Behavior. Goodyear Publishing Company, Inc. Santa Monica, CA. • Provides an overview of various research methods that can be used by researchers. • “Survey Design, Questionnaire Design Tips,” Creative Research Systems, http://www.surveysystem.com/sdesign.htm • This website is a relevant source referenced in the UMCP website. It covers scanning methods for collecting data from surveys. • Trochim, William M.K. “Survey Research.” http://trochim.human.cornell.edu/kb/survey.htm • Source cited as basis of UMCP’s website. Deals mostly with general survey methods, but provides methodology for selecting a survey type.