Conducting a Survey

1. Presentation To Faculty Athabasca University By W. Todd Rogers Centre for Research in Applied Measurement and Evaluation University of Alberta April 27, 2010 Conducting a Survey

2. Measurement is the process of assigning numbers according to some specified rule such that the numbers reflect differences in the amount of the variable possessed by different individuals. The numbers can give the extent of the difference on (i) an ordinal scale (ii) an interval scale; or (iv) a ratio scale.

3. EVALUATING INFORMATION OR A SCORE YIELDED BY AN ASSESSMENT METHOD

4. VALIDITY ARE WE TELLING THE TRUTH?

5. MEASUREMENT SAME OR SIMILAR INFORMATION OR SCORES CONSISTENTLY?

6. Survey Questionnaires Main source of information in today’s society: √market research √ students’ and parents’ attitudes toward schools. √ voters’ preferences for different products and/or the need for or effectiveness of economic, environmental, health, educational, research, and social programs. √ program evaluations

7. A Problem People think surveys are easy to conduct! But are surveys easy to conduct given the need for √ validity and √ low error or measurement?

8. What, then, do we need to attend to? Question or questions we want to answer. What is the attitude of adult people in the province of Alberta to private health care? Should we look at differences by Age? Gender? Income level? ? Now, what is meant by private health care? Clear to respondents Avoid jargon Must direct subsequent activity

9. Sampling • Population of people • Universe of items

10. Sampling People First, need to know the population. Second you need a sample design Take into account How survey instrument will be administered to ensure √ ahigh response rate and √ a representative sample?

11. Typical response rates: • mail and Internet surveys are around 40% • Telephone surveys, with three call backs for initial non contact, have response rates of about 70%, • in person surveys runs as high as 95%.

12. Sampling Frame and Design Mail survey, restricted Internet surveys in which who can access the survey questionnaire is controlled, telephone surveys, and in persons surveys require the use of a sampling frame. The sampling frame contains a list of the sampling units. Sampling units consist of persons, schools, hospitals, for example. Armed knowledge of the sampling frame, the representativeness of the sample of actual responders can be assessed. What stratification is needed gender? age? income level? ? Will clusters be sampled? Schools, hospitals How will data by analyzed? Analysis of the survey items Analysis of the data to answer the survey question(s)

13. Sampling Items Constructing a Sample of Items Another important component of a survey is the set of questions contained in the questionnaire. Often the questionnaire to be used is constructed too quickly and without a lot of prior consideration of what should be asked. Yet it is known that questionnaire construction is one of the most delicate components of a survey. Asking the right questions – questions that provide reliable information that can be validly interpreted to as to be able to make sound judgments and formulate fair and defensible decisions – is more an art than a science. If the right questions are not asked or asked in an ambiguous or wrong way, the survey will not yield meaningful and useful information, no matter how well other components (e.g., sampling, analysis) of the survey are conducted. Improper questions will most likely result in unreliable information that can not be validly interpreted and that, if used, will most likely lead to an unsound judgments about the worth and value of the object of the survey and indefensible decisions about what steps need to be taken next.

14. What is your reaction to this questionnaire? Six Steps

15. 1. Identify what information is needed and how it will be used. What is the attitude of adult people in the province of Alberta to private health care? What is meant by private health care? Publically funded, publically delivered Publically funded, privately delivered Privately funded in part, privately delivered Privately funded in whole, privately delivered ? 2. Determine the type(s) of survey questions to be asked. Selection Items Constructed Response Items What month were you born in? ______ Versus

16. Versus 0 000 1 1 11 2222 . . . . . . . . 9999 Are you aware of the new provincial legislation known as Bill 51? Yes If yes, what to you think of Bill 44? __________________________________________________________________________ ___________________________________________________________________________ ______________ No (Go to the next question) I think that funds for Fine Arts subjects could be better spent on the basic subjects of English, mathematics, and science. Strongly Agree 1 2 3 4 5 Strongly Disagree Versus I think that funds for Fine Arts subjects could be better spent on the basic subjects of i. English Strongly Strongly Agree Disagree 1 2 3 4 5 ii. mathematics 1 2 3 4 5 iii. science 1 2 3 4 5

17. Table 1 Anchor Descriptions for Likert Scales

18. Each of the statements below refers to classroom or course situations which may cause anxiety for some students. Anxiety is often described as feelings of dread and apprehension without specific cause for the fear (Chaplin, 1985). Feelings of anxiety vary in degree or intensity, depending on the individual and the situation. Anxiety is associated with negative or unpleasant emotions. Read each statement carefully, and then place a vertical line across the line that describes how much anxiety you currently feel regarding each situation. Your first response is usually the best one and there is no right or wrong answer. 1. Watching a teacher develop a statistical equation on the blackboard. I I not at all anxious very anxious 2. Buying a statistics textbook. I I not at all anxious very anxious Being given an assignment with many difficult problems that is due next class. I I not at all anxious very anxious Reading and interpreting graphs and charts in a statistics textbook. I I not at all anxious very anxious 5. Signing up for a course in statistics. I I not at all anxious very anxious 6. Thinking about a statistics test the day before the test. I I not at all anxious very anxious 7. Listening to another student explain a statistical formula in class. I I not at all anxious very anxious

19. Take the sentiment expressed by Tom Lehrer in his song about the Russian mathematician Lobachevsky: Plagiarize, plagiarize Let no one else’s work evade your eyes Remember why the good Lord made your eyes So don’t shade your eyes But plagiarize, plagiarize, plagiarize, Only be sure to always call it, please Research.