Surveys and Questionnaires

1. Surveys and Questionnaires M. Burns FCS 5470

2. A few fine points… • Survey (method) v. questionnaire (tool) • Assumptions • Can read • Willing and able • Will complete to the best of his/her ability • Usability Russ-Eft & Preskill, 2001

3. Advantages Inexpensive & easy Large, diverse groups Familiarity Anonymous Quantitative Disadvantages Low response rate Literate groups Validity/reliability issues Quantitative Usability Russ-Eft & Preskill, 2001

4. Types of Surveys • Post-course reaction forms • Behavioral or skill measures • Employee satisfaction • Knowledge tests Russ-Eft & Preskill, 2001

5. Post-course Reaction Surveys • Quality of instructional methods, facilities, materials, and instructors/facilitators • Many times poorly developed, thus limiting usability • Addresses what level in Kirkpatrick’s model? Russ-Eft & Preskill, 2001

6. Post-course reaction surveys do not measure… Russ-Eft & Preskill, 2001

7. Behavioral or Skill Measures • Evaluates performance and change initiatives • Self-reported learning • May be used pre and/or post course or long-term post • Addresses what level in Kirkpatrick’s model? Russ-Eft & Preskill, 2001

8. Employee Satisfaction Survey • Focuses on the impact that the intervention had on employee satisfaction • Commercially or self-developed • Administered pre and/or post course • Addresses what level in Kirkpatrick’s model? Russ-Eft & Preskill, 2001

9. Knowledge Test • Evaluates learning, such as certification examination • Typically completed post-course, but could use the pre-course design • Example is RD exam Russ-Eft & Preskill, 2001

10. Guidelines for Constructing Surveys • Anonymity and confidentiality • Determining question type • open-ended, fill-in-the-blank, dichotomous or two-choice question, multiple-choice, rating scales, and ranking • Question construction Russ-Eft & Preskill, 2001

11. Factors to consider when determining question type • Who will be answering the question? • How much time will respondents be able and willing to spend? • How many respondents will be involved? • How much is known about the range of possible answers? Russ-Eft & Preskill, 2001

12. Question Types • Open-ended • Fill-in-the-blank • Dichotomous • Multiple-choice • Rating • Likert, behaviorally anchored, behavior observation scale • Ranking Russ-Eft & Preskill, 2001

13. Question Construction • Use understandable terms • No acronyms or double negatives • Avoid leading or loaded questions • Leads to biased responses • Avoid ‘double barreled’ questions • May only respond to one part Russ-Eft & Preskill, 2001

14. Common Errors of Question Items • Double barreled questions 78% • Leading/loaded questions 36% • Already know answer 21% • Use non-neutral wording 17% • 1+ response categories 17% • Not clear, concise, and simple 17% Lee, 1998

15. Format Considerations • One-page – if not use attractive cover page • Colored paper – numbered pages and items • Letter of instructions/support/usability • Visual appearance • Grouping of questions • Clear directions at beginning and at different sections • NO Tipo’s Russ-Eft & Preskill, 2001

16. Pilot Testing • Purpose is to identify any weaknesses that you may have overlooked • Time to take the survey • Do the questions asked really address your evaluation goals? • Measure of validity and reliability Russ-Eft & Preskill, 2001

17. Logistics • Paper v email v online • List of ‘possibles’ or sampling frame • Timeline for data collection, analysis, and reporting • Cover letter • Purpose, uses, sponsors/approvals, signatures, directions, and due date Russ-Eft & Preskill, 2001

18. Increasing Response Rate • Importance of survey • Keep it short! • Personal touch • Appearance • Incentives • Deadlines • Follow-up procedures Russ-Eft & Preskill, 2001