Designing engaging questionnaires to deliver high response rates

1. Designing engaging questionnaires to deliver high response rates Presented by Steve Bax Managing Director Bax Interaction Limited

2. This webinar will explore the key principles behind effective questionnaire design. It will include current thinking on how best to engage potential respondents and thereby improve completion rates. Overview

3. Content • Introduction • Defining the research process • Questionnaire design and piloting • Engaging the respondents • Research design checklist • Questions

4. Research phases Engagement including pre-qualification Piloting and Design Phase Defining Phase Reminders Resources required Analysis, reporting and review phase Finish Start Time

5. Primarily to acquire quantitative information from respondents Also provide qualitative insights Reasons include:- Identifying, evaluating and sizing existing and new markets Testing reactions to new products and services Measuring performance Satisfaction surveys And more Why use questionnaires?

6. Defining

7. Purpose Research objectives The ‘shopping list’ Data to be collected Research vs.. volunteered Target audience(s) Delivery mechanisms Previous experience Defining

8. The objectives are key and must generate the topics for investigation. The wording can imply a certain method E.g. to measure = quantitative E.g. to explore = qualitative Consider other terms: goal, aim, purpose and hypothesis Research objectives

9. What is my ‘shopping list’? What data do we need at the end of the process? Is it pure marketing research or volunteered data? Who do we need to gather data from? What are the implications on method and design? What have we learnt from previous or others’ research successes or failures? Stop and think

10. Selection of delivery mechanisms

11. Advantages Low cost Fast International coverage Creative opportunities Effective for qualitative and quantitative data collection Online surveys • Disadvantages • Declining response rates • Incomplete cover • Biased responses possible

12. Advantages Geographical reach Control over the process Quantity of data that can be gathered Respondent perceptions Disadvantages Cost due to postage rates and printing vs. online Slow to administer Response rates Self-selecting samples - bias Written and postal surveys

13. Advantages Low cost vs. face to face Fast International coverage Control over process Understanding checking possible Disadvantages Refusal rates Sugging and frugging perceptions Limited duration Telephone based questionnaires

14. Snap Surveys and Silver Dialogue findings with National Trust visitors Multi mode approaches

15. MMR ‘DiaLogs’ APP

16. Design and piloting

17. Fundamentals Relevance Usability Layout and appearance Question and response construction Methodology Qualitative vs.. quantitative building blocks Multi-mode approaches Sampling Open and closed question mix Piloting Design

18. .. the rigour, integrity and quality of data. Neither interviewers nor respondents should be confused or bored by any questionnaire. The guiding rule can be expressed as ‘doing justice to how people would normally talk about things’. Source: MRS – Questionnaire design guidelines Key design principles

19. As a rule – progress from the general to the specific Gain understanding of broad context of opinions held THEN the nature and strength of opinion in a given area Step by step to the detail underpinning these ALWAYS PILOT TEST ‘Funnel’ sequence

20. Target audience Develop question topics Select question and response formats Select wording Determine sequence Design layout and appearance Clear instructions MRS/DPA considerations Questionnaire construction

21. Introduction Spacing Numbered headings and sections Numbered questions Length Quality of production – on and offline Use of sequencing, routing, piping and a variety of response mechanisms Clear instructions Use of colours to indicate changes in areas being questioned Font and text size Layout and appearance

22. Behavioural - market size, market share, usage rate, awareness Attitudinal - image and attitude surveys, brand mapping, help build market share Classification - all surveys Types of question

23. Have you ever ……? Who do you know …………? How many ………….? When did you last ………? Do you have ………? Who does …………? Behavioural questions

24. What do you think of ……? Why do you …………? Do you agree or disagree ………? How do you rate ………? Which is best (or worst) for ………? Attitudinal questions

25. Closed questions Open-ended questions Scales Variety is key Question / response formats

26. Dichotomous Two possible answers Multiple choice Three or more answers! Collectively exhaustive Mutually exclusive Pros and cons Closed questions

27. Advantages Full answer Exploratory Disadvantages Misinterpretation Speed Coding / analysis Open-ended questions • Completely unstructured - interviewer or respondent writes answer • Three key open-ended questions are useful: • 1. What is good? • 2. What is bad? • 3. What improvements could be made?

28. Examples Age Gender Household status Marital status Social class Education level Industrial occupation Number of employees Location Neighbourhood Classification questions

29. Forced vs.. non forced Forced Doesn’t allow ‘middle ground’ Non forced Allows neutral views e.g. ‘neither expensive or inexpensive’ Balanced vs.. unbalanced Most balanced e.g. equal number + as – Unbalanced used where ‘end piling’ is expected e.g. Car safety features Use of scales

30. Ambiguous questions Double barrelled Leading / loaded questions Checking questions Choice of statements to measure Implicit assumptions Language Tone Question wording

31. Design variations Desk research A B C Qual stage D E Quant stage F Final report

32. Screening Sampling Research currency Timeliness Clear explanation of reason target chosen and mutual benefit Ensuring relevance to the audience

33. Progress indicators Forward and back buttons Self checking Filtering Routing Piping Radio buttons Sliders Online design options

34. Video / visuals / documents as stimuli and intuitive response mechanisms Audio comments ‘Click and drag’ Select and deselect in a visual context Voting on visual contexts Hotspot mapping Timed responses Online design options

35. Defining parameters Improving design aspects Adding key questioning areas missed Assessment of mix of closed vs.. open questioning Ensuring understanding and logic Checking completion timing Checking perceived response likelihoods The power of pilot testing

36. Engagement

37. Pre-qualification Landing timing Reminders’ timing and impact Incentivisation Current environment Engagement

38. Why should the target respond? Transparency and consent Justification Relationship building Mutual benefit Sampling should be used to reduce data fatigue Relevance critical Incentive? Pre-contact Multi-mode approaches Attention grabbing headlines and titles needed Pre-qualification – “What’s in it for me?”

39. Response rate trends

40. Initial survey invitation – online Monday pm optimal* Reminders Avoid over-use of reminders 2 reminders appear to generate optimal additional returns currently** Typically 50% of initial response following reminder 1 and… 50% of response to reminder 1 following reminder 2 Sources: *Lightspeed Research survey timing tests / **Bax Interaction further education surveys results 2012/2013 Timing

41. Consider target audience(s) Perceived value MRS rules Include in pilot testing Previous experience Comparable surveys Incentivisation

42. Time poor audiences Data fatigue – many are over researched Instant gratification culture Low attention spans Desire for intuitive design Desire for simplicity Base of ‘tech savvy’ respondents growing Digital environment changing attitudes to research Current environment

43. Ofsted example Getting it wrong?

44. Design checklist- getting it right Engage your targets to gain optimal response rates

45. 1. Methodology Relevance to the target audience? Delivery mechanisms – right for the target audience? Expected response rates to be measured? Has previous experience been applied? Checklist - 1

46. 2. Questions Number of and logical order? Relevance and appropriateness to the audience and the required outcomes? Tone / language? Ease of understanding? Clarity of instructions? Classification questions at the end? Checklist - 2

47. 3. Responses Variety and use of scales and other mechanisms for scoring responses? 4. Testing Has a pilot test to identify possible improvements including length, layout, question order and whether content is meeting the objectives been done? Checklist - 3

48. 5. Introduction Timing for completion? Explanation of purpose? Gaining consent? Explanation of how respondents’ data will be used? Incentives being offered? Meeting regulatory requirements? Checklist - 4

49. 6. Layout Attractiveness to the respondents? Colour, fonts, branding, visual appeal? Clear routing instructions? Effective use of filtering and piping? Use of numbered questions and sections? Checklist - 5

50. 7. Structure Is there the right mix of open and closed questions? 8. Post survey planning How will the results be analysed, findings presented and followed up? Checklist - 6