Field test options instrument, instructions, non-response/ refusals, and Interviewer debriefing

1. Field test options instrument, instructions, non-response/ refusals, and Interviewer debriefing Washington Group Regional Training Workshop Rio de Janeiro 19 – 20 September 2005 Margie Schneider HSRC, South Africa

2. Recap What we have covered so far: • Purpose of the questions and where they came from • Core set of questions and extended set and Q X Q specifications • Objectives of the test • Translation protocol What we still need to go through: • How to design the test and related issues • Enumerator training • Plan for analysis and report writing

3. Different components of the testing process • Pre-testing and expert review • Testing internal validity on people with known disability status (quota sampling and linked to cognitive testing) • Pilot testing – as for full Census or survey • Full field test Different sampling approaches will be discussed in relation to different testing protocols

4. Field Test options • 2 questions to be answered: • What set of questions should be used in field test? • What field test approach should be used? Consider your own context, need, funding, skills, etc. in deciding

5. Question sets • Core set – 4 + 2 (6 domains of functioning): Compulsory • Core + extended set on core domains : highly recommended • Core + extended set + further additional questions: nice to have for analysis • 2 additional domains (learning and interpersonal interactions) • Psychological distress • Country specific question set : nice to have for comparison with prior data

6. What approach to testing? • What factors to consider in deciding: • Country needs • Capacity to undertake test • Funds available • Time available • Planned activities, e.g. other surveys • ?

7. Census or national survey • Use of question set in planned Census or national survey • Advantages: • Large sample (5000 – 10 000 or whole pop) • Minimal additional cost for disability questions • Disadvantages: • Not sufficient space for extended set • Need to administer extended set and cognitive test to sub-sample (costs of time and additional training)

8. Special study – small sample • Select sample on basis of known ‘disability’ status • Allows for construction of 2X2 table for sensitivity and specificity calculations and identifying true and false positives and negatives – use of ‘gold standard’ • Suggest around 200 true positives and 200 true negatives • Interviewer does not know status of respondent (unless visible and obvious)

9. Special study (contd.) • Advantages: • Larger set of questions • Close observation of interview • Cost is not exorbitant • Disadvantages: • Might not get full population representation • Cost of establishing true status is high • What counts as the true positive? (beyond the easily observable) • What is the ‘gold standard’?

10. Special study – large sample • Uses population based sample • Sample size based on expected prevalence of the different types of activity limitations • Prevalence rates are usually low and so need large sample to yield enough ‘disabled’ respondents

11. Special study – large sample (contd) • Advantages: • Large data set • Detailed set of questions - more than extended set • Provides an indication of prevalenceusing core and extended sets • Disadvantages: • Costly and time consuming • Requires capacity to run and analyse

12. Field test and cognitive test • All respondents have core set • One sub sample of respondents have cognitive test (core + extended are embedded) • A second sub-sample of respondents do • Core + extended • Core + extended + further Qs • Core + extended + country specific questions • Some do all ?

13. Non-responses and refusals • How do you define these? • How do you deal with these? • Aim to: • Reduce non-response and refusals • Manage them when they do arise

14. Reducing NR and refusals • Use up to date sampling frame • Clearly defined reasons for data collection • Preparing the way – prior contact, letters, etc. • Adequate interviewer training • Allow budget for call-backs • Plan for follow up of NR and refusals • Separate refusals, part refusals, non-contacts and sample loss (e.g. vacant dwellings)

15. Effect of each on analysis • Sample loss: does not create bias but reduces sample size; need large enough initial sample to take these into account • Refusals and non-contact: bias where these respondents might be different to those respondents reached • Need to understand reasons for refusals and non-contacts – during pre-testing and pilot stage

16. Interviewer debriefing • Interviewers are well placed to evaluate process and content at an early stage • Interviewers should note comments in margins of questionnaires • Different approaches: • Group discussion (focus group technique) • Interviewer rating forms • Standardised interviewer questionnaires • Combination of all three

17. Enumerator training Washington Group Regional Training Workshop Rio de Janeiro 19 – 20 September 2005 Margie Schneider HSRC, South Africa

18. General points • Select some disabled interviewers • Importance of all interviewers having a good understanding of what disability is and is not and how it relates to the questions asked • Avoid using term ‘disabled’ or ‘with disabilities’ • All interviews are face to face • Translation training

19. General points (contd.) • Confidentiality and understanding what this means • Getting informed consent (ethics clearance?) • Interviewers must be able to explain purpose of survey • Read questions in set order and with set wording • Editing in field of completed questionnaires • Submitting of completed interviews to head office

20. Types of interviews • Direct: respondent answers for him or herself • Interpreted: an interpreter ‘translates’ and respondent answers directly • Facilitated: a third party assists in explaining (e.g. intellectually disabled person) • Proxy: a person responds for another (e.g. child)

21. Interviewing disabled people • Show respect and treat the person like anyone else • Don’t use first names unless permitted • Address the person directly (not their attendant) • Ask how you can adapt your presentation to make it easier (no need to ask what is wrong with person)

22. Hearing difficulties • Lip reading • Lighting • Face the person • Get person’s attention before speaking • Reduce background noise • Set the context – especially when changing topics • Use written communication (literate)

23. Physical difficulties • Accessibility of building where conducting interviews • Presence of attendant and confidentiality issues • Get to same level (e.g. sitting for person using wheelchair) • Person to be seated comfortably • Address person directly • Pointing may be difficult • Person may need breaks to move around

24. Visual difficulties • Large print and small print for cue cards • Braille versions of cue cards • Good contrast printing for pictures and print (black on white or yellow) • Identify yourself and others in the room verbally

25. Communication difficulties • Clarify preferred mode of communication • Repeat what you think was said to clarify unclear speech • Limit to yes / no questions

26. Specific learning difficulties • Manage problems in spatial orientation, hand-eye coordination • Limit auditory, visual and tactile distractions • Avoid written text • Explain carefully (if verbal language skills are affected)

27. Intellectual difficulties • Be careful with informed consent • Explain terms simply • Listen carefully • Have familiar person (friend or relative) close by • Use pictures or role play with little human or animal figures

28. Emotional or mental health difficulties • Side effects of medication • Break up interview if too fatigued • Give encouragement and support • Manage expressions of frustration • Manage stress

29. Hidden difficulties • Might not come forward with information because of fear of stigma • Effect of medication • May need to break up interview