REACH MENA Regional Workshop 5-7 March 2014 Jordan

1. REACH MENA Regional Workshop5-7 March 2014Jordan Research Ethos Terms of Reference Secondary data analysis Objectives & indicators Quantitative & Qualitative data collection – methods Quantitative & Qualitative data collection – tool design Sampling Field work preparations 9.a. Quantitative data analysis 9.b. Qualitative data analysis 10. Reporting and representing data 11. REACH products Assessment Workshop

2. 1. Research Ethos Who does research quality matter to? • Research community: developing sound research literature; ‘knowing the causes of things’ • Research funders: value for money, continued investment, • Research users: confidence in the results, belief that they are relevant • Research respondents: ethical considerations, cooperation It matters to these groups because quality ensures we are able to inform more effective humanitarian action

3. 1. Research Ethos How to do quality research? • Develop a strong research question – clear objective(s) • Develop/adopt a strong conceptual and theoretical framework – to identify context, needs, response, gaps and priorities • A fit between the method and the research question/orientating framework • High-quality data and analysis • Firm basis for our conclusions • Being expansive: highlighting the significance of our work - making recommendations

4. Presenting Initial Analysis • Final Analysis & reporting • Development of REACH Products • Peer Review • Dissemination • Primary Data Collection & Entry • Preliminary Data Analysis The REACH Assessment Life Cycle

5. 2. Terms of Reference

6. 2. Terms of Reference

7. 3. Secondary data analysis Why do Secondary data analysis? • To provide context • To identify gaps in information needed to measure indicators • To inform sampling • To inform primary data collection tools

8. 3. Secondary data analysis

9. 4. Objectives & indicators

10. 4. Objectives & indicators

11. 4. Objectives & indicators DISCUSSION Why do we need consistent indicators? Why do we need regional and country specific indicators?

12. 5. Quantitative & Qualitative data collection – methods

13. 5. Quantitative & Qualitative data collection – methods QUANTITATIVE researchers typically seek – • Causal determination, prediction, robust dependence/associations, generalization of findings • Structured and systemizing method • Controlled conditions • Usually large samples • Tests theories and hypotheses

14. 5. Quantitative & Qualitative data collection – methods QUALITATIVE researchers typically seek – • Depth rather than breadth: integrity of perspectives through rich own-word accounts; description, insight and understanding • Discovery (iterative) rather than verification (hypothesis testing) • A mainly inductive rather than deductive analytical process: can develop theory from information collected • Less structured / exploratory method (NOT UNSYSTEMATIC) • Uncontrolled conditions, usually small samples, no statistical analysis

15. 5. Quantitative & Qualitative data collection – methods Decline in pirates causing climate change? • QUANTITATIVE: Relational/causal – typically addressed through representative sample surveys and experiments where “robust dependence” is established if a relationship refuses to go away once other factors AND explanations are taken into account • ALWAYS REMEMBER: Correlation is NOT causation > > > >

16. 5. Quantitative & Qualitative data collection – methods • QUALITATIVE: Aim is to draw conclusions about mechanisms within group being studied – NOT to infer to general population • MIXING QUALITATIVE & QUANTITATIVE METHODS: • Qualitative method used to INFORM or EXPLAIN data gathered through quantitative method • Quantitative method used to MEASURE PREVALENCE of factors identified in data gathered through qualitative method

17. 5. Quantitative data collection – methods • Individual or household interviews – structured (survey) • WEAKNESS: Time consuming, expensive, requires specialized knowledge of survey design to provide valid information. • STRENGTH: When properly done provides hard evidence of basic statistics (e.g. malnutrition rates; demography; disease rates, etc.) which are representative of the entire population. • Only use a survey • If you are confident of your design and sampling methods • If the objective is to produce findings that are representative of the broader population • Key Informant interviews – structured (survey) • We use a KI survey to gather quantitative data – where we essentially ask KIs to estimate household level information

18. 5. Qualitative data collection – methods • Individual or Household interviews – semi-structured • A cross-section of people interviewed on the same topic to reveal a range of attitudes, opinions and behaviours. • Interviewees must be selected to give a good-cross section and avoid sample bias. • Enables more private reflections and broader perspective of each individual than in-group interviews – more likely to reveal conflicts. • Key Informant interviews – semi-structured • Key informants can be • specialists in topics you are interested in, • outsiders within the community (like teachers) who may give a more objective view, or • others who are in some way especially knowledgeable. • Beware, key informants can reinforce inherent power structures and existing inequalities within communities. Bear in mind the local power structure before interacting with key informants. • Focus Group Discussions/Interviews – semi-structured • Small groups (6-12) of people with something in common: special knowledge or interest in certain topics. • We often want homogeneous groups – e.g. Age/gender to allow free discussion – beware power dynamics • Facilitator keeps the discussion balanced and on track. • Enables insight into group interactions/social norms • Can help identify themes to measure prevalence of in individual interviews • Can help explain trends in prevalence already measured in individual interviews

19. 5. Qualitative & Qualitative data collection – methods DISCUSSION Which methods have you worked with? What challenges did you find using quantitative methods (structured interviews) What challenges did you find using qualitative methods (semi-structured interviews)

20. 6. Quantitative & Qualitative data collection – tools

21. 6. Quantitative data collection – tools

22. 6. Quantitative data collection – tools STRUCTURED QUESTIONNAIRE– EXAMPLE QUESTION • BAD: How much do you spend on essential needs? • Measurement (IQDs? USD?) • Essential needs (what are they?) • Spent by who (all household members? The respondent?) • Time period (during the past day, the past week?) • BETTER:Howmuch did your household spend on education (school materials, fees) during the most recent 7 days?

23. 6. Qualitative data collection – tools SEMI-STRUCTURED INTERVIEW/DISCUSSION TOPIC GUIDE • Identify the major objectives – what exactly do I want to know? • Do not ask the research question directly but through indirect questions and conversations around the issue (translate into everyday language) • The topic guide should help to ensure a comfortable conversation • Funnel approach: from general to specific • The topic guide should be short (rule of thumb: 5 to 8 questions) but well prepared (piloting) • Moderators should take detailed notes, using the same language as participants to not loose context • 1 hour maximum to complete.

24. 6. Qualitative data collection – tools RANKING & SCORING • Placing something in order, reveals differences within a population. • Helps to identify main problems or preferences of people, and the criteria they use when deciding in what order to place things. • Enables the priorities of different people to be compared. • Can be used in interviews or on their own • Can lead to more direct and revealing questions (for example, Why is X a more serious problem than Y?).

25. 6. Qualitative data collection – tools TYPES OF RANKING & SCORING • Preference ranking (where people vote to select priorities), • Direct matrix ranking or scoring (breaking down criteria for preference and scoring on each, such as scoring different kinds of trees on a scale from 1-4 on their usefulness for fuel wood, building, fruit, medicine etc.) • Pair-wise ranking (where people choose between two options in different combinations)

26. 6. Qualitative data collection – tools TYPES OF RANKING & SCORING • Wealth (or well-being) ranking: • Can investigate perceptions of wealth differences and inequalities in a community, • to discover local indicators and criteria, and • to establish the relative wealth of households in the community. • Done by making a list of all households and asking different people to sort them into categories according to their own criteria of ‘wealth. • The term ‘well-being’ is often used, since perceptions of wealth usually include non-economics criteria. • Often only three categories are needed: the poorest, middle and richest. MODIFICATION: • When a list of all households is not feasible, as in a situation of recent displacement: • Get people to identify attributes (material and otherwise) of households in three categories (poorest, middle, richest), e.g. only the richest households have tin roofs, only the poorest use hand-hoes, etc. • Direct observation can assess how many households fall into each category.

27. 6. Qualitative data collection – tools MAPS & DIAGRAMS • Social maps: • Maps of a village or area showing where groups of people live. • Can be combined with wealth ranking exercises to identify which are the poorest households, landless, female headed households, different ethnic groups, number of children in a household, etc. • Similar maps can show key installations like water points, schools, and children’s play areas. • Seasonal calendars: • Ways of representing seasonal variation in climate, crop sequences, agricultural and income-generating activities, nutrition, health and diseases, debt, etc. • Can help identify times of shortage—of food, money or time • Daily routine diagrams: • Can help compare daily routines of different groups of people, and seasonal changes in the routines. • Can help identify suitable times for meetings, training courses, visits, etc.

28. 6. Qualitative data collection – tools MAPS & DIAGRAMS • Flow diagrams: • Shows causes, effects and relationships between key variables. • For example: Refugee and IDP movement; Relationships between economic, political, cultural and climatic factors causing environmental degradation; Flow of commodities and cash in a marketing system; Effects of major changes or innovations (impact diagrams); Organisation chart. • Venn diagrams: • Show key institutions and individuals in a community and their relationships and importance for decision-making. • Different circles indicate the institutions and individuals. • When circles are separate there is no contact between them. • When circles touch, information passes between them. • If circles overlap a little there is some co-operation in decision-making. • If they overlap a lot there is considerable cooperation in decision-making.

29. 6. Quantitative & Qualitative data collection – methods & tools Quantitative or qualitative method?

30. 6. Quantitative & Qualitative data collection – tools DISCUSSION What tools have you used in the past? What challenges do you face when designing tools?

31. 7. Sampling • Types of sampling • Purposive • Random • Stratified • Clustered Sampling parameters • Population of interest • Size (known/infinite) • Key characteristics • Significance level • Sampling frame – bias • Resources

32. 7. Sampling • Random sampling – key concepts • Central Limit Theorem • Confidence level • Confidence interval • Margin of error • Standard Deviation • Kurtosis

33. 7. Sampling DISCUSSION How have you been sampling refugees and non-refugees? How have you been stratifying your sample? How have you been randomising your sample?

34. 8. Field work preparations

35. 8. Field work preparations FIELDWORKPLAN • Outlines logistical issues that need to be followed and considered during fieldwork • Should include the following decisions: • Number, size and make-up of the assessment teams; • Allocation of assessment teams to specific locations; • Proposed itinerary of visits to specific locations; • Frequency of interim reporting from field teams; • Time to allow for fieldwork at each location; • How teams will travel; • Time to allow for travel; and • Where teams will eat and sleep.

36. 8. Field work preparations FIELD TEAM TRAINING • Should Cover the following topics: • Terms of reference for the assessment • Plan of action, including methodology to be used and time frame • Flow diagram linked to TOR for each position • Working relationships: responsibility of each team member, reporting lines, etc • Logistical arrangements for the assessment (transport, accommodation, etc.) • Security: existing situation and procedures during the assessment

37. 8. Field work preparations DISCUSSION

38. 9. Data analysis PRIMARY DATA ANALYSIS – Often made unnecessarily complicated! • No matter what kind of analysis you do (statistical or non-statistical), the main issues that you would investigate in any REACH assessment are: • CHANGE: how the situation is different now compared to before the crisis, crisis impacts and pre-existing vulnerabilities; • GROUP DIFFERENCES: compare the situation of different groups (age, gender, ethnicity); • GAPS: any holes in the information that you still need. PRIMARY DATA ANALYSIS PLAN – structure around: • Your indicator list • Potential correlations identified during data collection • Potential correlations identified during initial analysis & when presenting initial results to specialists • Potential correlations identified during Secondary Data Analysis

39. 9.a. Quantitative data analysis • DATA CLEANING • DESCRIBING RESULTS • GENERALISING RESULTS TO YOUR POPULATION OF INTEREST

40. 9.a. Quantitative data analysis DATA CLEANING • Missing values (blanks) – how to avoid them in the first place • Always make sure your categories for each questions includes all possible answers – no one should be forced to leave a question blank! • Missing values (blanks) – how to treat them in your analysis • Identify if non-random (e.g. caused by confusing question) or random (e.g data entry mistake, interviewee got tired) • IF very few missing values for one question = likely to be random • IF many missing values for one question = check question! • Could it be confusing/difficult to answer for respondents? = non-random • Or is it a question that is part of many options (e.g. individual expenses items that many may not spend on)? = likely to be random • IF many missing values for one interview = exclude it from the final analysis • Exclude missing values from your final analysis – if you don’t your results may be misleading

41. 9.a. Quantitative data analysis DATA CLEANING • Frequency errors – out of range entries • Check variables for out of range entries, e.g. HHs with an abnormally high number of members– can to a large extent be prevented by adding restrictions to ODK • Note that consistently high numbers in one interview may simply mean that it is a large household (as opposed to having e.g. 2 or 3 in all age groups and then suddenly 50 in males aged 50+ which is clearly and error) – use common sense! • Frequency errors – others • nobody ever went to school (schooling Y/N) but schooling expenditures recorded • more people at school than in HH • No medical expenditures but someone in HH received care, treatment, went to the hospital • Exclude cases with frequency errors from your final analysis – if you don’t your results will be misleading!

42. 9.a. Quantitative data analysis DATA CLEANING • Text entries • Make sure spelling is consistent • Review all text entries (e.g. ‘OTHER’) and categorize where enumerators have failed to assign to an already existing category – e.g. where daily labour has been entered as ‘Other’ instead of existing category • Where no categories exist, categorise by creating new binary variables (e.g. Daily labour) entering ‘0/1’

43. 9.a. Quantitative data analysis Describing CONTINUOUS variables – where values are numerical • E.g. household expenditure, household income, exact age of household head What to report: • Averages, maximum, minimum, distribution (standard deviation) How to show: • Bar charts – to show difference in means, maximums, minimums, across e.g. governorates • Line graphs – when showing evolution over time • Tables • Maps

44. 9.a. Quantitative data analysis

45. 9.a. Quantitative data analysis

46. 9.a. Quantitative data analysis • Averages do not tell the whole story - why we need to explore distributions….. One way of doing this is to look at Standard Deviation • Explains how far from the overall mean, each individual observation is on average • So here, the average distance from the mean is 10 IQD in the red population and 50 IQD in the blue population

47. 9.a. Quantitative data analysis

48. 9.a. Quantitative data analysis • One way of illustrating distributions – box plots • EXAMPLE: Distribution of food consumption scores within governorates

49. 9.a. Quantitative data analysis • Another useful illustrations of distributions – histograms • EXAMPLE: Average HH size is 5

50. 9.a. Quantitative data analysis Describing CATEGORICAL variables – values are categorical • E.g. pit latrine; flush latrine; etc • E.g. 0-3 years old; 4-6 years old; etc What to report: • Proportions How to show: • Bar charts – stacked • Pie charts (when few categories or when showing proportions within proportions)