Applied Opinion Research Training Workshop

1. Applied Opinion Research Training Workshop Sharon Felzer, Public Opinion Research Specialist, Development Communications Division (DevComm) Mary McIntosh, Princeton Research Survey

2. Why do Attitudes Matter? Mary McIntosh

3. Why do attitudes matter? • Attitude research is a valuable resource for the Bank to • Identify perceptions, underlying motives, and beliefs of specific audiences that influence behavior and shape attitudes toward policies, programs, institutions, organizations, etc.

4. Why do attitudes matter? • Perceptions of the Bank’s effectiveness • Where people believe the Bank needs to focus its efforts • Areas in which people feel that the Bank has failed in its objectives • Areas in which people feel that the Bank has not met their expectations • People’s perceptions of a Bank program’s impact

5. Why do attitudes matter? • We can differentiate people’s attitudes as a function of: • Standing in the country (e.g., elite, media, beneficiary) • Familiarity/experience with the Bank • Geographic location • Other attitudes, such as opinions on national government, privatization

6. Why do attitudes matter? • To distinguish between reality and perception: • Our attitudes color our perceptions of the world and it is our perceptions, not reality, that count • e.g., stereotypes, self-concept • e.g., one’s understanding of poverty reduction plays a large role in determining how effective they regard the Bank’s poverty reduction efforts Thus, attitudes are a critical element of evaluation

7. Why do attitudes matter? • To complement other evaluative approaches: • Attitude research is not programmatic • Attitude research provides a different kind of information than economic data • Attitude research is only as good as the instruments used and the data collection effort • The findings of attitude research are highly dependent on the sample used Thus, attitude research complements other sources of research

8. Why do attitudes matter? • Uses of attitude research: • Findings from attitude research can be used to inform country team strategies, program initiatives, project designs, etc. • Findings can also be used to improve an evaluation by making it more comprehensive and informed (potentially increasing the impact of the evaluation)

9. Using Attitude Research in Evaluations Sharon Felzer

10. Using Attitude Research • Why use attitude research when conducting an evaluation? • To understand the paradigm of perception vs. reality • As an assessment rather than in program/project preparation (ex-post rather than ex-ante) • To understand gaps in knowledge • To expose barriers to behavior/attitude change of Bank programs and services • To identify more effective ways to reach beneficiaries • To track attitudes, opinions, and behaviors over time

11. Qualitative Focus groups In-depth interviews (e.g., elites, sensitive issues) Brainstorming sessions Ethnographies Quantitative Polls Attitude surveys Tracking studies Using Attitude Research Research Methodologies

12. Quantitative Measures level of occurrence/distribution Asks how many? how often? why? with statistical analysis Provides evidence for hypotheses Objective Qualitative Provides depth of understanding Asks why? with follow-up Enables people to speak in their own words describing how they frame issues Enables hypothesis development Subjective Using Attitude Research 

13. Using Attitude Research • Common pitfalls of research efforts: • Poorly designed Terms of Reference • Poorly designed instruments • Poorly designed sample, poorly recruited focus group/interviewees • Unskilled moderators/interviewers • Cost overrun • No buy-in

14. Using Attitude Research • Tapping into sources of research • Budget issues exist • Use available resources to acquaint yourselves with your specific audiences. For example: • Government commissioned population studies • Global/regional polls done by research firms like PSRA, Environics, Eurobarometer, Gallup, and Latinobarometer. Access may cost a fee, but less than commissioning research yourselves. Make sure the research is relevant! • Add questions to omnibus surveys

15. Using Attitude Research • Using attitude research tools in limited efforts • Budget issues exist • Even when there is no budget for research, it is wise to incorporate research approaches into intelligence/ information gathering exercises. For example: • Carefully designed & recruited qualitative research efforts are appropriate for particular audiences in particular times. • These efforts require well designed guidelines (questionnaires for discussion) that ensure a consistent and systematic conversation followed by research-based analysis. • Focus groups/guided discussions with a carefully selected sample of the general population tease out important issues related to the Bank’s efforts. Less likely with opinion leaders.

16. Using Attitude Research • Ethics Attitude research is guided by stringent ethical rules laid out by organizations such as AAPOR, ESOMAR, and APA • The confidentiality of respondents’ responses must be maintained • The anonymity of respondents to the ultimate client must be maintained • Results should be reported clearly and appropriately, always within the limitations of the research • All precautions should be taken to reduce researcher/interviewer bias

17. Workshop Program Details • Exploration and hands-on work, including extensive use of case studies developed from a range of sources • The following areas will be covered: • Defining objectives and identifying potential hypotheses (Day 1) • Choosing a methodology and sampling (Day 1) • Designing quantitative instruments and qualitative instruments (Day 2) • Preparing to conduct research (Day 3) • Analyzing and reporting results (Day 3)

18. Defining Objectives and Identifying Potential Hypotheses Mary McIntosh

19. Objectives and Hypotheses • Defining objectives • Before starting any research program, you must first identify the objectives of the research • Why is it important to define your objectives? • Helps you to fully visualize the research project • Enables you to answer who? what? why? how? • Helps identify potential hypotheses/assumptions • Helps to determine a budget • Helps to ensure that the Terms of Reference clearly lay out the scope of the project

20. Objectives and Hypotheses • Defining objectives • In defining your objectives, you might want to consider the following questions: • Why does the Bank want the research done? • What are the goals of the research? • How does the Bank intend to use the findings? • Is the research to be part of a larger project? • What other research has been done that can complement your efforts?

21. Objectives and Hypotheses • Defining objectives • First, you should determine which questions need to be answered • These questions can be specific or broad depending on the research objectives • For instance: • What are benefits of debt-relief in the context of general foreign assistance, as perceived by client governments ? • How successful was a health project perceived by health professionals in a village? • Value of OED reports to Board staff?

22. Objectives and Hypotheses • Defining objectives • Finding the right balance in your research questions between being too specific or too broad is key • Don’t use attitude research to learn what you can determine from other sources • e.g., how may households in a community have electricity?

23. Objectives and Hypotheses • Defining objectives • You need to understand how your research objectives are different from the intended objectives of the program, services, or other Bank initiative that you are evaluating • To do this, you will want to speak to the task team leader, country team, etc. • Do the research objectives seem appropriate to the task at hand?

24. Objectives and Hypotheses • Defining objectives • Finally, you will want to provide a rationale for your proposed objectives • How will the research meet the objectives you have laid out?

25. Objectives and Hypotheses • Defining objectives allows you to: • Define the parameters of the research • Inform the design of the instrument • Organize the analysis • Logical hypotheses will then follow.

26. Choosing a Methodology and Sampling Mary McIntosh

27. Choosing a Methodology • Qualitative Research • What is it? • Open-ended: Questions are asked such that respondents are able to answer in any manner, at any length • How is it conducted? • Focus groups, in-depth interviews, brainstorming sessions, ethnographies • Guidelines (questionnaires for discussion) • Key sub-populations are recruited

28. Choosing a Methodology • Qualitative Research • Why use it? • Provides depth of understanding • Answers why? how? with follow-up • Can be used to develop hypotheses • Often used to inform quantitative research • Drawbacks: • Subjective • Results may not generalize • Language/translation nuances

29. Choosing a Methodology • Quantitative Research • What is it? • Close-ended: Respondents are given specific responses to choose from in order to answer questions • How is it conducted? • Polls, surveys, tracking studies • Fixed questionnaires • Random sample/quota sample

30. Choosing a Methodology • Quantitative Research • Why use it? • Provides breadth of understanding • Answers how many? how often? why? with statistical analysis • Measures level of occurrence/distribution • Results can be generalized to population • Objective • Systematic (can be replicated) • Drawbacks: • May not capture full picture

31. Choosing a Methodology • Qualitative vs. Quantitative Sample Questions: • Qualitative: Why do you think corruption is the most important development challenge in your country? • Quantitative: Which one of the following do you think is the most important development challenge in your country? Poverty, Education, Corruption, etc.

32. Choosing a Methodology • Qualitative vs. Quantitative Sample Questions: • Qualitative: How do you feel about the future of the next generation in your country? • Quantitative: How do you feel about the future of the next generation in your country? Very pessimistic, somewhat pessimistic, somewhat optimistic, very optimistic

33. Choosing a Methodology • Qualitative vs. Quantitative When to Use which Approach: • Question you want to answer • Participants’ ability to complete surveys • Type of data/responses desired • Sensitivity of issues • Budget

34. Choosing a Methodology • Qualitative vs. Quantitative When to Use which Approach: • You want to explore why people think the Bank has certain problems within a program • Qualitative • You want to find out how prevalent a certain problem is in Bank programs • Quantitative

35. Choosing a Methodology • Qualitative vs. Quantitative When to Use which Approach: • You want to find out how a few key decision makers feel about the Bank • Qualitative • You want to find out how a representative sample of key decision makers feels about the Bank • Quantitative

36. Sampling • Census (Full Population Representation) • The collection of data from all people in a given population • Sample • A subset of people from the population Any time data is collected from less than the whole population, a sample has been taken

37. Sampling • Random Sample • A sample in which people have been selected from the population by a process involving the use of random numbers. All people within a population have an equal probability of being selected • Non-Random Sample • A sample in which people have been selected by means other than random selection

38. Sampling • Random Sample • Used to make generalizations about a population or subpopulation • Reduces the possibility of sampling bias • Only type of sample for which inferential statistics can appropriately be computed • Does not guarantee that the sample will closely represent the population

39. Sampling • Simple Random Sample • Most common form of random sampling • Used when objective is to draw inferences about the whole population • Well-suited to drawing moderate-sized samples (50-500) from relatively homogeneous populations or larger samples (500+) from heterogeneous populations

40. Sampling • Sampling is as much of an art as it is a science • Often the size of the population is unknown • e.g, how many key decision makers are there in a country? • In qualitative research especially, it is often unclear how large of a sample is necessary • e.g., how many focus groups are necessary in order to understanding the perceptions of a Bank program?

41. Sampling • Response Rates • Percent of the intended sample from which at least partial data is actually collected • Low response rates pose the risk of substantial response bias

42. Sampling • Response Rates • How to maximize your response rate: • Clearly indicate how the requested information will contribute to something the respondents value • Provide credible promises of confidentiality • Use influential people to help solicit participation • Minimize the inconvenience and burden on respondents • Offer incentives (e.g., a report) • Be persistent (e.g., call back)

43. Hands-on-Work:Choosing a Methodology