Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask

1. Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask January 10, 2008 Presented by: Yvonne M. Watson, Evaluation Support Division National Center for Environmental Innovation Office of Policy, Economics and Innovation U.S. Environmental Protection Agency and John McLaughlin McLaughlin Associates

2. Workshop Objectives Participants will learn: 1) when to use qualitative data; 2) what data collection methods are available; 3) how to select participants for qualitative data collection; and 4) the steps for analyzing qualitative data.

3. Session Overview Module 1: Data Collection I. Overview II. Qualitative Data Collection Methods: • Interviews • Focus Groups • Survey/Questionnaire (Open-ended questions) • Document/File Review • Observation Module 2: Data Analysis III. Steps for Analyzing Qualitative Data IV. Assessing the Rigor of Qualitative Data Module 3: Appendix, Resources and References

4. Module 1: Data Collection

5. Orientation Exercise As a group, discuss your perceptions regarding qualitative data versus quantitative data with respect to: • Quality • Collection • Analysis • Utility

6. Numerical data Highly structured Creates precise measures Relatively easy to analyze May not explain “why” Closed Risk of bias Quantitative and Qualitative Data Quantitative Qualitative • Text (Descriptions of reactions, opinions, behaviors, experiences) • Structured Unstructured • Creates lots of rich data regarding perceptions • Challenging to analyze • Labor intensive to collect • Risk of bias (evaluator and subject) (World Bank , Module 6: Data Collection Methods, Slides 20 and 21)

7. Quantitative Data • Answers questions about: How much? How many? How often? • Use quantitative data when you: • Want to do statistical analysis • Want to be precise • Know exactly what you want to measure • Want to cover a large group or population • Quantitative Methods: • Examples: Survey questionnaires, tests, checklists, monitoring data. • Often used to obtain information on outcomes and causal relationships.

8. Qualitative Data • Answers questions which begin with: Why? How? In what way? • Use qualitative data when you: • Are concerned with opinions, experiences and feelings of individuals producing subjective data. • Want anecdotes or in-depth information • Are seeking understanding, themes, issues • Are not sure what you want to measure • There is no need to quantify • Are unable to collect quantitative data • Qualitative Methods: • Examples: Interviews, focus groups, document review, direct observation. • Often used to obtain information on processes, meanings, in-depth understanding.

9. Levels/Types of Qualitative Information • Different levels/types of information can be gathered from respondents. • Formulate questions that yield information regarding: • Reactions, feelings and emotions • Opinions and values • Knowledge and learning • Changes in skills • Behaviors/experiences • Effectiveness • Background/history/context (Hancock 1998)

10. Considerations in Selecting a Data Collection Method • Your evaluation or study question • Stakeholders’ desired sources of data • Resources (Financial and Skills) • Time (available to collect data) • Access to and availability of subjects/respondents • Information Collection Request (ICR)

11. Qualitative Data Collection Methods

12. Data Collection Methods (con’t)

13. Format Structured Semi-structured Unstructured Questions Open-ended Closed-ended Sequencing Location In-person Telephone Duration Selection of Interviewees Equipment/Supplies Recorder (tape or digital) Laptop Note Paper Schedule Interviewer Skills Resources Financial, Staff Interviews: Things to Consider

14. Interviews: Format • Structured Interview - Interviewer asks a specific set of questions of each respondent in the same way. This allows the interviewer to obtain a uniform set of data from each respondent. • Semi Structured- Includes a series of open ended questions based on the subject of interest to the interviewer but provides flexibility to explore issues in greater detail. • Unstructured Interview – General sets of questions are asked so that subjects respond in a free flowing manner resembling a conversation. The interview is designed to find out more information about a topic. (Hancock 1998)

15. Interviews: Questions Open-ended Questions: Solicit additional information from the respondent and will require more than one or two word responses. Respondents are encouraged to explain their answers. • Advantages: • Respondents can provide more information about a subject. • Researchers can better understand respondents true feelings, reactions about an issue. • Allows for an unrestricted response • Disadvantages: • Time-consuming • Challenging for respondents that are less articulate (Hancock 1998)

16. Interviews: Questions (Cont’d) Close-ended Questions: Limit interviewee’s responses to a pre-existing set of answers e.g., yes/no, true/ false, or multiple choice with an option for other or a ranking scale response option can be used. Questions can be restrictive and can be answered in a few words. • Advantages: • More easily analyzed • Answers can be assigned a numerical value • Questions can be more specific • Disadvantages: • Can yield incomplete responses • Discourages disclosure • Results could be misinterpreted

17. Interviews: Questions (Cont’d) Example 1: • Open-ended: Tell me about your relationship with the program’s Project Officer. • Closed-ended: Do you have a good relationship with the program’s Project Officer? Example 2: • Open-ended?: Can you describe your satisfaction with the program? • Closed-ended?: How satisfied are you with the program? □ Very satisfied □ Somewhat satisfied □ Dissatisfied

18. Interviews: Location, Duration and Schedule • Location • Decide whether to conduct in-person or telephone interviews. • Select a time and place that is quiet and free of distractions. • Duration • Schedule the interview for no more than 1 hour. • Schedule • Leave ample time to review transcripts and notes after each interview (at least one hour).

19. Interviews: Selection of Interviewees How Should Participants Be Selected? • Snowball sampling: Identify a few members of the community of interest, and then ask them for additional contacts. • Contrasting cases: Select cases with high contrast to learn about what underlies the differences between them. • Typical cases: Select cases that appear to represent the average, normal, typical situation. • Critical cases: Select cases that are considered to be crucial to understanding the study/ evaluation topic or which are assumed to represent the perspective of many other cases. (Kakoyannis 2007)

20. Interviews: Equipment/Supplies Needed • Equipment/Supplies • Note paper, recorder (tape or digital) or laptop to record/document responses • Note taking tips • Take good notes without detracting from the conversation • Write while maintaining eye contact • If interviewee says something you want to capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question. (World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

21. Interviews: Skills and Resources Needed • Interviewer Skills • Identify an experienced Interviewer • Interviewer should be aware of any cultural norms: eye contact, direct questions, gender issues • Stick to the script: • If asking close-ended questions, ask exactly the way written. • If asking open-ended questions go with the flow, not too directive. • Avoid asking yes/no questions. Ask, how, who and why* • Don’t step outside of your role as an interviewer • Good listener • Resources • Ideally, have a second person to help take notes or use a recorder *In some instances, when the interviewer consistently asks the respondent why, it may be interpreted as aggressive (World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

22. Format Group Size Number of Groups Questions Open-ended Closed-ended Location In-person Conference Call Duration Selection of Focus Group Participants Equipment/Supplies Recorder (tape or digital) Laptop Note Paper Schedule Skills Resources Financial, Time, Staff Focus Groups: Things to Consider

23. Focus Groups: Format and Questions • Size • Recommended size of group is 6-10 • Focus group members should have something in common • Number of Focus Groups • No rules here. However, more than one is recommended to ensure sufficient information is collected. • Questions • Start broad and then be specific (Hancock 1998)

24. Focus Groups: Location, Duration and Schedule • Location • Comfortable, neutral, safe environment • Free from distractions and accessible • Setting: around a table or in a circle • Duration • Typically 1-2 hours (clear start and stop times) • Schedule • Piggy back on existing meetings/conferences • Do not over schedule: 2 or 3 in a day is plenty for one moderator/facilitator.

25. Focus Groups: Selecting Participants • May need to have homogeneous groups with respect to gender, race, social class, managers vs. staff etc. • Cultural norms are important. • Things to Consider: • What is the geographical spread of your potential participants? • Are there any specific inclusion criteria for selecting participants • Where or how could you obtain a list of potential participants? • Are there any pre-existing groups and what are the advantages and disadvantages of using members? (World Bank , Module 6: Data Collection Methods, Slide 73), (Hancock 1998)

26. Focus Groups: Equipment, Supplies and Resources Needed • Equipment • Bring equipment and supplies needed to document/record the focus group. Note paper, recorder (tape or digital), laptop • Resources • Facilitator and note-taker • Other • Consider providing food, incentives, childcare, transportation etc., to respondents

27. Skilled facilitation is essential. Facilitator should know the script so focus group appears conversational. Ensure that everyone is heard. Ask: “What do other people think?” State: ”We have heard from a few people, do others have the same views or different views?” Active listener Develops and adheres to ground rules Accept all views while managing differences of opinion. So we have different perspectives Probe for elaboration Tell me more. Manage time Closing off discussion and moving to next topic. Invisible: say as little as possible Let conversation flow across the table with minimal direction. Keep personal views outside the room. Focus Groups: Skills Needed (World Bank, Module 6: Data Collection Methods, Slides 78, 80 and 81

28. Format Type of Questions Open-ended Closed-ended Method of Administration Self-Administered vs. Guided by Interviewer Mail, Telephone, Electronic, In-person Length Duration Consider 10 – 20 minutes Confidentiality Response rate May decrease if mailed. Survey/Questionnaire (Open-ended questions): Things To Consider

29. Meeting minutes Organizational mission statements Letters, records and laws Memoranda Correspondence Official publications and reports Personal diaries Photographs and memorabilia Progress reports Studies Document/File Review Collection and examination of documents produced in daily life as a means for better understanding the values of people in the study.

30. Document/File Review: Things To Consider • Type of information • What data are you looking for? (context, process, outcome, satisfaction) • Accuracy • Were data accurately recorded? Is it trustworthy? Has it undergone QA? • Access/Availability • Is permission needed to access files? • Are files in a central location or dispersed geographically? • Completeness • Are data available for appropriate years, stakeholders

31. Document/File Review: Things To Consider • Confidentiality • Can data be shared publicly? Do legal restrictions exist? (e.g., CBI, personnel data) • Informative • Will data collected from the files help provide information to answer the study question? • Time • Does the volume of documents/files increase the level of effort needed to complete the review?

32. Document/File Review • Advantages: • Unobtrusive • Analysis can easily be replicated because the data are stable • Documents can allow broader coverage of data by giving insight into past events that form the context within which the current study is operating in • Often less expensive and faster than collecting original data • Disadvantages: • Difficult to access and retrieve certain documents • Data gaps exist • Data do not explain why something is occurring/ happening • Data may not be “exactly” what is needed • Selection of documents might be biased if researcher does not collect a broad range of data

33. Exercise 1: Selecting a Qualitative Data Collection Method

34. Tribal GAP Data Collection Efforts • Reviewed a sample of files for 111 Tribes in 9 EPA Regions w/ Federally Recognized Tribes • GAP Accountability Tracking System • Grants Information and Control System • Audit Database • Strategic Goals Reporting System • Reviewed Regional Files (e.g., quarterly reports submitted by Tribes) • Conducted Interviews with GAP Project Officers in 8 Regions • Organized Panel Discussions w/Tribal Representatives • United South and Eastern Tribes (USET) Impact Week, Arlington, VA • EPA Region 5, Indian GAP Conference, Chicago, IL • EPA Region 8, Tribal Operations Committee, Denver, CO

35. Module 2: Analyzing Qualitative Data

36. Qualitative Data Analysis Analysis and interpretation are employed to bring meaning, order, and understanding to the data. (Taylor-Powell and Renner 2003) The purpose of qualitative data analysis is to describe, interpret, explain and understand data that are collected. (Dey 1993)

37. What is Content Analysis? A systematic process for identifying themes and patterns in the data, coding and characterizing the themes in order to understand the issue being studied. (Russ-Eft and Preskill 2001)

38. Steps for Analyzing Qualitative Data Step 1: Focus the analysis Step 2: Get to know the data • Refocus the analysis if necessary Step 3: Create Code/Categorize the data • Check validity of codes Step 4: Identify patterns and themes using codes • Check categorization of coding Step 5: Interpret the data Step 6: Conduct member check (Taylor-Powell and Renner 2003), (McNamara 1998)

39. Step 1: Focus the analysis • Review the purpose of the evaluation • Review the key study questions • Using the research question as a guide, think about which parts of the text help inform that question • Consider a framework for analyzing the data • Processes – Data are organized to describe an important process • Issues – Data are organized to illuminate key issues (often equivalent of primary evaluation questions) • Questions – Responses to data are organized question by question • Concepts – Data organized by key concepts (Patton 2007)

40. Step 1: Focus the Analysis • Inductive analysis • Involves discovering patterns, themes, and categories in one’s data. Findings emerge out of the data, through the analyst’s interactions with the data. • Deductive analysis • Involves analyzing data according to an existing framework, e.g., the program’s logic model. • Use both • Build on the strengths of both kinds of analysis. For example, once patterns, themes, and/or categories have been established test the appropriateness of the categories. (Patton 2007)

41. Step 2: Get to know the data • Transcribe the data • Listen to audio/recorded tapes • Read notes and develop a transcript • Read through the transcript first as a whole • Make brief notes (in the margin) of interesting or relevant information you are seeing in the data (McNamara 1998)

42. Step3: Create codes and categorize the data Codes are labels, abbreviations or symbols that are used to identify a particular concept, theme, idea or behavior reflected in the data. Coding involves breaking down, labeling, comparing and organizing data in order to group them into similar categories. (Taylor-Powell and Renner 2003)

43. Step3: Create codes and categorize the data • Preset Categories: • Start with a list of themes or categories in advance, and then search the data for these topics. • Emergent Categories: • Rather than using preconceived themes or categories, you read through the text and find the themes or issues that recur in the data. (Taylor-Powell and Renner 2003)

44. Step 3: Create codes and categorize data • Review margin notes, and make a list of the different types of information found. • Review the list of data items and categorize them in a way that describes what it is about. • Categorize the code words into similar groups • As you read, add or modify the descriptive code words so they better reflect the newer data. • Consider whether they can be linked in some way. Develop major and minor categories if needed • Examine the list of minor and major categories of data. Compare and contrast the categories. (Taylor-Powell and Renner 2003)

45. Step 4: Identify patterns and themes • Identify common, recurring patterns and themes, ideas, words or phrases • Look for associations, connections and causal relationships in the themes • Display summaries of data to enhance/illuminate interpretation e.g., compilation sheets, flowcharts, diagrams, matrices; (Taylor-Powell and Renner 2003), (McNamara 1998)

46. Step 5: Interpret the data • Reflect on the themes and patterns and data collected to make sense of the data and to find meaning and significance • Draw conclusions • If possible relate these to other data sets (Taylor-Powell and Renner 2003), (McNamara 1998)

47. Step 6: Conduct member check • Share theories and conclusions with respondents to verify the accuracy of your interpretation

48. Exercise 2: Analyzing Qualitative Data

49. Assessing Rigor of Qualitative Data Demonstrating data analysis is rigorous is important given criticism and skepticism associated with qualitative data. The rigor of qualitative data may be addressed by assessing: • Reliability (of the methods employed) • Validity (of the interpretation of the data) • Internal validity (credibility) – Extent to which the findings are credible and the “reality” that is described are credible to the people interviewed. • External validity (transferability) – Extent findings can be generalized to a larger population of people, settings, or situations. • Objectivity (Lacey and Luff 2001)

50. Strategies for Increasing Reliability and Validity Reliability • Describe the approach to and procedures for data analysis • Clearly document the process of generating themes, concepts or theories Validity • Consider and discuss alternative interpretations of the findings • Carefully consider and discuss cases and data that don’t fit overall patterns and themes, • Triangulate the analysis (use of multiple data sources) (Lacey and Luff 2001), (Patton 2007)