Two Cultures: Contrasting Qualitative and Quantitative Research Gary Goertz James Mahoney

1. Two Cultures: Contrasting Qualitative and Quantitative Research Gary Goertz James Mahoney

2. Contents 1 Introduction 2 Mathematical prelude: a short introduction to logic and set theory for social scientists I Causal models and inference 3 Causes-of-effects versus effects-of-causes 4 Causal models 5 Asymmetry 6 Hume’s two definitions of cause II Within-case analysis 7 Within-case versus cross-case causal analysis 8 Causal mechanisms and process tracing 9 Counterfactuals III Concepts and measurement 10 Concepts: ontology and epistemology 11 Meaning and measurement 12 Semantics, statistics, and data transformations 13 Conceptual opposites and typologies IV Research design and generalization 14 Case selection and hypothesis testing 15 Generalizations 16 Scope

3. Qualitative vs. Quantitative Research -- There are many differences across nearly all aspects of methodology. -- These differences are not well captured by the idea of words vs. numbers. -- There is no single difference that drives or explains all other differences.

4. Culture: A shared set of values, beliefs, norms, and practices. Alternative cultures are often associated with different toolkits or resources for solving problems.

5. We seek to promote cross-cultural understanding and communication. We believe this understanding must be founded upon a recognition and appreciation of differences.

6. Our goal is not to criticize either qualitative or quantitative research. We maintain a kind of anthopological neutrality about both cultures.

7. There is a place for qualitative, quantitative, and multi-method research in the social sciences. Multi-method research is essential for projects that require the analyst to pursue both qualitative and quantitative goals.

8. The nature of the research question and the goal of research shapes whether qualitative, quantitative, or multi-method research is most appropriate.

9. One Culture, Many Cultures, or Two Cultures? KKV: One culture founded on mainstream quantitative techniques. Quantitative tradition: Many subcultures – e.g., frequentist vs. Bayesian approaches.

10. Qualitative tradition: Many subcultures too. Split between behavioral and “causal inference” approaches (e.g., QCA, process tracing) vs. “post-positivist” approaches (e.g., interpretive analysis, critical theory, postmodern approaches).

11. Our two cultures approach: • Both of our cultures share with KKV a concern with scientific inference, including especially causal inference. This means that interpretive, critical theory, and postmodern approaches tend to drop out of our discussion.

12. One could write another book focused on the differences between our “causal inference cultures” and “post-positivist cultures.” That is not the book we wrote.

13. (2) We insist that there are two main cultures oriented to causal inference: qualitative and quantitative cultures.

14. Some Evidence for existence of two cultures: formal organizations, graduate training, informal networks. Other types of data: methods books, exemplary studies, and research articles.

15. We are focusing on actual practices, not necessarily best practices. In this book, we are neither judging nor coaching researchers. We are looking at what they are actually doing. We do hope that our descriptions are informative and useful.

16. Also, we are not focusing on “possible practices.” For instance, one might reconfigure fuzzy-set methods to do most of what regression analysis does. It is a possible or hypothetical practice. But no one does it in actual practice.

17. The qualitative research culture is the less well known of the two cultures. A couple of its big defining features are: (1) A focus on individual cases and the use of within-case analysis; (2) The implicit or explicit use of mathematical logic and set theory.

18. Process tracing tests provide a good example of both within-case analysis and set theory. For example, consider a “hoop test.”

19. Hoop test: Passing a hoop test is necessary but not sufficient for the validity of a given hypothesis. This kind of test can eliminate a given hypothesis but it cannot always provide strong support that the hypothesis is valid.

20. Example Hypothesis: O.J. Simpson intentionally caused the death of Ron Goldman.

21. Hoop test: Was O.J. in the general area at the time that Goldman was killed?

22. Some hoop tests are harder to pass than others: • Was O.J. on the planet Earth at the time that Goldman was killed? • Was O.J. at the Nicole Brown Simpson home at the time that Goldman was killed?

23. Failing a hoop test always eliminates a hypothesis. Passing a hoop test lends support in favor of a hypothesis in proportion to the degree that it is a hard test.

24. What makes a hoop test easy or hard?

25. The difficulty of a hoop test is related to the frequency at which the necessary condition is typically or normally present. Hoop tests that make reference to rarely present necessary conditions constitute difficult hoop tests.

26. Set of people on Earth Goldman’s murderer

27. Goldman’s murderer Set of people near Nicole Brown Simpson’s home at time of murder

28. Other hoop tests: • Is O.J. right handed? • Did O.J. have motive to carry out a violent murder? • Does O.J.’s hand fit the glove?

29. Logic and set theory are basic to most qualitative methods, including within-case methods such as process tracing.

30. Logic and set theory are not the same mathematics as statistics and probability theory. Within-case analysis is not the same approach as cross-case analysis. Qualitative research is different from quantitative research.

31. Causes-of-effects versus effects-of-causes Causes-of-effects approach: Start with an outcome to be explained and work backward to its causes. Effects-of-causes approach: Start with a potential cause and ask about its effect (if any) on an outcome.

32. Contemporary quantitative research: Favors effects-of-causes questions. In particular, quantitative research seeks to estimate the average causal effect of a treatment or independent variable.

33. KKV: Define causal effect in terms of average causal effect. Mortonand Williams (2010: 35): “A lot of political science quantitative research – we would say the modal approach – focuses on investigating the effects of particular cases. Sometimes this activity is advocated as a part of an effort to build toward a general model of the causes of effects, but usually if such a goal is in a researcher’s mind, it is implicit.”

34. The Neyman-Rubin-Holland model of causality “is purely a model of the effects of causes. It does not have anything to say about how we move from a set of effects to a model of the causes of effects” (Morton and Williams 2010: 99).

35. What about regression models that try to maximize variation explained? “If your goal is to get a big R2, then your goal is not the same as that for which regression analysis was designed. The best regression model usually has an R2 that is lower than could otherwise be obtained. The goal of getting a big R2 . . . is unlikely to be relevant to any political science question” (King 1986: 677).

36. We have found that few statistic articles now use R2 as a basis for explanation or evaluation of a causal model. Often the R2 in published work is quite low. (This is not intended as a criticism).

37. Qualitative researchers: They often try to comprehensively explain outcomes. What caused World War I? What caused sustained high growth in Korea and Taiwan? What caused social revolutions in France, Russia, and China? What caused the end of the Cold War?

38. To answer causes-of-effects questions, qualitative researchers must also employ an effects-of-causes approach. That is, they must establish that causes had certain effects. But qualitative researchers do not usually estimate average causal effects.

39. Instead, they tend to understand causal effects in terms necessary conditions and INUS conditions (i.e., conditions that are jointly sufficient for the outcome). So qualitative researchers have their own way of thinking about the effects of causes. It is rooted in logic and set theory.

40. Qualitative researchers often seek general explanations that apply to more than one case. However, for them, to provide a convincing general explanation entails providing a convincing explanation of individual cases.

41. Basic principle of qualitative research: A good general explanation of Y is also a good explanation of all individual cases of Y.

42. Qualitative researchers need to be sure that their causal model works in their individual cases. They do not view the estimation of a significant average effect as the end point or the main goal of research.

43. Quantitative and experimental research: Less oriented toward the individual case. Generating a good explanation for each individual case is not the main goal.

44. In fact, the Neyman-Rubin-Holland model of causality seems to assume it is impossible to estimate a causal effect for the individual i, which is precisely why one estimates an average effect for a population of causes.

45. Within-case analysis goes hand in hand with the effort to say something about the Xs that caused a particular Y. There is an affinity between within-case analysis and answering causes-of-effects questions in qualitative research.

46. Conclusion: • To understand and evaluate research, one must take into consideration research goals; • Qualitative and quantitative researchers often have different research goals; • Failure to recognize this fact generates miscommunication and misunderstanding.

47. END