Lecture 1 What is a good question?

1. Research MethodsNational Research UniversityHSE Economics, Ph. D ProgramDr C S Leonard June 2011 Lecture 1 What is a good question?

2. Outline of lecture • Knowledge Claims • Classical Approaches • Post-positivism • Strategies of inquiry • Data mining • Design-based research • Good questions emerge from good research designs Research Methods

3. Knowledge Claims Research Methods

4. Knowledge Claims • What warrants knowledge? • How is scientific method applied? Research Methods

5. Two Positions Leonard GSOM Ph.D. Research Methods 2010 • Two scientific positions, inductivism and deductivism • From which method emerges • Deductive (logic) argumentation • If premises are true and no fallacies in the argument, then conclusion will be true • Not concerned with truth or falsity • Inductive arguments may have true premises, but we cannot be certain that conclusions will also be true (ampliative reasoning)

6. Inductive, Deductive Leonard GSOM Ph.D. Research Methods 2010 • Deductive, from general to particular, and inductive, from particulars to general • Inductive: Frances Bacon vs the medieval Church: purging ourselves of idols • Problem (Hume) Can we predict the future? No • Positivism is descended from Bacon • Research becomes historical, truth confined to a systematic empirical study, that might obtain general laws • Empirical findings worthless to some deductivists

7. Falsification: Karl Popper (1902-1994) Leonard GSOM Ph.D. Research Methods 2010 • Karl Popper’s critical rationalism hasgenerated much debate since the 1930s • Intellectual autobiography, Unended Quest • Background, early Marxist, training with Adler and Freudian theories convinced them that the theories were too broad • Later rejected psychologism • Favored theory of relativity, Einstein, could be tested, verified, falsified

8. Logic of Falsificationism • Scientific theories are abstract • can be tested only indirectly, by reference to their implications. • Scientific theory, and human knowledge generally, is irreducibly conjectural or hypothetical • to solve problems that have arisen in specific historio-cultural settings • Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory • A single counterexample is logically decisive: it shows the theory, from which the implication is derived, to be false. Research Methods

9. Vs inductivism Leonard GSOM Ph.D. Research Methods 2010 Direct antithesis of inductivism Growth of knowledge requires overturning previous beliefs Have a theory, test it, falsify it and move on Many economists seek to prove theories correct; the job of science is to disprove them Don’t strive for certainty (verificationism) Bans ad hoc adjustments to a theory to prevent it from being falsified A new theory will possess greater empirical content than its predecessors

10. Critics Leonard GSOM Ph.D. Research Methods 2010 • This hinders not promotes science • Can’t reject theories so easily, some theories are better at some things than others • Marxism, accommodates business cycles and disequilibrium better then orthodox Keynsianism, but the Monetarists better understand inflationary processes than the Keynsians, who argue that their theories and policies are more effective against unemployment • Should not critique a new theory too rigorously, because it may have something good in it

11. defender: Blaug • Blaug (be taken seriously) • Have a prediction about the future • Require a formal model • Falsification is essential • Be scientific, or not • Falsificationism: much tougher • Lay down restrictions on what Popper calls immunizing strategems Research Methods

12. Enormous influence Leonard GSOM Ph.D. Research Methods 2010 • Mark Blaug • Even econometricians, however, econometric results difficult to falsify • Plain fact • Most economists tend to verify..

13. Thomas Kuhn Leonard GSOM Ph.D. Research Methods 2010 • Structure of Scientific Revolutions • Science is not good science unless it is working under the framework of a theory, makes no progress • It adopts a particular view of the world, and all subsequent research adds to that • Pre science—lots of theories hoping to explain the same thing • The paradigm: an achievement so important that it attracts an enduring group of adherents • Commitment and consensus are prerequisites for normal science

14. Paradigms Leonard GSOM Ph.D. Research Methods 2010 Older generations stay with their paradigms, new ones acquire new paradigms Releases scientists from the necessity of debating fundamentals They can then concentrate on subtle, esoteric aspects of their subject

15. Kuhn: Normal Science Leonard GSOM Ph.D. Research Methods 2010 • Mopping up • Determination of facts • Setting the facts within theory • Articulating the theory • Then, anomalies, followed by crisis, followed by fundamental change

16. Applied econometrics Leonard GSOM Ph.D. Research Methods 2010 Middle ground Use theory, provide initial specification Data exploration techniques to extend or refine it Bridge theory and empirical data analysis How do we know a theory is correct? Different users have different tastes and beliefs Complications with computation: large data sets numerous models possible

17. Paradigms Leonard GSOM Ph.D. Research Methods 2010 How researchers will learn/what they will learn, assumptions Philosophical assumptions, epistemologies (how we know something), ontologies (what is knowledge), axiology (what values go into knowledge), methodology (process for studying)

18. Extensions Leonard GSOM Ph.D. Research Methods 2010 • These debates shaped much social science theory about • Innovations • Science • Path dependence • Historical legacies • Nature of change • Nature of reform and timing

19. Positivism • Can we be positive about our claims of knowledge when studying behavior and actions (Comte, Mill, Durkheim, Newton and Locke)? • Causes probably determine effects? • Reductionism: reduce ideas into small discrete sets to be tested Research Methods

20. Positivism vsInterpretivism • Interpretivism: Weber (Verstehen) Research Methods

21. Positivism and its Opponents • Quantitative, positivist, post-positivist research, empirical science • Challenge to positivism: against the traditional notion of the absolute truth of knowledge; playing tennis with the net down Research Methods

22. Methodological pluralism? • Bruce Caldwell (let 100 flowers bloom) • Little economics will survive if we take this seriously • Confirmationism Verification • Falsificationism is never practiced because it is unpracticeable Research Methods

23. Baysian methods • Test, verify and refine the laws and theories governing behavior • Baysian methods: from theory, to test, to revision Research Methods

24. (1) Post Positivism • Knowledge is conjectural • Research is to make claims and refine or abandon them • Data, evidence and rational considerations shape knowledge • Being objective is key Research Methods

25. (2) Social construction • Social construction • Mannheim, Burger, Luckmann, Neuman • Look at participants views • Judgments are subjective, meanings are varied and multiple • Interviews: open ended questioning, the more the interviewee talks spontaneously, the better • Participants allowed to construct meaning (rather than responding to concrete situations) • Process of interaction, context of work Research Methods

26. (3) Pragmatism • Pierce, James and Dewey • Knowledge claims arise out of actions, situations, consequences, rather than ex ante conditions • Concern with what works • Pluralistic approach • Mixed methods, qualitative, quantitative • Research always occurs in social contexts • Stop asking questions about the laws of nature Research Methods

27. Logic of Research Design Influence Research Design Research Methods

28. “Across most fields… applied economists are now less likely to pin a causal interpretation of the results on econometric methodology alone. Design-based studies are distinguished by their prima facie credibility and by the attention investigators devote to making both an institutional and a data-driven case for causality.” Joshua D. Angrist and Jörn-Steffen Pischke Research Methods

29. General to Specific Methodology “LSE” tradition of time-series econometrics that began in the 1960s at the London School of Economics Mizon (1995) a brief history The practitioners of LSE econometrics are now widely dispersed among academic institutions throughout Britain and the world. The LSE approach is described sympathetically in Gilbert (1986), Hendry (1987,1995, esp. chs. 9-15), Pagan (1987), Phillips (1988), Ericsson, Campos and Tran(1990), and Mizon (1995). For more sceptical accounts, see Hansen (1996) and Faust and Whiteman (1995, 1997) Research Methods

30. General to Specific Context: Linear (cross-country growth) Use: Time-Series Step 1. General regression will include every possible variable -- all the information about the true determinants. Step 2. The information content is then sharpened by a more parsimonious regression – the specific regression • it is statistically well specified (for example, it has white noise errors); • that it is a valid restriction of the general regression, and • that it encompasses every other parsimonious regression that is a valid restriction of the general regression Criticism: data-mining, Research Methods

31. Extreme Bounds Analysis: the new critique • Edward Leamer’s “extreme-bounds analysis” (1983, 1985). • A coefficient of interest is robust only to the degree that it displays a small variation to the presence or absence of other regressors. Leamer and Leonard (1983) define the extreme-bounds for the coefficient of a particular variable within a search universe as ranging between the lowest estimate of its value minus two times its standard error to the highest estimate of its value plus two times its standard error, where the extreme values are drawn from the set of every possible subset of regressors that include the variable of interest. A variable is said to be robust if its extreme bounds lie strictly to one side or the other of zero. Research Methods

32. Judgment Calls The main difference between structural and experimental (or ``atheoretic'') approaches is not in the number of assumptions but the extent to which they are made explicit. (Michael Keane) Research Methods

33. Summary • Quantitative (numbers) • Experimental design--controls • Non experimental design, surveys • Qualitative (words) • Narratives, phenomenologies, ethnographies, grounded theory, case studies • Mixed methods • Sequential, concurrent, transformative Research Methods

34. Quantitative • Random assignment of subjects to treatment • Quasi experiments: non random designs • Surveys cross sectional and longitudinal, generalize from sample to population Research Methods

35. Qualitative • Ethnographies: researcher studies an intact cultural group in its setting over time (responses) • Grounded theory • Derive an abstract theory of a process, action or interaction, grounded in views of participants • Case Studies • Exploring in depth a program, event, activity, process, or individuals, bound in time, variety of procedures • Phenomenological: lived experiences • Narrative research: lives, stories, retellings Research Methods

36. Mixed • Gets around biases in any one method used exclusively • Progress from one method to another • Illustrate • Determine what the concept is • Test assumptions on one case Research Methods

37. Logic • Qualitative: • Wider range of methods, non-numerical by definition • Small n, intensive interviews, depth analysis, discursive, account of event or unit • Focus on event, decision, institution, location, issue or legislation • Incident important in its own right (war, election, change in leadership, marketing strategy, community decision, etc) • Against bifurcation? mixed methods • Systematic, scientific research of all kinds • Most research does not neatly fit one or other category Research Methods

38. Goal of social science research • Inference (descriptive, explanatory) • Attempting to infer beyond immediate facts to something broader • Learning about causal effects from data • Public procedures (explicit, codified) • Replication • Conclusions are uncertain • Observes rules of inference Research Methods

39. Descriptive inference • Distinguish systematic from non-systematic features • Systematic from stochastic • Counter-factuals (what would have happened, had meters not struck the earth 65 million yrs ago) Research Methods

40. Rules for Research Design • Intuition: Choice of better topics is idiosyncratic • Two ways to test if it is a good topic • Is it about something important in the real world • A research topic should make a specific contribution to an identifiable scholarly literature by increasing our ability to construct verified scientific explanations • ie: locating it within the framework of existing social science literature • This is the subject of the second lecture today—what makes a theory or theoretical contributions valuable to the community of editors of journals Research Methods

41. Caution • There may be reasons a theory is practicable, even though its long term scientific value has been questioned • Theoretically incoherent models used to forecast the US economy—diversion of macroeconomic theory and applied macroeconomics (see Mankiw 1990) • New theories, however, remain speculative Research Methods

42. Design based Research • Leamer 1983 highlighted the benefits of sensitivity analysis, a procedure in which researchers show how their results change with changes in specification or functional form. Sensitivity analysis has had a salutary but not a revolutionary effect on econometric practice. • As we see it, the credibility revolution in empirical work can be traced to the rise of a design-based approach that emphasizes the identification of causal effects. Research Methods

43. Wow-Factor • Design-based studies typically feature either real or natural experiments and are distinguished by their prima facie credibility and by the attention investigators devote to making the case for a causal interpretation of the findings their designs generate. • Design-based studies are most often found in the microeconomic fields of Development, Education, Environment, Labor, Health, and Public Finance, but are still rare in Industrial Organization and Macroeconomics. Research Methods

44. Literature Review Research Methods

45. Literature Review • Classical • Systematic review • Meta-analysis • Narrative review • Search issues • Presentation Research Methods

46. Classical Literature Review • Large Disciplinary Differences • Sociology, Psychology, Business • Economics Research Methods

47. Systematic Review • A systematic review aims to provide an exhaustive summary of literature relevant to a research questions • The first step of a systematic review is a thorough search of the literature for relevant papers. • The Methodology section of the review will list the databases and citation indexes searched as well as any individual journals. Next, the titles and the abstracts of the identified articles are checked against pre-determined criteria for eligibility and relevance. Research Methods

48. More • Ability to control for between-study variation • Including moderators to explain variation • Deal with information overload: the high number of articles published each year. • It combines several studies and will therefore be less influenced by local findings than single studies will be. • Makes it possible to show if a bias for published works exists. Research Methods

49. Systematic Analysis • Meta-Analysis • Meta-analysis leads to a shift of emphasis from single studies to multiple studies. It emphasizes the practical importance of the effect size instead of the statistical significance of individual studies. This shift in thinking has been termed "meta-analytic thinking" Research Methods

50. Meta-Analysis(Average Effect Size, Forest Plot) Research Methods