Econometric Modeling

Econometric Modeling

How do we determine if econometric analysis is credible? • Leamer (1983) focused on robustness: • What is robustness? • The sensitivity of the results to the functional specification and other key assumptions • Concluded that there was little robustness and little empirical work was credible • Angrist and Pischke focus on the design base • Emphasize the identification of causal effects • Need real or natural experiments to clearly separate groups, allowing the analysis of causality • They argue that IO and macro have ignored this perspective

Empirical Microeconomics has experienced a “credibility revolution” • Much work is now considered empirically robust with significant policy relevance • Partly due to greater attention to robustness • But empirical research design has improved as well What is good research design? • Goal is to improve the data available for analysis • Gold standard is Randomized Controlled Trials (RCTs) • Another useful design is “quasi” or “natural” experiments • Use of instrumental variables • Selection correction (Heckman)

Main ideas behind RCTs • RCTs try to bring the controls of hard science research to social science analysis • Some treatment is envisioned • Participants are assigned randomly to the treatment and control group • Because getting the treatment is random, difference in the outcome, after controlling for covariates, is attributable to the treatment • Removes selection effects. What are selection effects • Covariates help control for differences in the way the treatment impacts differ across groups • A problem with RCTs is that there is selection into the experiment – people who agree to participate may be different than those not willing to participate

What is the idea behind natural experiments? • Basically the same as an RCT, but with less control in assignment to group • Looking for something natural that randomly assigns people into separate categories for getting treatment or not. • More rare than people like to think • Need to meet a high standard; many seeming exogenous differences are endogenous • Looking for something unrelated to the treatment that separates groups • Best are natural disasters, etc. Often different political outcomes are used, but that suffers from the “Tiebot” effect • Does eliminate the selection into RCTs problem

Gain in empirical microeconomics (Angrist and Pischke) • (Most fields) Applied economists don’t pin causal interpretation to the results based on econometric methodology alone • Focus is now on design, institutional or data-driven case for causality • But a caution about natural experiments and the Tiebot problem • Solon (1985) estimated effects of unemployment insurance on duration of unemployment spells • Compared states that recently changed standards • Ignores that the changed standards could be endogenous. Long spell states might have purposely tightened standards

Why things are better today • Better and more data • Fewer distractions on functional forms and methods • Understanding that regression and 2SLS approaches are good for finding average effects (and supported by econometric theory) • Differences in differences analysis • Better corrections for statistical anomalies like heteroscedasticity and serial correlation • Nonparametric analysis • Other advances like quantile regressions • Highlighting of specific forms of differences through better design

So what goes on in Macroeconomics? • Sims (1980, Macroeconomics and Reality) argued that structural macro models use assumptions about exogeneity to achieve identification • The “Lucas critique” and Kydland and Prescott, argue that can’t learn anything from econometric analysis of past policy changes • Would need to tease out the structural parameters underlying individual behavior • Policy changes change the constraints of that optimization problem • Inspired the work of Sargent and his group • Helped the move towards calibration models

More about Macro • Lucas claimed there is no way to do experiments in macro – the cost is too high • Angrist and Pischke claim Calibration models don’t do the job • Numbers are chosen so the theoretical model tracks real data • Produces no evidence of the magnitude or existence of causal effects, they are still assumed • So when a parameter is changed, the causal affects are still assumed • “Harmless, but still theory” • Romer and Romer (1989) and Friedman and Schwartz (1963) have looked for “natural experiments” in Fed behavior

Issues about experimental design • External validity: do the impacts that are observed carryover if the magnitude change of the variable used to define the experiment is very different? • Internal validity (the design) makes experiments narrow and idiosyncratic • But empirical evidence is always local to the data • The underlying variation never is completely representative, so extrapolation is always speculative • Calls for repeated experiments, with a range of values • Accumulate more evidence

Issues about experimental design (continued) • Relevance: experimental paradigm leads researchers to seek good natural experiments rather than to focus on important questions • Analyze game shows instead of poverty • But some supposedly “trivial” work has important implications • Applications of behavioral economics issues like presented-oriented bias in health club memberships reveals information about far reaching policy implications about obesity • Accumulate evidence across settings and designs, and we gain more general understanding

Sims Response • Economics is not an experimental science, and cannot be • Natural experiments and quasi-experiments are no more experiments than CGE model changes • Experiments must be replicable • Thoughts about Macro are nonsense • The problem is inference, not modeling • Same data are subject to different interpretations • Hence, use data to narrow subjective disagreements • He is very much in the spirit of Pearson; the best we can do is come up with models that mimic the data. We never have a model of reality.

Kennedy: Ten Rules for Applied Econometrics 1. Use common sense and economic theory • Use good statistical practices • Match like measured variables • Select functional forms appropriate for your dependent variable (beta function for a variable with values constrained between 0 and 1) • Don’t add trends for trendless variables • Don’t use a formula for your empirical work; think about what you are doing

2. Avoid Type III errors (producing the right answer to the wrong question) • Corollary, an approximate answer to the right question is worth more than a precise answer to the wrong question 3. Know the context, which means get the facts • How was the data collected and imputed? • How were observations selected? • These are parts of my Know your data rule • But also, understand the system you are trying to model

4. Inspect the data (I need say nothing more on this) • But put together graphs of the data to see patterns and anomalies 5. Keep it sensibly simple • Begin with simple models, then make them more complicated • This is the empirical analog to what I said about theoretical modeling • Conflict between complexity (general) and simplicity (specific) • Use the simplest method appropriate for your analysis

6. Use the interocular trauma test (what is this?) • Look at the results until the answer hits you between the eyes. • Then check that the results make sense • Signs, magnitudes, significance 7. Understand the costs and benefits of data mining • Goal is not a high R2 • Significance level is contextual • Specification depends on what data you have, and if it is relevant

8. Be prepared to compromise • Understand the gap between the statistical theory underlying your analysis, and the actual application you are doing • For example, there are few populations that are truly infinite 9. Do not confuses statistical significance with meaningful magnitude 10. Report a sensitivity analysis • Pay attention to robustness • Confess your errors and shortcomings (know the limitations of what you did, and admit to them)

Econometric Modeling