Methodological Issues in Tax Research

1. Methodological Issues in Tax Research Jake Thornock University of Washington

2. 10 Commandments of Applied Econometricsby Peter Kennedy 1. Thou shalt use common sense and economic theory2. Thou shalt ask the right question3. Thou shalt know the context4. Thou shalt inspect the data5. Thou shalt not worship complexity6. Thou shalt look long and hard at thy results7. Thou shalt beware the costs of data mining8. Thou shalt be willing to compromise9. Thou shalt not confuse statistical significance with substance10. Thou shalt confess in the presence of sensitivity

3. Before we begin… You will find very little technical detail in this lecture, for the following reasons: • “One of the most prevalent sins of applied econometrics is the mechanical application of rules and procedures…” • Econometrics usually has a “standard procedure,” but in the real world, we usually have non-standard problems • Stop thinking that research design is a science—it’s an art! • Focus more on specificationthan on the estimation • The right question is more important than the right answer

4. Rule #1: Use economic (or some other) theory! • Theory: • Helps keep you and your paper grounded • Helps you speak the language of your peers • Disciplines your ideas; your ideas need boundaries! • Leads to better questions • If you don’t have a specific theory • develop a conceptual framework • outline the basic intuition of your question • You are no longer consumers of knowledge--you are producers of knowledge.

5. Rule #2: Ask the right question! • Common sense in research question • Does RQ make sense? • Avoid multi-linked questions A B C D E • Usually every paper boils down to one or two coefficients. Don’t overdo it… • Most papers are remembered for a one-liner. Ask/answer one deep question rather than threading together several small questions • Why is it interesting? • Do we learn something new? What is the innovation? • Are there theoretical implications? • Does it have policy implications? • Descriptive papers are hard to publish

6. Rule #3:Know the context! • You likely know the tax code; now learn the theory, context, setting, controversy, etc. • This is MORE than a literature review. • This is truly understanding the importance of your research question. • Have some institutional knowledge (i.e., real-world evidence) • 2 personal experiences > 100,000 observations • Before making your idea public, you should ask yourself: Do I really think this happens? • You better believe your story, because no one else will!

7. Rule #4: Inspect the data! • Carefully examine the data after every major change in the data. • Look for: duplicates, missing values, outliers, numeric vs text fields • You know data better than the computer! • All databases are noisy and biased! • Be skeptical about the data, even if its been used in prior research • CRSP, Compustat, IBES, Value Line, Zacks, Thomson, Execucomp, etc. • See “Database Biases and Errors” on WRDS • Look in detail at the descriptive statistics • Mins, Maxs, Skewness, Correlations, Influential observations • Plot the data – a picture tells a thousand words • Every data cut is an important empirical assumption

8. Rule #5: Do not worship complexity! • More complex ≠ more rigorous • Clean answers are rare---shoot for them! • Identification is key • In my opinion, • cleanliness > power; • internal validity > external validity • Stop speaking Greek (unless you’re from/in Greece or you’re around your frat brothers) • Over-doing it • Need an economic justification for your control variables • Double-clustered standard errors are not the de facto correction • Fixed effects only sometimes necessary • should often be interacted with variable of interest

9. Rule #6: Look long and hard at your results! • Here is where common sense is hugely important. • Magnitudes matter! • Do control variables line up with prior research? • Coefficients should fall within a range of reasonableness. • R2 should too • If there is a pop quiz on your paper (e.g., in a workshop), you should • Be able to interpret each coefficient • Have a prediction for both the sign and the magnitude of each coefficient

10. Rule #7: Beware the costs of data mining! • As costs of data management and analysis have gone effectively to zero,it is tempting to • Mine the data • Search for the perfect specification • Data mining and specification searches • Stem from poor research questions • Lead to mediocre results • Mediocre papers • Take the most time, • Are the hardest to publish, • But the easiest to overturn!!

11. Rule #8: Be willing to compromise! • You will face tradeoffs in research design • E.g., More control variables => more multicollinearity • Power vs noise • Applied econometrics is MESSY! • Trade in favor of • Economic or some other theory • Clean Identification • Internal validity • Consistency • Walk away from sunk costs: the mark of a good researcher is his/her ability to move on!

12. Rule #9: Statistical significance ≠ substance! • Form vs Substance. • A paper’s true substance is subjective and usually determined by others. • We live in a world of HUGE sample sizes (i.e., high power, low variance). Almost any relevant variable will have at least some influence. • We tend to worship p-values. Is 0.06 really that much worse than 0.05? • I think we have a lot to learn from p=0.34! • You will not get a job or tenure with a t-statistic. • Your career will be made by insightful research, steady teaching and being a good citizen … NOT t > 2.00.

13. Rule #10: Confess the presence of sensitivity! • Expect to be criticized! • Anticipate the criticism and then present tests of sensitivity • In nearly all cases, editors and referees are willing to allow for less-than-perfect results. • They are much less tolerant of over-selling the implications of the results • If your paper is published, some doctoral student will replicate it. • It’s better to be honest about its flaws/shortcomings than have someone else publish a paper about them.

14. The regression of life • Happiness = f( … ) • The model is/has • Relatively small “n” • Jointly determined • Exhibits clustering in time and age • Endogenous • Composed of non-linear relationships • Make sure your model • Includes the right independent variables • Well identified • Fully specified • You only have so many firm-years left in your life… • So make the best of them!