How close should experiments be to models and why?

1. How close should experiments be to models and why? • Drawing on “Experimental Economics: Rethinking the Rules” by N.Bardsley, R.Cubitt, G.Loomes, P.Moffatt, C.Starmer and R.Sugden, forthcoming, PUP. • Economics experiments tend to resemble formal models • Officially, this allays external validity worries • Internal validity = extent to which the lab results inform us about what’s happening in the lab • External validity = extent to which the lab results inform us about aspects of the world at issue

2. The Blame the Theory Defence • Vernon Smith (1982): “…what is most important to any particular experiment is that it be relevant to its purpose. If its purpose is to test a theory, then it is legitimate to ask whether the elements of alleged ‘unrealism’ in the experiment are parameters in the theory. If they are not parameters of the theory, then the criticism of ‘unrealism’ applies equally to the theory and the experiment.” • In other words: if we set up experiments to resemble economic models, we give theories their best chance. So we can ignore external validity if our aim is to test a theory. • But consider an example from James Lovelock:

3. Prof. David M. Bice, Department of Geology, Pennsylvania State University http://www.carleton.edu/departments/GEOL/DaveSTELLA/Daisyworld/daisyworld_model.htm

4. Economic Models as Claim-Ready Sets of Propositions • Theory Tx = (compound) claim T about aspect of the world x • Tx = mCx where m is (usually) a set of propositions with semantic content appropriate to x • m is expounded as a set of assumptions or stipulations, not assertions about the world • C is a predicate relating the model to the world • C might be realist, ‘as-if’, instrumentalist or parable-relating • C, and therefore Tx, is often oblique or implicit Cf. Lovelock’s explicit Daisyworld rationale: to demonstrate that purposeful action by nature is not necessary for life to regulate the Earth’s temperature. This could work by natural selection. Daisyworld is a prop for Gaia theory, which does not refer to Daisies as a mechanism at all.

5. Consequences • To test a theory by physically setting up the model is absurd [Daisyworld is a parable about the Earth!] • Whitehead’s (1925) “fallacy of misplaced concreteness” • Whether an experimental environment E tests a theory depends whether Tx predicts for it • This depends on the presence of x, not m: that E resembles m says nothing about this. • It doesn’t require intended instances of x; reference may be unintended

6. The Validity Problem • The artificiality of alteration versus that of contamination or isolation (Greenwood 1982) • ‘Relationality’ of social phenomena means Tx often does not make predictions for the lab • E.g. a jury trial requires a judge, and that participants recognise the judge’s formal authorisation etc. • The experimenter’s experiment may diverge from the subjects’ experiment (Orne 62, 73); internal (and so external) validity requires that the two converge • It’s the subjects’ experiment(s) that drives behaviour • Examples: Tax evasion experiments, dictator games

7. Tax Compliance and Evasion • πi = ei – t*di – I*m*(1-di); I = 1 or 0 • e.g. Alm et al. 1992: claims evidence that people don’t pay taxes out of moral / citizenship obligation • But a tax is a collection of revenue by the government, a specific authority;authority is relational • The data are on monetary gambles; nothing else is there • Ockham’s razor alternative for taxation (insurance &c) inferences • Reduction: E.g. suppose an experiment finds “dR/dt > 0”; R = t*Σidi • But the real tax-revenue relationship may vary with the perceived legitimacy of the government or tax: Seen as: Legitimate: dR/dt > 0; Illegitimate: dR/dt < 0 • So problems of causal holism are intertwined with relationality: • Tax-revenue-government-electorate-citizenship

8. Dictator Game Bardsley (2008) Dictator Game Giving: Altruism or Artefact? Experimental Economics,11 (2)

9. Taking Game Bardsley (2008) Dictator Game Giving: Altruism or Artefact? Experimental Economics, 11 (2)

10. Challenge: Demand Characteristics • Experimentalists: One can’t dismiss results of an experiment because DCs might be a problem • Sceptic: DC confounds might be frequent • Cf Placebo effects in medicine • Observations: • DCs seem plausible in some cases, not in others • Need for theorising about where DCs are likely to be (un-)problematic (e.g. inscrutable hypothesis?) • Need for empirical checking for DCs (à la Orne)

11. Theories and Domains (Cubitt 2005) Testing Domain Base Domain Intended Domain

12. Partner Search Decisions Insurance Purchase Decisions Children’s casino decisions Casino Decisions Theories and Domains (Cubitt 2005) Testing Domain Base Domain Intended Domain

13. Theories and Domains x in mCx Testing Domain Base Domain Intended Domain

14. Reflections on BTT and Falsificationism • BTT may still be valid if the experiment does implement x in Tx • E.g. most experiments on individual choice theory locate the design in the “base domain” of the theory, by using well-defined probabilities and outcomes for choices. Arguably, hypothetical choices (psychology) do not. • But without induction, we must say that failure of the theory in the lab is no guide to its future performance – either in the field or the lab! • Empirical work refines theories. Falsificationism is blind to this • If we always “give the theory its best chance”: 1. We may end up inductively refining our theories around phenomena that are externally rare, getting locked into our “own little world(s)”. 2. Success of the theory in the lab might be uninformative about its performance in the field. • Therefore even where BTT is valid for an individual design there are good reasons for researchers to pursue less model-constrained work.

15. Validity of BTT defence • Naïve Experimental Claim: • Any laboratory environment E in the base domain of a theory should be presumed to be in the T-domain • Modified Experimental Claim: • unless there is some difference between E and the I-domain, which can reasonably be expected to make behaviour in the I-domain markedly more consistent with the theory (Cubitt, 2005, Bardsley et al. 2008). • Proposed further modification: • or in the case of bold, Popperian testing, which can reasonably be expected to make behaviour in the I-domain less consistent with the theory • NB: E is in base domain only if Tx predicts for E • Implementation of m is neither necessary nor sufficient

16. Is there a trade-off between EV and IV? • local vs. broader EV; IV→local EV, absent demand problems • There is a trade-off between internal and external validity involving conditional probability: • “Own little world” designs may score highly on IV and weakly on broader EV • Designs closer to naturally-occurring situations make internal validity “harder” to achieve [e.g. free communication erodes statistical independence]. But conditional on achieving it we may be more confident that the results hold outside the lab. • This trade-off is about harnessing local EV to induct to situations of interest. Justifying claims to broader external validity is problematic even in the philosophy of natural science (Bruno Latour, Nancy Cartwright, Francesco Guala). • Very local EV has no meaning in model-implementing “Applied Economics” designs, literally interpreted.

17. Conclusions: How close should experiments be to models and why? • There’s no requirement for designs to resemble models; for testing, the requirement is for the theory to predict for the design • Proximity to a model may reduce internal (& so external) validity, since implementing the model may miss the target of Tx entirely • Where designs do implement phenomena of interest, one may wish to give the theory its best chance by sticking to its “base domain” • But that’s just one experimental strategy, we need more to break out of “own little worlds” – let a thousand flowers bloom! • Evidential claims of model-implementing AE designs are suspect unless couched as decision-theoretic and analogical • Such analogical claims maybe unconvincing because of their reductive nature, and differences between human subjects and target entities

18. References • Alm, J. McClelland, G.H. and Schulze, W.D. 1992. Why do people pay taxes? Journal of Public Economics, 48, 21-38. • Bardsley, Nicholas 2008. Dictator game giving: altruism or artefact? Experimental Economics, 11, 122-133. • Bardsley, N., Cubitt, R., Loomes, G., Moffatt, P., Starmer, C. and Sugden, R. 2008. Experimental Economics: Rethinking the Rules, forthcoming, Princeton University Press. • Cubitt, Robin P. 2005. Experiments and the domain of economic theory, Journal of Economic Methodology, 12, 197-210. • Greenwood, J.D. 1982. On the relation between laboratory experiments and social behaviour: causal explanation and generalisation, Journal of the Theory of Social Behaviour, 12, 225-249. • Lovelock, J. 2005. Gaia: Medicine for an Ailing Planet. Gaia books. • Orne, M.T. 1962 On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications, American Psychologist, 17, 11, 776-783. • Orne, M.T. 1973 Communication by the total experimental situation in P. Pliner, L.Krames and T. Alloway (eds.) Communication and Affect, 2nd edition, 157-191, New York: Academic Press. • Smith, V.L. 1982. Microeconomic systems as an experimental science, American Economic Review, 72, 923-955. • Whitehead, A.N. 1925 (1919). An Enquiry concerning the Principles of Natural Knowledge 2nd ed. Cambridge University Press.