Context-dependency, risk analysis and policy modelling

1. Context-dependency, risk analysis and policy modelling Bruce Edmonds Centre for Policy Modelling,Manchester Metropolitan University

2. Part 1: the Problem

3. “A large amount of research is pertinent to public policy. But a seemingly tiny amount of that knowledge makes it through to the policy world. This is a frustration not only for the research community but also for the policy community…” (Chris Tyler 2010)

4. The “gap” between the worlds of research and that of policy • There is often a mismatch between the wishes of policy advisors/makers and academic researchers in terms of the kind of conclusion desired. • Those in the policy world seem to wish for a relatively simple “best guess” (or tools to produce these) possibly accompanied by a supporting narrative • Those in the research community might be all-too aware of the difficulties of the issues under consideration and so wish to be much more cautious in their conclusions, and wish to include special cases, caveats and complexities.

5. Some Bad Solutions • The policy world can try to ‘force’ researchers to produce simple answers which academics are unhappy with or insist they solve impossible problems with inadequate means • Researchers can essentially try to ‘con’ those who commission research, claiming that they will answer all the questions that are specified, but re-defining the goals/terms as they go along or ‘discovering’ obstacles to them satisfying the policy makers/advisors as they envisaged • The researcher and/or the policy advisor mutually convince themselves and each other that the research gives definitive, simple and possibly general answers and rush to implement the policy indicated

6. The “gap” between the worlds of research and that of policy • There are many reasons behind these very different views… • …including different: kinds of goal, motivations, training, strategies, language, habits, constraints, working environments etc. • This talk will focus on a particular difficulty, that caused by the context-dependency of human socio-political behaviour and make some suggestions to improve things

7. Part 2: An Analysis of a Part of the Problem:Context-Dependency

8. A (simplistic) illustration of context from the point of view of an actor

9. Situational Context • The situation in which an event takes place • This is indefinitely extensive, it could include anything relevant or coincident • The time and place specify it, but relevant details might not be retrievable from this • It is almost universal to abstract to what is relevant about these to a recognised type when communicating about this • Thus the question “What was the context?” often effectively means “What about the situation do I need to know to understand?

10. Context-dependency of Cognition • “The” context is the situation of an event, but this is indefinitely extensive • The brain somehow categorises and recognises different kinds of situation and preferentially gives access to knowledge on this basis, it is context-dependent • Many aspects of human cognition seem to be context-dependent, including: memory, visual perception, choice making, reasoning, emotion, and language

11. The Context Heuristic • The kind of situation is recognised in a rich, fuzzy, complex and unconscious manner • Knowledge, habits, norms etc. are learnt forthat kind of situation and are retrieved forit • Reasoning, learning, interaction happens with respect to the recognised kind of situation • Context allows for the world to be dealt with by type of situation, and hence makes reasoning/learning etc. feasible • It is a fallible heuristic… • …so why do we have this kind of cognition?

12. Social Embedding • Granovetter (1985) • Contrasts with the under- and over-socialised models of behaviour • That the particular patterns of social interactions between individuals matter • In other words, only looking at individual behaviour or aggregate behaviour misses crucial aspects • That the causes of behaviour might be spread throughout a society – “causal spread” • Shown clearly in some simulation models

13. Illustration of Causal Complexity Lines indicate causal link in behaviour, each box an agent (Edmonds 1999)

14. Implications of Social Embedding • In many circumstances agents can learn to exploit the computation and knowledge in their society, rather than use or reason with their own knowledge • That is perceiving and acting throughtheir network of social relationships • Social embedding means that human behaviour can not be understood well separate from the surrounding social context • This is not a problem for normal human action as human cognition automatically (and unconsciously) adapts to context • However such normal action and cognition is hard to systematically justify or check

15. Some Researchers’ Responses to Context-Dependency A number of responses: • Only do research within a single context, resisting any generalisation • Only use discursive, natural language approaches where context is implicitly dealt with (and thus masked) • Try to see what (inevitably weaker knowledge) is general across the various contexts in what is being studied

16. Some Policy-World Responses to Context-Dependency • Wishing to know what is the “educated guess” as to which policy is best for most • Using their detailed and informed intuition as to what the best policy is and looking for a more “scientific” justification for it • Wanting a general narrative that “explains” (in comprehensible terms) why a certain policy direction is desirable or which can be used to build a consensus around

17. The “Simple is more General” Fallacy • If one has a general model one can make it more specific (less general) by adding more processes/aspects… • …in which case it can become more complex • However, the reverse is no true… • If one simplifies/abstracts then you don’t get a more general model (well almost never)! • there may be nosimpler model that is good enough for your purpose • But, even if there is, you don’t know whichaspects can be safely omitted – if you remove an essential aspect if will be wrong everywhere (no generality)

18. Context-Dependency and Randomness Lots of information lost if randomness used to “model” contextual variation

19. Context-Dependencyand “Being Scientific” • Ifthe relevant context can be reliably indentified then… • …context-dependency is notthe same as subjectivity • Generality is nice if you can get it, but its no good pretending to have it if you can’t • Science should adapt to what it wishes to understand, not the other way around • It does mean (often) an acceptance that general/generic approaches are not appropriate

20. What does notwork… • Inventing a model which only relates macro-level variables, not taking into account the social structures, using randomness to “cover” the variation across contexts (might as well use a straight line projection or random number generator!) • Presenting a policy maker/advisor with a “warts and all” model/account that is so complicated it is hard to work out a coherent policy response

21. Part 3: Some Suggestions

22. Fund Ethnographic Observations of Researchers Interacting with Policy Makers/Advisors • To “debug” what goes wrong (or right) with this relationship (randomly selected) projects need to be observed by a third party who… • Watches and makes notes about the interactions • Interviews the participants at various stages • Produces an impartial report reporting on the observations and conclusions (but anonymising the participants/project involved) • Participants get to add their thoughts on reading it as appended commentaries • Periodically collections of such reports be publically released

23. Encouraging the development of multiple models • Getting competing models from contrasting researchers for the same target • Allowing/encouraging different models for different aspects of complex issues • Encouraging the staging of abstraction from evidence to conclusions • Making models publically accessible at an early stage (well before the project ends) to allow comment and/or criticism • Models must be accompanied by documentation on: intended use, what is validated, whether replicated, etc.

24. Tools that could (used appropriately) help with context-dependency • Multi-dimensional data in its original state (or near to), that is different kinds of data about the same set of actors/events • Descriptive agent-based simulation where the different contexts agents might face are matched with different behavioural models • Local Data-Mining which searches out patterns that hold but also what parts of the data they hold for (indicating context-dependency) • Visualisation techniques which allow for researchers and stakeholders to explore, envisiage and check ideas about data • Tools to aid the tracking of relations between multiple models and multiple data sets

25. Layering Reports on Research Structuring Policy Modelling Research Conclusions: • A general summary of the results and recommendations without caveats • A short summary of the reasons for the above in narrative form (non-technical language) • A risk analysis specifying how the above might fail, i.e. the contexts under which the recommended policies will work less well or not at all • A description of the kinds of situation where the policy will be most effective (to aid targeting of a policy) • A (perhaps lengthy) appendix with details, maybe broken down into the various cases, documenting assumptions, evaluation criteria etc.

26. Towards some Rules for Engagement Establish a set of rules/norms that should hold in policyresearch interaction • Researchers accepting layered reporting in exchange for ensuring the risk-analysis does not get seperated from the summary advice 1&2 (which is what they fear) • An acceptance of context-specific solutions/models/specifications (or that general versions might not be applicable or effective) • Not asking for, nor offering, predictionsof complex social systems, but rather accepting evaluations of possibilities • Not asking for impossible things from researchers (e.g. “general solutions for complexity”) • Researchers being honest about their limitations and admitting when they fail (post hoc evaluation)

27. Writing “User Manuals” for Policy Advisors and Researchers Two Manuals: • A manual written for academics/researchers for how to deal with and communicate with policy adivsors/makers • Another written for policy makers/advisors of what to expect from academics/researchers, how to effectively use them and their shortcomings (as might be seen from the policy world)

28. The EndBruce Edmondshttp://bruce.edmonds.nameCentre for Policy Modelling http://cfpm.orgManchester Metropolitan Universityhttp://www.business.mmu.ac.uk