Evaluation use and usability: theoretical foundations and current state of international thinking Murray Saunders IOCE

1. Evaluation use and usability: theoretical foundations and current state of international thinking Murray SaundersIOCELancaster University

2. Mapping the territory

3. The urge to �sense make� in increasingly complex environments Evaluations tell us what is going on and what works?Social and political imperatives (issues of transparency, resources, legitimacy and equity) Evaluations contribute to public debate?Methodological debate: difficulties and uncertainties in addressing �end points� (attribution, causality, alignment and design) Evaluations provide authoritative evidence for policy and developmental strategy? Evaluations cost time and money Moving away from evaluations as expensive ritual, or as compliance toward evaluations as �use objects�?

5. Instrumental: when decision makers use the evaluation findings to modify the object of the evaluation in some way Conceptual: when the evaluation findings help program staff understand the program in a new way Enlightenment: when the evaluation findings add knowledge to the field and thus may be used by anyone, not just those involved with the program or evaluation of the program Process use: cognitive, behavioural, program and organizational changes resulting from engagement in the evaluation process and learning to think evaluatively.Persuasive or symbolic: persuade important stakeholders that the program or organization values accountability or when an evaluator is hired to evaluate a program to legitimize a decision that has already been made prior to the commissioning of the evaluation.Dreolin N. Fleischer and Christina A. Christie American Journal of Evaluation 2009 30: 158

6. Refers to the unintended or intended effects of the process of carrying out an evaluation: formative evaluation in action (Patton now talks about developmental evaluation-continuous adaptation) Foregrounding new issues for managing an intervention Drawing attention to �hot spots� or problem areas Forcing attention on difficult areas Providing a �voice� for the powerless Drawing attention to time-lines Making participants think about �audience� and �users� Policing role 

7. 

10. Stages of use of an evaluation output: use audit for stakeholders 0 No Awareness Awareness: little concern or understanding of implication 2. Informational: awareness plus interest in knowing more about implications for policy or practice Personal: beginning to analyse potential implications for policy and practice and impacts on planning (contribution) Management: attention on difficulties in the processes and tasks involved in developing new practices/policies on the basis of evaluation outputs Consequence: attention on impact on stakeholders of new practices/policies, their relevance, evaluation and implied changes derived from evaluation outputs Collaboration: co-ordinating and co-operating with others in using new practices/implementing policies 7. Refocusing: attention now on adaptation, major changes, alternatives to original ideas, creativity

11. Use as �engagement�: engagement practice

12. Differences in meaning and practice between use and usability Usability refers to the design of an evaluation: I suggests that there are 7 design decisions that can critically effect its usability Use refers to characteristics of the organisational context and its capacity to respond to evaluation outputs Both dimensions are important in explaining �high or low use environments�

13. �Use� refers to the way in which the outputs of an evaluation act as a resource for onward practice, policy or decision making� Towards a framework; what counts as Use?

14. This requires �bridging� or boundary crossing practices Evaluation output as a bridging artefact or tool: Providing examples of interesting practice Suggesting ways of moving from A to B (theories of change or engagement strategies Connecting with existing practices (en-grooved practice and how to un-block) Designing evocative resources for change management Moving to decisions on continuation, funding or new policy Validity and authenticity of the evidence is fore-grounded

15. Embed the output in decision making cycles (clear knowledge on when decisions take place and who makes them) Clear understanding of organisational memory (how evaluations might accumulate and how this output connects) Analyse the capacity of an organisation to respond Systemic processes (feeding into structures that are able to identify and act on implications) Organisations that are lightly bureaucratised (complex adaptive systems) are better placed to respond to �tricky� or awkward evaluations Strongly connect evaluation to power structures Evaluations that are congruent: recommendations from evaluation need to build on what is already in place. Avoid suggestions which need a total change unless there are resources to back them up. Towards a strategy to maximise use

16. �Usability refers to the way an evaluation design shapes the extent to which it�s outputs can be used� Towards a �utilisation� framework; what counts as usability?

17. Reasons and Purposes [planning, managing, learning, developing, accountability] Potential users know why an evaluation is taking place and have an intention to use it Uses [providing and learning from examples of good practice, staff development, strategic planning, PR, provision of data for management control, planning and milestones Rehearsing use environments in real time with real people by identifying a list of specific practices, for example: Tabling the report at a meeting to assess its implication Deciding on what those implications might be and acting on them Doing so in an agreed timeline Undertaking staff development activities on the basis of the findings Publicising and disseminating more widely etc � Designing evaluations for usability and use: critical questions produced through RUFDATA

18. Foci [activities, aspects, emphasis to be evaluated, should connect to the priority areas for evaluation] Create a need to know with key stakeholders by careful selection of foci (co-construction?) Data and Evidence [numerical, qualitative, observational, case accounts] Render evidence and data sets in ways that the non technical stakeholder can �read� them Audience [Community of practice, commissioners, yourselves] Discriminate between different audiences by style, form and content of output Timing [Coincidence with decision making cycles, life cycle of projects] Make sure the evaluation output and deliverable deadlines within a proposal coincide with other decision making cycles Agency [Yourselves, external evaluators, combination] Involve as wide a group as possible in design