Out of the Box: Evaluation

1. Out of the Box: Evaluation • for Neighbourhoods North West • dr brendon Harvey • aspects associates • 07973 266976

2. today.... • explore evaluation and your context • why is a different view of evaluation needed? • put forward examples from practice of different ways of working with evaluation • encourage dialogue, critical reflection, a space of possibility

3. ..having new eyes? • “the real voyage of discovery lies not in seeking new landscapes, but in having new eyes”marcel Proust

4. A reminder...Creative Thinking • One of the main purposes for the use of • creative thinking is to find better ways of • doing things. It would be quite wrong to • assume that creative thinking means only • risk. Creativity also means insight and • new perceptions that at once make sense. • De Bono (1996)

5. Using Left Brain/Right Brain • Numbers • Sequences • Languages • Detail • Logic • Song Script • Analysis • Maths Formulae Daydreaming Spatial Awareness Colour Rhythm Forms and Patterns Big Picture Imagination Visualisation Musical Appreciation Duckett and Tatarkowski (2005) ...so what is the link with evaluation?..........

6. do you recognise this? • evaluation an onerous chore put off when the project has ended? • outside evaluators commissioned and then the project is told it has not accomplished its goals after much time and expense? • just a ritual by project directors to justify to funders that use of resources been successful and now we need additional funding?

7. performance management • The performance management movement. • Is the current, nearly exclusive emphasis on the OUTCOMES or results of programmes (the performance management movement) leading to flawed social policy and misguided judgements of programmes ? • Is there a commitment to collect information about WHY and HOW those results occurred?

8. understanding evaluation • what do you understand evaluation to mean in terms of your role? • Use the ‘post-its’ supplied and write on three words that sum up your understanding

9. evaluating potential? • … the most basic determinants of the extent to which one is likely to realize one’s potential are affective and conative (attitudinal, motivational) and not cognitive. Desire, internal motivation, and commitment are more important… than either domain-specific knowledge or knowledge of specific creativity-enhancing techniques… (nickerson in Sternberg, 1999) • what implications has this view for evaluation of individuals?

10. Why does this matter? • distinguishing between results that can be attributed solely to the unique intervention of these programmes, or be due to variety of influences WITHIN or OUTSIDE of the programme, or is it just CHANCE?

11. Have we sufficient information that has been researched to say that programme implementation may be influencing programme results? As performance management has grown in importance there has been a tendency to undervalue the critical role of evaluation research.

12. Being true to our principles • Great play has been made in recent years of empowering staff, citizens and communities. The message has been of empowerment, involvement, listening and engagement. Where evaluation has been concerned can we truly say that these principles have been upheld in the way we work with others to learn?

13. learning • Learning is paramount, especially when resources are tight. Such learning involves knowledge we can gain from the workings within a programme. but also, as important, the learning that can be transferred between programmes within an area, a region, a continent.

14. sound economic sense • Wasting valuable resource on a programme without any evaluation, continuing to support a programme that has been inadequately evaluated that seems to work but we don’t know why, or ignoring the knowledge and information locked up in the experiences of service users, practitioners and managers of programmes on the ground, does not make sound economic sense.

15. Planning and Managing • A Performance Management approach is a planning and management tool - it is there being driven by management accounting practices. THIS IS NOT WRONG in itself.... • ........but it cannot be our only evaluation process.

16. what are our projectslike? • simple and straightforward? • we can be certain what is going to happen? • there is always agreement on goals and actions to accomplish them?Take a recent project and discuss above

17. integrating knowledge and Experience • What point is there to have a solution to a problem if the elements of that solution are scattered in the knowledge and experiences of many participants (or non-participants) with no way to integrate them?

18. Action Evaluation: • “seeks to generate knowledge that is a useful, valid, descriptive of the work, and informative about how we might change it" (Argyris, 1985)

19. who is involved? • at root we believe in high participation. • viewing all stakeholders as legitimate participants in the evaluation process. • Action Evaluation, like Action Research, focuses on creating conditions of collaborative inquiry. • Evaluation is owned by participants

20. So what happens..? • data gathering on establishing shared goals - what, why, how? • Formative: During the project determining across stakeholders if these are being achieved • summative -The final stage which is the more conventional “so have we achieved” phase of determining the project’s success.

21. be like Lara Croft • collects a range of seemingly disconnected items, people and data as she travels, some of which prove indispensable and others surprisingly useless. But we don’t know which is which when she picks them up. • She also shows us that progress is not necessarily linear – going straight on can lead to a dead end but trying a few side alleys may create unforeseen opportunities. • Just when you think that the conundrum is unsolvable, you find yourself breaking through to the next level. • Lara falls and gets back up over and over again, inspiring us to keep trying and keep learning.