Evaluation linked to strategy

1. Evaluation linked to strategy A presentation for the Trust Company MBS Symposium, October 24, 2013 Kate McKegg, Director of The Knowledge Institute & member of the Kinnect Group

2. “However beautiful the strategy, you should occasionally look at the results.”Winston Churchill Trust Company MBS Symposium, October 2013

3. Some common thinking about evaluation It’s just not that useful! It’s time consuming It’s expensive It’s produces long reports that are hard, and often boring, to read! It doesn’t necessarily provide us with answers to the really important questions we have It’s something we feel we have to do – but would rather not! Trust Company MBS Symposium, October 2013

4. What is evaluation? • It is the systematic determination of the quality, value or importance of something (investment, project, initiative, program, organisationetc) in order to take action It is underpinned by the collection of information / evidence about the inputs, activities and outcomes of something i.e., a project, program, initiative or activity Its key purposes are to determine how well something is doing or was done, what its value is or was, and if it is worth doing more of it Trust Company MBS Symposium, October 2013

5. Evaluation measurement • May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber. • They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment." Trust Company MBS Symposium, October 2013

6. ‘Results’ and ‘Performance’ • Mainstay of private sector and increasingly important in social/public / philanthropic sectors • In the public / social / philanthropic sectors - measurement is often considered to be one of biggest challenges… However…. • Establishing what results are to be achieved – and what performance actually means is probably one of the most difficult aspects of managing for results or outcomes Trust Company MBS Symposium, October 2013

7. Measurement & results • Private sector results – ‘the bottom line’ reasonably straightforward • Social / public sector / philanthropic results or outcomesare different – not as straightforward • Linkages between various levels of results are not always known and measurement is often very challenging • We often do not control outcomes …but we do seek to influence them Trust Company MBS Symposium, October 2013

8. Some challenges • Unpredictability of initiative direction and likelihood of emergent ‘unknowable’ outcomes • Many changes that are difficult to measure e.g., resiliency, equity, etc. • Lags between interventions, activities and outcomes • Difficulty in sorting out the extent to which changes have been influenced by our initiatives Trust Company MBS Symposium, October 2013

9. Common responses • Tail wags the dog: Only do what can be easily measured • Miss the parade: Evaluate only those ‘easy-to-evaluate’ things and miss the rest • Sink the ship: Try to evaluate everything extensively and end up capsizing along the way under the volume of effort, for very little return Trust Company MBS Symposium, October 2013

10. Defining performance, quality and value For good evaluation, we need an explicit way of defining performance, quality and value for ourselves and for others Trust Company MBS Symposium, October 2013

11. Performance & ‘goodness’ = values • E- valu - ation has three components Descriptive facts + values + evaluative (what so) conclusionsand action (so what?) • Values get us from what so to so what See E. Jane Davidson, Evaluation Methodology Basics, The Nuts and Bolts of Sound Evaluation, Sage, 2005 Trust Company MBS Symposium, October 2013

12. Visible When the ‘values’ step is transparent and explicit, then it is more likely to expect: • Genuine evaluative conversations about: • what ‘success’, performance and ‘goodness’ looks like • what constitutes ‘good’ or ‘excellent’ programme, organisational or strategy delivery and performance looks like Trust Company MBS Symposium, October 2013

13. Values in evaluation? On what basis will we decide or demonstrate to ourselves and to others that something we are investing in, or doing is high quality, valuable, worth doing, or important to pursue? And on what basis will we determine ‘how good is good’? Where will we get the criteria to make the judgments we need to make? It is values that we look to help us make these judgments. When ever we remark or comment on how good something was, or how well something was done, we are drawing on our values to make evaluative judgments about the quality and value of things. Trust Company MBS Symposium, October 2013

14. General guide for deciding how ‘good’ performance is (E.J. Davidson, 2010)

15. Many other rating options possible Trust Company MBS Symposium, October 2013

16. Mercalli Intensity Scale (1906) for earthquake impacts

18. An everyday example: How valuable was the last consultation about my son’s aspergers? • My criteria (values) for judging the quality and value of a consultation: • The extent to which I have confidence that GPs and specialists are working collaboratively around his care • The extent to which I have a clear idea of my son’s progress and future options • The extent to which Patrick is respected and included in the consultation • The extent to which GPs and specialists demonstrate an interest in Patrick’s wider life and context • The amount of stress involved in seeing specialists and GPs Trust Company MBS Symposium, October 2013

19. If the consultation was excellent? • I would have total confidence that my GP and the specialists were working together, and had full and shared knowledge of Patrick’s condition and his developmental progress. • I would leave the consultation with a thorough understanding of son’s progress and feel confident that there were practical future options that I and my family can use and are relevant to us to support his development. • I would feel that Patrick was being respected and included by everyone, that his feelings and point of view would be taken seriously • Being able to see the key specialists was a straightforward experience. Trust Company MBS Symposium, October 2013

20. My criteria and the data

21. What was the value / how good was the appointment ? Overall = Pretty Good

22. Evaluative reasoning – the critical component for evaluation of strategy Intelligent, well-informed and open-minded human reasoning is what is required to go through the systematic process of reaching judgements about quality, value and importance Trust Company MBS Symposium, October 2013

23. Evaluative reasoning Evidence does not “speak for itself” Arguments and reasoning are needed to connect evidence to judgment “think slow” vs “think fast” (Kahneman) “how do you know?” “discover not impose” Trust Company MBS Symposium, October 2013

24. Evaluation linked to strategy – it’s all about learning What? Now what? So what? Strategy - cycles of sensing, venturing, learning, and re-visioning Ongoing attention to strategy should focus on learning and adaptation Learning in real-time & adapting Evaluation + evaluative reasoning & thinking for reflection= learning and use in decision making Trust Company MBS Symposium, October 2013

25. What capabilities do we need? Trust Company MBS Symposium, October 2013

26. Demand Capabilities • Aptitudes, attitudes and skills that enable organizations to commit to and engage in evaluative thinking and practice Evaluation Capacity and Capability • Organisational Evaluation Capacity • Structural Capabilities • Institutional, management, process and resource capabilities that support the production and use of evaluative information. • Supply Capabilities • Skills, knowledge and experience that enable organizations to produce high quality evaluative information Trust Company MBS Symposium, October 2013

27. Linking evaluation & strategy Preparedness to lead and commit to, and hold yourselves accountable to outcomes or key results Make measurement of outcomes thoughtful, meaningful and credible Clear articulation of dimensions of quality, value and importance are in your context Commitment to reasoned evaluative thinking and interpretation of information about what is really going on Use the results for reflection and learning - and model for others serious use of results. Trust Company MBS Symposium, October 2013

28. THANK YOU For more information: Contact Kate McKegg, Director of The Knowledge Institute Ltd www.knowledgeinstitute.co.nz (a member of the Kinnect Group www.kinnect.co.nz ) Email: kate@kinnect.co.nz