Service Evaluation Toolkit for Health and Social Care Commissioners

1. Service Evaluation Toolkitfor Health and Social Care Commissioners Avon Primary Care Research Collaborative 2015

2. Evaluation Toolkit Introduction Improving the quality of health and care is a priority for the NHS. Evaluation an integral part of policy making, strategic planning and commissioning. It can help you make judgements about the policy or service as well as understanding what works and what needs improving. It is likely that you are reading this because you have been tasked with considering how to evaluate a service or intervention or you have been asked to participate in an evaluation. This toolkit has been developed as a practical resource to help you to: • Frame the important questions about your services • Plan a pragmatic evaluation that will answer those questions • Share and use the findings from the evaluation This is a simplified approach, but by going through each section, you will gather the information you need to be able to evaluate your service effectively. How to use this toolkit The toolkit is set out in four steps: Assess, Plan, Do and Review. Each provides you with a set of templates and tools to support your evaluation. Used as a whole, this toolkit will help you formulate your evaluation plan, collect and analyse your data, and then use and share your findings. The toolkit is part of a series of documents, including a complimentary guide and a plain English summary so that you can evaluate with confidence. It is important to note that while the steps do not necessarily follow a linear process, by completing each step you will have addressed the main points required to carry out a well-planned and successful evaluation. Guidedtemplates Tools Top Tips Resources Note: Service is used through out but relates to projects, programmes, interventions, pilots etc

3. What is Evaluation? What is Evaluation? There are multiple definitions for evaluation, many of which have common themes around judging the merit, worth, value or quality of something. Our definition we have adopted locally was produced in collaboration with stakeholders by the West of England Evaluation Strategy Group is as follows: Monitoring and Evaluation We routinely monitor the performance of services however monitoring alone can only tell us so much – in that we are doing things – but it unable to tell us if we are doing the right things, what is working and what needs improving. However, monitoring data is a useful source of information to help you to evaluate your service. In turn planning your evaluation alongside your monitoring framework can help to reduce duplication and unnecessary data collection, shape and improve the monitoring data you plan to collect and identify appropriate baseline and benchmarking data. • What is Service Evaluation? • There are multiple facets to evaluation: • from research to a service evaluation; • from identifying areas for improvement to providing accountability; • from the simple (such as a single survey) to the complex (using multiple methods); • and it can be used in many different contexts. A central component to evaluation is one of comparison, either comparing before, with another service or with itself. • In health and care evaluation tends to be distinguished as either research or service evaluation. The key distinguishing feature between the two is dependent on the purpose of the study: • Research aiming to derive “generalisable” new knowledge beyond the setting it took place; where as • A service evaluation is performed to meet a specific local need and is focused on what is (or will form) current care with the evidence collected to inform recommendations not intended to be used beyond the setting it took place (although may be transferable to other areas). • This toolkit is focused on helping you to plan and conduct an appropriate evaluation. “A study in which research procedures are used in a systematic way to judge the quality or value of a service or intervention, providing evidence that can be used to improve it” (West of England Evaluation Strategy Groups, 2013) Health Research Authorities - Defining Research Leaflet - http://www.nres.nhs.uk/applications/is-your-project-research/ provides accepted definitions for research, service evaluation and clinical audit Charities Evaluation Service http://www.ces-vol.org.uk/about-performance-improvement/about-monitoring-evaluation guide to monitoring and evaluation

4. Top Tips These top-tips have been developed from years of experience of supporting those working in the NHS and Public Health with evaluation. They are useful practical tips to support you to start to develop your evaluation plans. Remember evaluations do require adequate resourcing, skills and time so it is important to ensure that your evaluation is feasible, proportionate and appropriate to the service being evaluated and that it will provide you with useful and valuable information. A useful question to ask yourself is “can you afford not to do an evaluation”?1 E.valua.tion Extremely Valuable Information2 • Ref 1: HM Treasury (2011) The Magenta Book. Guidance for Evaluation • Ref 2: Carolyn Cumpsty Fowler, John Hopkins University (2011) • http://www.odh.ohio.gov/~/media/HealthyOhio/ASSETS/Files/injury • %20prevention/evaltrainingfowlerpresentation.ashx

5. Assess Plan Do Review Navigation This toolkit provides you with a suite of resources (tips, tools, templates, signposting) to enable you conduct a service evaluation. You can use the tools and templates as a package or in isolation if you just need support with one aspect. The questions below will help you to navigate through the toolkit or signpost you to specific areas you would like help with.

6. Assess Plan Do Review • Resource 1. Do I need to be doing an evaluation? • Evaluation requires an investment of time, money and resources and so before you begin you need to assess whether you should be conducting an evaluation before making this investment. The following questions will help you to start to consider whether you need to and can do an evaluation, we recommend that you consider these in partnership with your stakeholders. You may not be able to answer them all straight away but useful to keep in mind when planning your evaluation. An Evaluability Assessment may be useful in determining whether the service is ready to be evaluated and what type of evaluation is required. For more information on “Evaluability Assessment” the following resources will help: http://betterevaluation.org/themes/evaluability_assessment

7. Assess Plan Do Review Resource 2. Understanding your service – Context When considering your evaluation you first need to understand your service in terms of the outcomes you want the service to achieve and for whom (what need are you trying to address?), what will deliver these changes and in what context. The following are aspects of the service that are useful to consider when planning your evaluation.

8. Assess Plan Do Review Resource 2: Understanding your service – Theory of change A useful process for planning your evaluation is to develop a “theory of change” for your service. This can be useful way of articulating and providing a visual representation of the links between the various activities of service and how this will lead to the long term outcomes it is trying to achieve. To develop your “theory of change” working with your key stakeholders, some flip chart paper, pens and post-it notes consider the following questions1: Who the service is for? i.e. your case for change will usually set out the population group, their needs and characteristics, the problem it is trying to address What are the long – term outcomes you want to achieve? Then working backwards What are the intermediate outcomes (short and medium term) that will lead to these long term outcomes? What are the activities that the service will undertake to deliver these intermediate outcomes? What evidence (from research and local learning) is available to inform and support the links between activities and outcomes? i.e. what are your assumptions based on the evidence, expertise and learning? What other factors need to be in place to enable this service to work? i.e. what are your enablers? NPC guide to developing Theory of Change1- http://www.thinknpc.org/publications/theory-of-change/ Kellogg Logic Model Development Guide: http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

9. Assess Plan Do Review • Resource 2: Understanding your service – Accessing and reviewing the evidence base • It is important to consider what evidence is available to inform your planning and decision making in terms of the service as well as the evaluation. You will be used to using a broad range of evidence from multiple sources including needs assessments, public health and performance data, evidence from research and best practice as well as expertise and local learning. This evidence can be used to inform your evaluation, help identify the outcomes you hope to achieve and the activities (processes) and outputs that will deliver these outcomes. Understand what evidence is already available to inform not only your service design, but the type and level of evaluation you need. Our evidence toolkit http://evidence.apcrc.nhs.ukcan help you with accessing the best available published evidence from research, evaluation and the grey literature. How does this help me? Understanding your service, the context in which it is operating, providing a visual representation of your service and understanding the evidence and theory behind it can help you to identify what your should focus your evaluation on and the type of evaluation you might undertake. The diagram below tries to outline how this helps.

10. Assess Plan Do Review Resource 3. Involving and engaging the right stakeholders. It is important to consider who will be interested in this service, evaluation or its findings. Engaging and involving all the right stakeholders right at the beginning is an important step not only in developing your service but also in ensuring a good evaluation. Stakeholders will be any one affected by the service or service change and this will include the users, providers and commissioners of the planned services. Engage all key stakeholders, including the patients and their carers, in the design, delivery and dissemination of the evaluation • Who are my stakeholders? • You may have already completed a stakeholder analysis as a part of your service planning, however if you have not completed one already then we recommend that you conduct a stakeholder analysis.This is ideally done as a group and could use existing forums such as the project team, steering group or advisory group. Don’t forget to involve your local and organisational experts in patientand public involvement, and your equalities and communications leads to help you. • Once you have conducted the stakeholder analysis this can then be used to inform: • Who to involve in the evaluation • What expertise and people are available to support the evaluation (including expertise in data collection, data analysis, communication, patient and public involvement) • Your project, evaluation and communication plans • N.B. Make sure you consider the needs of those that are seldom heard and are vulnerable. If you have not already, consider completing an equalities impact assessment for the planned service or service change. The NHS Institutes tool, part of its quality improvement tools, http://www.institute.nhs.uk/quality_and_service_improvement_tools/quality_and_service_improvement_tools/stakeholder_analysis.html will help you to conduct a stakeholder analysis. For more information about involving patients and the public then INVOLVE www.invo.org.uk, who are funded by the National Institute for Health Research (NIHR) to support public involvement in NHS, public health and social care research, are a useful resource.

11. Assess Plan Do Review Resource 4. Resources It is important to consider what resources are required for the evaluation from the outset. This may need to be reviewed throughout the planning process. The resource requirements will be informed by the type of evaluation you need (and quite often impact on the design of your evaluation). This is an important step and should be considered early when planning and developing your service (i.e. business case, QIPP scheme, service specification, intervention, programme or project). Plan your evaluation early in the commissioning cycle. This will enable you to ensure that you use the most appropriate methodology, collect the right data, allocate adequate resources and set appropriate timescales. Our service evaluation guide and website tbcprovides you with guidance on commissioning an independent external Evaluation and template to set out your invitation to tender.

12. Assess Plan Do Review Resource 5. Evaluation Planning Template This template sets out the key elements to consider when planning your evaluation. Don’t forget to build the evaluation into your project plans and processes as much as possible. Make sure it is feasible and utilises appropriate existing data that is aligned to, and informs, your monitoring plans. Health Foundations Evaluation: What to Consider Guide http://www.health.org.uk/publication/evaluation-what-consider is a very useful overview of what to consider when planning an evaluation

13. Assess Plan Do Review Resource 6: Setting Aims and Objectives It is important to get your aims and objectives right at the outset as this will guide your evaluation design.Involving key stakeholders in setting the aims and objectives of the evaluation helps elicit buy-in; promotes cooperation; ensures all interests are accounted for; and that expectations are managed. The overall aim will be a statement of what the evaluation will do. The objectives are narrower statements of what the evaluation will try to achieve in order to meet the overall aim. The case study below summarises a real-life process of co-producing evaluation objectives with a stakeholder group conducting a training pilot in partnership with volunteers to increase the skills and knowledge of people with Fibromyalgia. The group met to discuss what was important for each of them and considered how the objectives linked together. Note how some of the objectives meet the needs of more than one stakeholder.

14. Assess Plan Do Review Resource 7: Types of Evaluation – which type of approach should I use? There are a number of different types of evaluation, the most widely used are formative, Process and summative (outcome) evaluation. The following information provides you with a guide to the types pf evaluation, when they are used and the types of approaches and data collection methods that are associated with them. (Roberts, Cavill and Rutter 2009) http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx http://www.civicpartnerships.org/#!quantitative--qualitative-eval-methods/c1belhttp://www.institute.nhs.uk/research_and_evaluation/general/useful_resources.html

15. Assess Plan Do Review Resource 8. Data Collection Planning Template During the initial planning phase you would have identified the purpose of your evaluation (aims and objectives), the type of evaluation you might need as well as starting to consider the types of data and information you need to be able to answer your aims and objectives. Use the information you have identified so far as well as resources 9 and 10 to complete the following table, setting out your objectives against the data you plan to collect, the sources of that data and who is responsible for collecting and analysing the data. Identify and utilise existing data and data collection methods, where it is available and of good quality, to avoid duplication and unnecessary data collection. Make sure you collect baseline data, consider benchmarking and utilise validated tools where they exist. Consider using multiple (audit, quality improvement) and mixed (qualitative and quantitative) methods. Explore your evidence review to see how others have evaluated similar initiatives including whether there are any validated tools (for example surveys, patient reported outcome measures, patient reported experience measures) available that you can use?

16. Assess Plan Do Review Resource 9. Data Collection - Decision Tree Tool To help you decide how to collect the information or data you need to support your evaluation the following decision tree can help. Think about the types of information you need and then use this as the starting point to navigate the tree to identify a suggested method for collecting the information. This decision tree was developed in 2005 by Marsh and Glendenning so remember that with advances in technology there are also other ways to collect the information such as online surveys, text messaging, video messaging, social media, apps and other mobile technologies.

17. Assess Plan Do Review Having an adequate baseline for your key metrics is extremely important as a key aspect of evaluation is comparison, whether it is comparing with it self, before or after the service change or with another service. Benchmarking is also a useful tool to compare with other similar services or with national trends. Resource 10: Types of data and information

18. Assess Plan Do Review Resource 11. Am I doing an evaluation? Once you have identified the need to do an evaluationand planned your approach it is important that you check that you are doing a service evaluationand not research. To do this we recommend you use the Health Research Authorities tool to help you to assess this or contact your local experts such as your research and development department. We suggest that you keep a record of the result as a part of your evaluation planas this acts as an audit trail. We also recommend that you check with your own organisation to ensure that you comply with local arrangements when it comes to the governance and ethics of your evaluation. http://www.hra-decisiontools.org.uk/research/ Ethics and Governance If you are doing a service evaluation then you will need to comply with your local and/or organisational ethics and governance processes. These local and organisational processes may vary and so it is important to ensure that you obtain the right advice and seek the right permissions. If you are unsure check with your local experts, these could be a Research and Development lead, Clinical Governance lead, an Evaluation lead or a University Ethics department. Engage with your local research and information governance experts As a service evaluation you do not need any formal ethical approval from an NHS Ethics Committee, however every service evaluation should ensure they apply best practice when it comes to ethics ensuring that adequate safeguards are put in place and that the benefits of conducting an evaluation outweigh the risks. This will involve reviewing your evaluation during the design, delivery and dissemination to ensure any ethical issues are identified and actions are put in place to address them. This will include ensuring appropriate informed consent is obtained from participants, confidentiality and anonymity are maintained and the study complies with your organisation‘s information governance and data protection policies and procedures. Particular consideration needs to made for any participants that may be considered vulnerable such as children and people who lack capacity.

19. Assess Plan Do Review 12. Reporting Template The template below outlines the key information you need to provide within your evaluation report. Note: use this as a guide as your own organisation may already have a reporting template which incorporates your organisations house style. Charity Evaluation Service guide to report writing http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Writing%20better%20evaluation%20reports%20Final.pdf

20. Assess Plan Do Review 13. Action Planning template Once you have reviewed the results of your evaluation (and taken account of limitations of these findings) you can use these to make your recommendations for going forward based on the evidence. Ideally these would be reviewed by and agreed with your project team and tested out with your key stakeholders. The next step is to develop an action plan for implementing these recommendations. Your organisation may already have a standard template for this, however the following is a simple tool. To maximise the impact of your evaluation make sure you use your findings to inform your decision making going forward And identify areas for improvement NHS Scotlands Quality Improvement Hub Guide to Action planning http://www.qihub.scot.nhs.uk/knowledge-centre/quality-improvement-tools/action-plan.aspx

21. Assess Plan Do Review 14. Share your findings (Dissemination/Communication) planning template Working with your organisations communications experts and project team draw up a dissemination plan for your evaluation to maximise its impact. Consider who your audience is, what the different approaches are to communicating the messages and the different formats these could be communicated in and by. Where available, build into your project/service communications strategy. Use multiple formats to communicate your findings for the greatest impact and where possible utilise existing forms and channels of communication (engage your communications experts!) Better Evaluation blog on infographics http://betterevaluation.org/blog/infographics_to_make_your_eval_results_go_viral NHS Wirral’s guide to dissemination http://info.wirral.nhs.uk/document_uploads/evidence-factsheets/19disseminatethefindings11052011.pdf

22. Front Evaluation Planning Checklist (1 of 2) Planning Checklist Planning is a crucial part of the evaluation process. The following checklist (based on the original APCRC checklist) has been designed to help with the process of carefully planning your evaluation, outlining the key steps going forward.

23. Back Evaluation Planning Checklist (2 of 2) Step 5: Evaluation Plan

24. Additional Resources There are a number of resources available to support your evaluation, some are included within the toolkit to support specific aspects of your evaluation, this section provides you with other toolkits that may be of use to you. Evaluation Toolkits • CLAHRC Evaluation Guide: Developed by NIHR CLAHRC Leicestershire, Northamptonshire and Rutland in 2012 for clinicians and NHS Managers to help guide them through the process of evaluation • http://www.clahrc-cp.nihr.ac.uk/wp-content/uploads/2012/07/Evaluation_GUIDE.pdf • NHS Cambridge Full Evaluation Toolkit: Adapted from the PRIMARY CARE SERVICE EVALUATION TOOLKIT Version 1.5 Peter Marsh and Robert Glendenning to support NHS Cambridge CCG with Evaluation • http://clahrc-cp.nihr.ac.uk/wp-content/uploads/2012/07/Full_Evaluation_Toolkit.pdf Evaluation Guides • Magenta Book: Developed by HM Treasury to support evaluation of policy • http://www.hm-treasury.gov.uk/d/magenta_book_combined.pdf • MRC Framework: Developed by the MRC to help with evaluating complex health interventions • http://www.mrc.ac.uk/Utilities/Documentrecord/index.htm?d=MRC004871 • Health Foundations Evaluation: What to Consider Guide provides insight into the things you need to consider when planning an evaluation, the different types of evaluation and different methodological approaches • http://www.health.org.uk/publication/evaluation-what-consider Other useful resources • Charities Evaluation Service Evaluation Tools and Resources • http://www.ces-vol.org.uk/tools-and-resources • NPC Evaluation Tools and Resources • http://www.thinknpc.org/ • Social Value UK (SROI network) guide to Social Return on Investment • http://socialvalueuk.org/what-is-sroi/the-sroi-guide