Sheffield City Region Perfect Patient Pathway

1. EVALUATION KEY FINDINGS Steven Ariss, Mathew Franklin, Jeremy Dawson,Jennifer Read, Heather Dunn, Rachael Hatton, Elaine Scott, Nasrin Nasr, Becky Field Sheffield City RegionPerfect Patient Pathway Contact: s.ariss@sheffield.ac.uk

2. Programme Overview

3. The Test Beds • The NHS Five Year Forward View (2014, p34) “Develop a small number of ‘test bed’ sites alongside our Academic Health Science Networks and Centres. They would serve as real world sites for evaluating ‘combinatorial’ innovations that integrate new technologies and other novel approaches that offer the prospect of better care, and better patient experience of care, at the same or lower overall cost”

4. Sheffield City Region Area covered by the Working Together Partnership Sheffield City Region PEPPA Test Bed

5. Our Test Bed integrates technology for patient empowerment with service integration and strategic decision support: A central command centre combines real-time information and decision-making from across the health economy with sophisticated predictive analytics and population health stratification. Technology monitors key risk factors. Includes wearable and home-based technology and well as online resources Test Bed provides an ‘eco-system’ for testing and evolving technologies, before the most successful are rolled out more widely. Technologies promote patient activation and shift self care from the margins to the mainstream, as well as guiding active care planning for selected conditions Data available in real time, from all technologies and existing services Models of care are re-designed to exploit a co-ordinated, data-rich environment and patients empowered by technology Care and technology packages will be integrated around individual needs. Supporting digital health literacy for patients who need it.

6. ‘Big tech’/‘Little tech’; ‘Consumer tech’/’Provider tech’Devices/Infrastructure; Education/support • Categories determine the general approach to implementation Professional Training Needs SOS UK QTUG Digital Care Homes Categories can be determined by the details of the implementation context (hybrid consumer/provider tech) Care TRx: Asthma Diabetes

7. Evaluation Work-Streams Formative and summative with 3 work-streams: PROMS: Falls prevention= 49. Asthma=59 (EQ-5D: T1=75 & T2=34). Diabetes=33 Interviews=75. Focus groups=2 Effectiveness • Assessed on key performance and outcome measures, to determine the standards achieved by elements of the programme using comparisons where available. Economic analysis. Established by describing the costs of the programme, and associated benefits, specifically in relation to the key outcomes and PROMS (EQ5-D, ReQoL, ICECAP-A, PAM-13). Efficiency Attribution and understanding. Developed and tested using a Realist Evaluation approach, incorporating mixed research methods. Programme Theory/Process Evaluation

8. Rapid Change (owing to the need for fast implementation and encountering delivery-barriers and opportunities) • Diabetes: an attachment to an insulin injection pen that provides information about timing of previous injections a) Insulcheck classic, pilot (Community Nursing) b) Insulcheck connect (Outpatient Clinics) • Falls prevention: a falls risk assessment device using sensors and a tablet with bespoke software a) Primary care b) Community groups c) Hospital services (Balance Clinic) • Asthma: attachment to medication inhaler which links to software platforms (& phone app) to record medication use + remote behavioural intervention a) GP six-month intervention b) GP three-month intervention c) Hospital services (Outpatients) • SOS UK: a healthcare smart phone app that records medical information for access in an emergency and can contact carers to alert them in case of emergency (Promotion: Carers Centre, GPs, Newspapers, TV, Radio, Websites, Journals, Workplaces, Hospitals) • Digital care home: using vital signs monitoring linked to National Early Warning Scores (NEWS) and a digital communication platform in care and nursing homes to alert remote clinical teams in case of notable deterioration (Ongoing iteration)

9. Projects

10. Insulcheck

11. Insulcheck Pilot: Device was not suitable for a number of people: unable to consent, cognitive demands, and reduced dexterity. It was also not desired by some people: current strategies working, no problem remembering. Practical issues highlighted, which made the device unsuitable, such as use of two different insulin pens. Pilot recruitment details (community nursing) • 32 people approached by clinicians • 20 people declined to use the device • 12 people used the device • Of these 7 people consented and provided evaluation PROMS data • Of these 3 people consented and provided qualitative data InsulcheckConnect: Combination of innovators (Outpatients) • Bluetooth device • App and data sharing • Development of a new product

12. Asthma: TEVA/Care TRx

13. Asthma • An overall adherence level of 54.9% was recorded. This was after removing 53% of all scheduled events, which were considered to not be adhered owing to drop-out of participants • Positive perceptions of the project: meeting a need for some patients, enhancing self-management, supporting patient education, providing detailed information to GPs, iteration of technology design, and developing relationships and processes of collaborative working • Rapid deployment of technology alongside planning and build stage of the CareTRx Programme, combined with the many challenges being worked through in parallel (i.e. contractual, intellectual property, Information Governance, stakeholder engagement, evaluation design, ethics…) • Challenges of meeting the agreed recruitment target. Eventually 132 were recruited from 6,043 patients with asthma on GP lists. 46 patients attending the CareTRx Clinic could not be enrolled in the programme, due to several reasons. 4,500 invitations were sent out (2.9% response rate) • Innovator benefit: enabled iterative development/design of their product, being able to test it out in the real word, with all the challenges that involved, was useful to them as a company

14. Asthma Although there is no way to estimate change or attribute adherence levels to the intervention, it is worth noting that adherence in month-one was 65%, which is higher than Barnes et al (2015) highest levels of adherence in a normal population. Whilst this does drop-off over time, the mean average remains above 50%, which some studies define as above adherent level. Interestingly, there is 78% adherence in 50-59 year olds, which is close to the >79% highest definition of adherence in the Barnes et al (2015) review.

15. Asthma: Patient feedback Some patients found the programme useful as an adherence reminder but overall patients had to put effort into making the system work, often with technical glitches along the way. The market research reported the following patient feedback: • Low levels of participation and engagement with the app and website (improve introduction) • Lengthy set-up & close down clinics (reduce length & provide flexible times. Consider linking induction with routine appointments) • Technology glitches early on • Lack of personalisation and updated content (timing, length, content, goals and personal objectives. Offer ability to opt out of text messages) • Concerns about accuracy of data • Information was viewed as being more about supporting research rather than being of personal benefit

16. Asthma: Nurse feedback • Some patients were frustrated that nurses had nowhere to record their comments as to why they offered particular scores. Nurses themselves felt it didn’t reinforce the supportive role they felt they were supposed to offer to patients. • Questions that asked about depression and suicidal thoughts (e.g. “I feel life is worth living"), nurses received some antipathy from patients (“patients really, really did not like those questions”). This point was reinforced by patients during the focus groups. • Some of the question language was considered Americanised e.g. for the question “how much love do you feel you have”, some patients found this intrusive about their family life, some misconstruing this as a question about frequency of sexual intimacy, making them feel embarrassed. • Perceived ‘non-asthma’ related questions embedded the feeling that the project was really not about them as a patient and/or benefiting them. Some refused to respond to all the questions, which caused the system to record “incomplete"

17. Kinesis: QTUG

18. Kinesis: QTUG-GP

19. Kinesis: QTUG-GP • Questionnaire data available for 12 participants • 50% (6) reported having fallen in the last 12 months • All were on 4 or more prescription medication and many had issues with mobility, their feet, dizziness blood pressure and vision • Of the 10 subjects for whom we have baseline and 6-month follow-up data: • 3=decreased frailty estimate, 7=increased • 5 showed a decrease in falls risk estimate (range -30.76 to -1.13 percentage points). • 5 showed an increase in falls risk estimate (range 0.3 to 14.7 percentage points)

20. Kinesis: QTUG-GP • Exploratory economic model:

21. Kinesis: QTUG-GP • Exploratory economic model: • The falls prevention care pathway has a high probability of being cost-effective at a specified willingness to pay (WTP; e.g. £20,000 per QALY) threshold when screening and falls prevention interventions are utilised in a population aged 75 to 89 (89 is the upper age limit due to perceived inappropriateness) • The probability of the care pathway being cost-effective when screening is implemented in those aged 65 to 69 is almost zero

22. Kinesis: QTUG-GP • Relative cost-effectiveness of the care pathway is dependent on: • Accuracy of the falls risk assessment • Effectiveness of the falls prevention intervention to reduce the risk of falling and rate of falls • Cost of the care pathway (i.e. cost of the falls prevention intervention and falls risk assessment) • Rate of falls which require medical attention and have an effect on health-related quality of life which could be avoided • Cost of inpatient hospital care and social care

23. Kinesis: QTUG-Community The community group and balance clinic projects have shown potential to be effective approaches to identifying people at risk of falls and raising awareness. Further investigations would be required to estimate the cost effectiveness of these approaches. • 281 seen & 272 assessments completed • 137 assessed at high risk of falls and referred (50.7%) • 13 declined referrals (124 successful referrals) • 72 assessed at moderate risk of falls • 16 were referred (2 declined, 14 successful referrals) • 63 assessed at low risk of falls • 1 individual with several recent falls declined referral • 16 individuals in total refused referrals to the community strength and balance team (2 of these due to having had this intervention recently)

24. QTUG: Balance Clinic • Staff noted that the QTUG assessment provided a useful interface and opportunity to discuss issues related to falling (e.g. reassuring people that considered themselves at high risk of falling). • Takes around 5-7 minutes to do the test; ‘hence sits well in a busy clinic’. • 62.5% (35) of all patients were assessed (aged 65+) from total of 7 pts per clinic, one clinic a week (= 56 approximate total). • 14% (5) of those assessed were referred to falls prevention services (those with higher falls risk) – ‘more targeted referrals than previous random ones’ • Is continuing beyond the test bed programme

25. Humetrix: SOS UK

26. SOS UK Downloads and Comms Around 10,000 flyers were sent to individuals on the Carer’s Centre register and all Sheffield GP practices in Jun and July. There were emails sent to 71 GP practices in October and further flyers sent if required (14 practices). There was a further press release (23 newspapers, 14 TV and radio, 15 clinical journals, 18 med tech journals and websites) and email and Facebook activity in October and November 

27. SOS UK • Between August 2017 and January 2018 there were a total of 99 installations of the app, 88 instances of the app being started (89%) and 40 published profiles (40.4%) • 22 clicked the link to the evaluation survey website (22%), 5 completed surveys • The project demonstrated some successful approaches to collaborative working between an international innovation organisation, University evaluators and an NHS implementation team • Innovator Benefits: The experience of promoting the SOS UK app and engagement between innovators and the NHS resulted in innovation partners making other application developments • Linking communication activity with uptake There are noticeable peaks in download activity, which coincide with intensive periods of promotional activities • 5 survey responses: The most attractive feature of the app for these respondents was the ‘Red Button’ for making an emergency call. Four respondents wanted people to be able to access their health details in an emergency (QR code on lockscreen), three wanted to record health details to show a clinician and two wanted to track their own health details (self-recorded details). Only two of the five respondents wanted to use the app for all four of the main functions. One of the respondents only wanted to use the app for the emergency call function

28. Inhealthcare: Digital Care Homes

29. Inhealthcare: Digital Care Homes • 67 residents across 7 care homes (5 to 16 per home) • Compared emergency contacts pre-post intervention • 81 emergency contacts in the baseline period (44 A&E, 37 inpatient), and 60 in the intervention period (35 A&E, 25 inpatient)

30. Inhealthcare: Digital Care Homes • There is a slight but non-significant decrease in emergency contacts in the intervention period compared with the baseline period Incidence ratio 0.74 (95% confidence interval (0.45, 1.23)), p = 0.245 • Therefore we should conclude that there is no evidence of any change • Limitations: sample size is small (low power) & no comparison group. We cannot assume that the comparison is a fair one

31. Whole Programme Evaluation • Effectiveness in introducing sustainable processes, systems and behaviours for the identification, purchase, implementation and evaluation of combinations of new technologies

32. Whole programme: Challenges • Initial focus on technology, rather than needs-led service redesign (ongoing; identifying needs) • Fast recruitment required, despite long implementation chains; inter-organisational relationships; service readiness. Resulting in opportunism, constant change • Completeness of pathways including rationale and key outcomes • Imbalances in capacity and implementation timings • Coordination of service development and data availability with the evaluation • Establishing what evidence is required, for whom

33. Whole Programme: Reflections • High levels of uncertainty; could be reduced by increasing the readiness of projects (and accepting that this takes time) • Partners need allocated time and resources to build relationships and to reach a mutual understanding of each other’s intentions, expectations and approaches • Consider available evidence and what additional evidence is required (e.g. evidence reviews, developing business models and understanding the requirements of commissioners and decision-makers) • Initial concern should be whether the intervention works at acceptable cost and promises benefits eventually • Very few interventions resulted in sustainable, final models of service delivery: longer-term commitment combined with rapid evaluation and rapid change required for development

34. Whole programme: Tech development • Despite the technologies that were tested being considered market-ready, and had often been used in other settings, there was a considerable amount of useful feedback that was produced when implemented within the test bed. • Recommendations for product improvement and development were produced from engagement with implementation team, front-line staff, service-users and evaluators. • This would indicate that technology deployment does not always result in useful feedback, and this could be considered a crucial function for NHS infrastructure similar to the test bed. • The unique combination of organisations and functions integrated into a core team seems to be a critical condition for this mechanism to operate.

35. Whole Programme: Achievements • The programme management and governance infrastructure developed over the lifetime of the programme, specifically in terms of technology implementation when: • combined with pathway re-design, • incorporating co-design with a range of important stakeholders • and supporting the production of appropriate evidence to support development, spread, sustainability etc. • This learning resulted in, for instance; actionable tools; appropriate forums and means of engagement; systematic processes for technology assessment, documentation and administration processes; creative approaches to data sharing and integration

36. Whole Programme: Achievements • Large number of different technologies implemented in numerous settings over a short amount of time in primary, community and secondary care, third sector, social care and private nursing settings • Developed surrounding infrastructure to support commissioning, data sharing, integration and capacity development • Demonstrated the potential for this type of programme to support and drive the agenda for implementation and testing of health care technology in general; as well as carrying out a large number of specific projects

37. The Test Beds “There is no shortage of innovation in the NHS, but too often innovations do not have the reach or impact that would be expected in other industries. This is due to innovations being tested in isolation from the complementary NHS services needed to unlock their full potential. Innovations have also been implemented without rigour and discipline, generating little evidence about how to achieve impact in real world NHS settings” https://www.england.nhs.uk/ourwork/innovation/test-beds/

38. Digital Care Homes: Further Exploration • Recruitment and selection of residents: consider the inclusion/exclusion criteria for residents and the potential impact this may have on raising alerts • Monitoring and alert criteria: the appropriateness of the NEWS score for all people • Timing of observations: the best time for a deadline for uploading observational data (needs to fit with time for responses) • Dissemination & Feedback loop: how to update staff and stakeholders throughout the course of the project • Sharing data with residents: there was an appetite from residents for knowing their ‘results’ from the observations • Care plans: explore ways to support  - possibly with technology - MDT advanced care planning, including ceilings of treatment • Advice: explore appropriateness of advice for the care homes • Clinical/rapid response: clarify possible responses, provide examples so that SPA and care home staff are clear about when and how to use services

39. Digital Care Homes: Further Exploration Evaluation questions: • How has the long-term use of a digital monitoring service been received by key stakeholders? • How might a digital monitoring service be optimally designed for embedding in care homes? • What are the downstream effects from such a monitoring service on the healthcare system? 5 work-streams • Qualitative and quantitative description of alerts, subsequent activities, associated costs, to be developed into a process-based logic model • Qualitative interviews (currently 3 residents, 4 GPs, 4 SPA, 7 care home staff), Process evaluation, and Theory development • Care plan review • User-centred design • Evidence review

40. Digital Care Homes: Further Exploration Planned outputs • Descriptions of alert types and frequencies and associated activities, including assessed hypotheses of counterfactuals and causal relationships with potential outcomes and economic impact estimates of downstream activities • Qualitative report of key stakeholders experiences of the system • Recommendations for successful embedding in care homes with different characteristics • Recommendations on what other monitoring interventions might be incorporated into the platform and how this might be achieved based on feedback on ‘value’ to care homes (i.e. UTI, sepsis, hydration etc.) • User-centred design specification • Assessment of potential benefits

41. EVALUATION TEAM Steven Ariss, Mathew Franklin, Jeremy Dawson, Jennifer Read, Elaine Scott, Richard Simmonds, Nasrin Nasr Sheffield City RegionPerfect Patient Pathway Contact: s.ariss@sheffield.ac.uk