*Brian Gugerty, DNS, MS, RN **Michael Maranda, PhD ***Dona Rook, MS, RN

1. The Development and Piloting of a Tool to Evaluate Satisfaction with Implementation of a Clinical Information System *Brian Gugerty, DNS, MS, RN **Michael Maranda, PhD ***Dona Rook, MS, RN *Siemens Medical Solutions,Malvern, Pennsylvania, USA **Independent Evaluation Research Consultant, Brussels, Belgium ***Pepin Heart and Research Institute, Tampa, Florida, USA

2. CIS Electronic Health Record (EHR) Business/ Admin. Systems or Modules Departmental Support Systems or Modules Clinical Information Systems or modules Other systems or modules Electronic Medication Administration Computerized Provider Order Entry Clinical Documentation (C) B.Gugerty & M. Maranda, 2005

3. Importance of CIS • Patient Safety • Care Improvement • Cost-effectiveness • EHR 2014 (C) B.Gugerty & M. Maranda, 2005

4. CIS Implementation T u r n O n Closure Go-Live Pre-implementation Post-Implementation Kickoff = phases = milestones , (C) B.Gugerty & M. Maranda, 2005

5. Why evaluate user satisfaction with CIS implementation • Billions of dollars are spent world-wide on CIS implementation • Yet few if any published instruments to evaluate user satisfaction with CIS implementation • impetus to develop an instrument to fill this void resulted in CISIES (C) B.Gugerty & M. Maranda, 2005

6. Clinical Information System Questionnaire (CISQ-15) • 15 item tool, measured clinicians’ perceptions CIS implementation • developed via grounded theory and observations of multiple CIS implementations • administered twice on same group (C) B.Gugerty & M. Maranda, 2005

7. CISQ Family of Measurement Instruments • Expanded CISQ-15 to CISQ-36 via an expert panel and piloting • Created • CISQ-ER • CISQ-LTC • CISQ-MA • CISQ-FR (C) B.Gugerty & M. Maranda, 2005

8. Change of Course • intent of the questions were not changing as we created the new instrument • specific referents were changing • type and/or name of system • title or role of clinician • department name • Validation of original grounded theory (C) B.Gugerty & M. Maranda, 2005

9. CISIES • Clinical Information System Implementation Evaluation Scale (CISIES) • system type, system name, staff role and department neutral • Aimed to create a tool that could measure clinicians’ perceptions of any CIS implementation (C) B.Gugerty & M. Maranda, 2005

10. Some Sample CISIES Statements • The system has improved my practice. • The system has added to my workload. • The system facilitates communication of patient information among members of our health care team. • Overall, the introduction of the system has been effective. (C) B.Gugerty & M. Maranda, 2005

11. Scoring the CISIES • 37 items scored on a six point Likert scale • Range: Strongly Disagree to Strongly Agree. • uses a forced choice format • most items are “positively” worded • during data analysis, the 10 “negatively” worded items are recoded (C) B.Gugerty & M. Maranda, 2005

12. Piloting of the CISIES • administered in a Florida to evaluate a recent Critical Care Clinical Information System implementation • administered in a confidential manner • IRB approval (C) B.Gugerty & M. Maranda, 2005

13. Characteristics of Respondents • 44 individuals returned questionnaires with useable data • respondents were staff nurses from two ICUs • women (68.2%) • two-fifths of the respondents were from 36 to 45 years of age (C) B.Gugerty & M. Maranda, 2005

14. Psychometric Characteristics of the CISIES • Cronbach alpha for this administration was .94 • too few respondents for a factor analysis • corrected Item-Total Correlation varied from .058 to .812 • several items with low inter-total correlations (C) B.Gugerty & M. Maranda, 2005

15. Results • Possible scores range • from six which suggests extremely satisfied to • one which suggests extremely dissatisfied. • Mean item scores ranged from a high of 4.84 to a low of 2.28. • majority of items had means of 3.5 or greater, which indicates • Satisfaction with the implementation, although not strong satisfaction. (C) B.Gugerty & M. Maranda, 2005

16. Report to the Site • report was written and returned to the site within two weeks • informatician at the site indicated that the report was very helpful in understanding staff’s attitude toward the implementation. • It provided an objective measure that was used to stimulate discussion about the implementation and how to improve it. (C) B.Gugerty & M. Maranda, 2005

17. Highlights of the Report • adequate level of satisfaction with implementation especially at this point post go-live • highest scores • 1. a strong commitment to the successful use of the system; (mean=4.84, n=43) • 2. a comparatively high satisfaction with system’s impact on team functioning; (mean=4.58, n=43) • 3. a comparatively high satisfaction with their department’s role in the introduction of the system; (mean=4.56, n=43) • 4. a comparatively high satisfaction with the training they received about the system; and, (mean=4.55, n=44) • 5. a comparatively high satisfaction with patient information being more confidential and secure. (mean=4.47, n=43) (C) B.Gugerty & M. Maranda, 2005

18. Highlights of the Report (con’t) • lowest scores 1. had a strong sense that people who use the system should have had more to say about the design of the system; (mean=2.28, n=43) 2. had a relatively strong belief that the system added to their workload; (mean=2.73, n=44) 3. believed that the system did not allow them to spend more time on other aspects of patient care; (mean=2.91, n=44) 4. believed that the use of the system had a neutral impact on the quality of patient care; and, (mean=2.93, n=44) 5. did not believe the system improved their practice (mean=2.95, n=43) (C) B.Gugerty & M. Maranda, 2005

19. Highlights of the Report (con’t) • Although results generally positive, the low item scores indicate room for improvement. • Based on the data, it is recommended that: • The site conduct focus groups of staff that use the system; • Based on the CISIES and focus group data, plan an informatics intervention aimed at increasing satisfaction with the system and execute the intervention; • Re-administer the CISIES; • Analyze both administrations of the CISIES and issue a final report (C) B.Gugerty & M. Maranda, 2005

20. Actions take to date at the Site in response to the Report (C) B.Gugerty & M. Maranda, 2005

21. Conclusions • CISIES needs to be studied further, but the results of this administration indicate that the CISIES is a useful instrument • easy to administer • acceptable to respondents • easy to score and understandable by non-evaluators • More administrations are needed before we can use factor analysis to better understand the instrument and empirically verify any subscales (C) B.Gugerty & M. Maranda, 2005

22. Implications • Formative Evaluation Purposes: • Assess overall staff attitudes towards implementation • Assess staff attitudes towards sub-dimensions of implementation • Provide data for post implementation phase improvement efforts • Decrease time to closure • Summative Evaluation Purposes: • Assess overall staff attitudes towards implementation • Assess staff attitudes towards sub-dimensions of implementation • Direct next project(s) towards expected results • Compare two or more implementations effectiveness (C) B.Gugerty & M. Maranda, 2005

23. References • Gugerty, B., Wooldridge, P., & Brennan, M. The CISQ: a tool to measure staff involvement in and attitudes toward the implementation of a clinical information system. Proc AMIA Symp 2000;: 320-324. • Brennan, MM. Team Performance Profile administration and analysis. Unpublished Master of Science Project, State University of New York at Buffalo, 1997. • The expansion of a tool to measure staff involvement in and attitudes toward the implementation of a clinical information system. Proc AMIA Symp.: p. 989, 2001. • Gugerty, B., Wooldridge, P., Coleman, S., Ziemba, J., Pierce, J. and Wenkosky, A.. Developmnet of the CISQ-MA to assess nursing attitudes towards electronic medication administration modules of healthcare information systems. Proc AMIA Symp.: p. 1034, 2002. • Gugerty, B., Woodridge, P. Maranda, M.J. & Sowan, A. “The Clinical Information System Questionnaire’s use in summative and formative evaluation” Presented at for the Eastern Evaluation Research Society's 27th annual conference, April 2004. • Bhola, H. S. Evaluating "Literacy for development" projects, programs and campaigns: Evaluation planning, design and implementation, and utilization of evaluation results. Hamburg, Germany: UNESCO Institute for Education; DSE [German Foundation for International Development]. xii, 306 pages. 1990. (C) B.Gugerty & M. Maranda, 2005

24. Address for correspondence • Brian Gugerty DNS, MS, RN Siemens Medical Systems Health Services Corporation 51 Valley Stream Parkway, A17 Malvern, PA, 19355 USA Email: brian.gugerty@siemens.com (C) B.Gugerty & M. Maranda, 2005