Measuring Your Impact: Using Evaluation to Demonstrate Value Phoenix, Arizona May 19, 2006 Medical Library Associati

1. Measuring Your Impact: Using Evaluation to Demonstrate Value Phoenix, Arizona May 19, 2006 Medical Library Association

2. Instructors Betsy Kelly Evaluation and Assessment Coordinator, NN/LM MCR Associate Director, Becker Library, Washington University kellyb@msnotes.wustl.edu and Maryanne Blake Evaluation & Outreach Coordinator, NN/LM PNR blakema@u.washington.edu

3. Workshop objectives • By the end of the workshop participants will: • Understand the library’s value in terms of the mission of the larger organization • Describe the tools used to assess the library, its users and stakeholders

4. Workshop objectives Identify the tools and methods used for data collection and analysis Design an evaluation plan for a service in a library Communicate evaluation results

5. Why demonstrate the library’s value? • To show the impact of the library on the larger organization’s mission and goals • To show accountability for your resources; the library’s contribution to the bottom line • To demonstrate that the library provides value-added services to solve staff problems • To use as an advocacy and marketing tool

6. I can’t believe I have to evaluate the whole thing! • Use “smallball” evaluation • What you decide to evaluate depends on: • What you need or want to know • What your users feel is important • What certain stakeholders want to have evaluated Environment

7. Yesterday’s Librarian “That was one of the delightful things about her job. It brought her in touch with so many people, all—or almost all—of whom made her not only feel welcome but that she was doing something really worthwhile. She gave a little sigh of pleasure as she unlocked the library door. She loved working here and she loved the work itself.” From: Jan Marlowe, Hospital Librarian. Margaret Malcolm. Toronto; New York : Harlequin, 1976, (c)1960. Harlequin's collection ed.

8. Today’s librarians… Must be: Proactive Political Practical Business–like Must reinvent librarianship

9. Reinventing librarianship T. Scott Pluchak, JMLA: July 2004 Editorial • Align the library’s priorities and services with the vision, mission and values of the larger organization to show its value … • Remember, the library is a tool; not an end in itself • It will not be supported simply because it has intrinsic value

10. Get started– get out of the library! Talk to people Talk about information problems they are trying to solve Show how the library can help them

11. Solving problems = value-added For hospital libraries • Medical Staff Support • Evidence-based clinical decision-making • Marketing • Research • Patient/family education and support • Community support • Legal/Risk Management • Human Resources Support • Staff education and development • Planning & Program Development • Evidence-based management support (competitive intelligence)

12. Solving problems = value-addedFor academic libraries • Educational Support • Resources for students and faculty • Liaisons to academic programs • Evidence-based clinical decision-making • Accreditation • Outreach • Patients/families • Greater community • Human Resources Support • Staff education and development

13. The bottom lineWill Welton • You are either generating revenue, or supporting those who doOR… • You are helping to control operating expenses or supporting those who doOR … • You are creating expenses that add recognized value OR… • You are creating expenses that must be controlled or eliminated to reduce overhead

14. So, ask yourself“What am I doing to support the bottom line?”And remember…

15. “There are no sacred departments ... Any function can be contracted to other professional agencies that want the business. For a department director an important question to consider is: What would we do differently if we had to compete with outside agencies to keep the internal client we have come to take for granted?” If Disney Ran Your Hospital: 9 ½ Things You Would Do Differently. Fred Lee, Bozeman, MT: Second River Healthcare Press, 2004.

16. A Plan To Show Your Value IsLike Building A House

17. The vision The environment The blueprint The tools And then… Build it The housewarming Before you dig…

18. The Vision

19. The vision - know what you want to build and why • Align library mission and goals with organization mission and goals • Library’s value = library’s contribution to achieving organization mission and goals Vision

20. Mission Concept Clinical care Education Management of operations Service Organizational Goal Provide excellent clinical care Provide services, resources needed for teaching and learning Reduce corporate risk Increase profitability Improve the lives of patients and their families Vision Abels EG, Cogdill KW, Zach L. Identifying and communicating the contributions of library and information services in hospitals and academic health sciences centers J Med Libr Assoc. 2004 Jan;92(1):46-55.]

21. The Environment

22. The environment • The organization • Your library • Clients/users • Stakeholders • The community Environment

23. Why look at the environment? • Understand needs, desires and problems in context • Validate assumptions about your contributions and services • Provide a baseline for future evaluation • Help to develop the blueprint to plan and evaluate your contributions and services Environment

24. SWOT analysis Strengths • Internal • Positive statements about your library Weaknesses • Internal • Statements about what is lacking in your library Opportunities • External • What do clients want you to do that you are not doing? Threats • External • Factors that can adversely impact your library’s goals Environment

25. Assessing the environment • User/stakeholder input • Surveys • Focus groups • Interviews • Unsolicited Feedback • Observation • Assessment by walking around • Library statistics and records Environment

26. Break time for Gobi (and for you too!)

27. What You Need Is A Blueprint!

28. The blueprint What you will build comes from analyzing: • What do you need? (Goal) • Comfort and convenience • What would meet your need? (Outcome) • A renovation of existing space? • An addition? • A new house? Blueprint

29. The blueprint • What actions must be taken? (Activities) • How much of what must be produced? (Outputs) • What resources will be required? (Resources) Blueprint

30. Plan backward;Implement forward Goals Outcomes Activities and outputs Resources Blueprint

31. Goals • Statement(s) of Purpose – why we do what we do and for whom • Examples: • Patients and their families will have improved health information literacy • Staff will have information for timely clinical decision-making • Acquisition of library materials will be cost effective Blueprint

32. Outcomes • Goal-related outcomes should be SMART • Specific • Measurable • Action-oriented • Realistic • Timed • Outcomes may be Short, Medium and/or Long Term Blueprint

33. Exercise model Goal: Nurses will have access to health information • Intermediate Outcome: Health resources web pages are available on the library website by the end of the program • Long Term Outcome: Nurses access health information through the library website • S: one or more web pages on the library website • M: done or not; do nurses access resources • A: created and posted; nurses access resources • R: it is possible within the timeframe • T: “when” is articulated Blueprint

34. Exercise – you do it now Goal: The library will cost effectively acquire and manage information resources Outcome: What outcomes do you expect? Blueprint

35. Activities and outputs • Activities • What will you do? • Outputs • How many did you do? • How many attended? • How many were distributed? • How many times was it used? Blueprint

36. Resources • What you have • Income • Equipment • Collection • Etc. • What you need • Operating expenses (e.g., personnel, acquisitions, maintenance, etc.) • Funds for new initiatives or services • Etc. Blueprint

37. The logic model • A tool for organizing your thoughts – both before and during a project/program • Framework for planning and evaluating programs • Makes evaluation easier • Good source for more information: W.K. Kellogg Foundation Logic Model Development Guideat http://www.wkkf.org/DesktopModules/WKF_DmaItem/ ViewDoc.aspx?LanguageID=0&CID=284&ListID=28&ItemID=2813669&fld=PDFFe Blueprint

38. Pieces of the logic model • Resources • What will you need to carry out your activities? • Activities • What will you do? • Outputs • How many of what will your activities produce? • Outcomes • So what – the difference your program makes, the benefits that accrue because of your program Blueprint

39. Sample logic model Goal: Cost effectively acquire and manage information resources

40. Logic model group activity • Chose a service to evaluate • Articulate the goal • Identify outcome(s) • List • activities • resources • outputs Blueprint

41. Logic model worksheet Goal:_________________________________________________

42. Logic models may change over time Blueprint

43. The blueprint part II:The evaluation plan • Builds on the logic model • Consider the purpose • Who is your audience? • Your users • Administrators • Fund raisers • How will the information be used? • Financial savings or justification • Intangible or non-monetary value of program benefits to community • Marketing and advocating for the library Blueprint

44. The evaluation plan - continued • Questions • What do you want to know about your program? • Indicators • How will you know you have achieved the outcomes? • Data • Sources • Methods • Frequency • Resources • Expertise or tools needed to collect and analyze data Blueprint

45. Evaluation plan Outcome (from logic model):________________________________________

46. Evaluation plan Outcome: Reduce institutional costs by buying shared resources. Blueprint

47. Evaluation plan II Outcome: Reduce hospital costs by buying shared resources. Blueprint

48. Design Your Own Evaluation Plan Blueprint

49. Evaluation plan group activity • Select an outcome from your logic model and list: • An activity from your logic model that will help achieve the outcome • Audience(s) affected by the activities • Indicators of outcome results • Targets for indicators • Data source • Data collection frequency • Data analysis methods Blueprint

50. Evaluation plan worksheet Outcome (from Logic Model):_______________________