Assessment, Evaluation, and Value: Current Trends.

1. Assessment, Evaluation, and Value: Current Trends. Rosalind F. Dudden, MLS, DM/AHIP, FMLA Library Services Director Tucker Medical Library National Jewish Medical and Research Center Denver, Colorado 80206

2. Outline • Background • Trends in Assessment • Trends in Evaluation • Outcomes Measurements • Reporting Outcomes • Value concepts C E N D I Principles Meeting

3. “When I read about the way in which library funds are being cut and cut, I can only think that American society has found one more way to destroy itself.” • Isaac Asimov (Russian born American science-fiction Writer and Biochemist. 1920-1992) C E N D I Principles Meeting

4. Response to Library Cuts • National Library of Medicine Publications Grant, now called NLM Grants for Scholarly Works in Biomedicine and Health • (#5-G13LM008520) Oct 2004 to Sept 2006 • Purpose: a book-length manuscript (or other scholarly work) of value to U.S. health professionals. • A good score - hope that other people thought the book would be of use to librarians in small library settings. C E N D I Principles Meeting

5. Response to Library Cuts Table of Contents Part I: Evaluating Library Quality and Performance 1. Why Evaluate? 2. The Effective Library 3. Library Measures New York: Neal Sculman, July 2007. C E N D I Principles Meeting

6. Response to Library Cuts • Part II:Working with Evaluation Methods 4. Method 1: Needs Assessment Workbook for Needs Assessment 5. Method 2: Quality Improvement Workbook for FOCUS-PDCA 6. Method 3: Benchmarking Workbook for Performance Benchmarking Workbook for Process Benchmarking 7. Method 4: Library Performance Standards Workbook for Library Performance Standards Workbook for Accreditation Standards C E N D I Principles Meeting

7. Response to Library Cuts 8. Method 5: Outcomes Measurement Workbook for Describing Published Studies of Outcomes Measurement Workbook for Cost Outcomes Workbook for the Logic Model 9. Other Systems for Quality Improvement and Evaluation Part III: Tools for Doing Evaluations 10. Data Collection and Analysis Methods 11. Skills for Communicating in Evaluation Projects 12. Tools for Improvement and Evaluation Appendix A: Glossary of Terms Appendix B: Recommended Reading C E N D I Principles Meeting

8. Straight Path of Evaluation C E N D I Principles Meeting

9. But Soon… Cover - The New Republic - 2/19/2007 C E N D I Principles Meeting

10. Background • It started with Lancaster • Lancaster FW. The Measurement and Evaluation of Library Services. 1977. 1991 (with Baker SL). • Lancaster FW. If You Want to Evaluate Your Library. 1988, 1993 C E N D I Principles Meeting

11. Background - Process • Libraries would do a process evaluation • No one would notice except those working in the part of the library being evaluated. • Processes were improved, customer needs met, the head librarian managed. • While not rich, libraries in general were supported. • Libraries were considered “essentially good.” • Why evaluate something that is going to exist anyway because of its basic value to society or the corporation? C E N D I Principles Meeting

12. Background - Comparative • To assure or perhaps to prove quality, library associations renewed their interest in writing library standards and guidelines. • At the same time ARL and AAHLS started collecting comparative data on library operations and still do today. • MLA followed recently by collecting data for non-academic health sciences libraries in their Benchmarking Network surveys. C E N D I Principles Meeting

13. Background - Technology • 80s - Technologically, libraries started feeling a paradigm shift from the printed word to other media. • 90s on - The impact of the Internet on library operations is unprecedented. C E N D I Principles Meeting

14. Background - Integration • The integration of all types of information, as described as early as 1982 in the Matheson report, is now being felt in many settings where libraries operate. • Information is being connected on a scale and with a speed often felt to be overwhelming. • Electronic health records • Industry-wide portals • Federated search engines C E N D I Principles Meeting

15. Background • Virginia Holtz, in her 1986 MLA Janet Doe lecture, “Measures of Excellence: the Search for the Gold Standard,” when the Internet was just a transmission device, stated what is still true today: “Typography of information is primitive. … Further study will continue to reveal its full structure and the relationships among the parts.” C E N D I Principles Meeting

16. Background What happened to just buying and cataloging a book? C E N D I Principles Meeting

17. 1900 Engineering linear development vertical integration 2000 Biology diffusion webs volatility adaptability Shifting Paradigms C E N D I Principles Meeting

18. Linear Arrangement of Types of Measures Dudden, RF. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To-Do-It Manual. New York: Neal Sculman, 2007. C E N D I Principles Meeting

19. Linear Arrangement of Types of Measures Dudden, RF. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To-Do-It Manual. New York: Neal Sculman, 2007. C E N D I Principles Meeting

20. User Behavior • Multidimensional nature of the typical graduate student. • A person who is struggling to complete his/her dissertation and uses information resources in a multiplicity of ways. • Martha Kyrillidou, Director, Statistics and Service Quality Programs, ARL C E N D I Principles Meeting

21. User Behavior • directly borrow library resources • talk to faculty members who have used the library themselves • borrow a book from his or her advisor • find another interesting resource while visiting a fellow-students' home • communicate via e-mail with other experts who refer him or her to additional resources and citations C E N D I Principles Meeting

22. User Behavior • browse the Internet and locate blogs and wikis in his or her specific field of interest • send reference inquiries to ask-a-service operations • talk with a fellow student whose spouse works in the library and discuss with him or her the articles he or she was able to locate in the library • - - with the help of a librarian C E N D I Principles Meeting

23. Inputs to Outputs to Outcomes “This inextricable and complex web of relations provides a richer context for the usefulness of inputs, outputs, quality issues, and the impact libraries are making.” • Martha Kyrillidou. From input and output measures to quality and outcome measures, or, from the user in the life of the library to the library in the life of the user. J Acad Libr. 2002;28(1/2):42-6. C E N D I Principles Meeting

24. “User in the life of the library.. • “The usual linear model (Figure 2) depicting a linear, sequential, rational, and controlled set of relations • is indeed limiting our understanding of a model (Figure 3) that is more cyclical, haphazard, and uncontrolled.” C E N D I Principles Meeting

25. “the library in the life of the user… • “This becomes even more complex when one tries to introduce the notion of motion (Figure 4) depicting a more dynamic and flexible model, • moving users and information resources • into a spiral swirl up and down • into the depths of knowledge, exploration, and experience.” C E N D I Principles Meeting

26. What do we measure? • Needs • Inputs • Quality Processes • Outputs • Quality Service • Outcomes • Impacts C E N D I Principles Meeting

27. Assessment - Evaluations • Definitions: • “Assessment is the gathering of meaningful or purposeful data that will provide information that informs, improves, or confirms.” • “Evaluation is assigning merit, value or worth to the findings.” • Lee-Thomas G, Robson J. The questions of academic library assessment. Indiana Libr. 2004;23(1):6-10. C E N D I Principles Meeting

28. Measure Needs • Needs assessment • Customer focus • Perley CM, Gentry CA, Fleming AS, Sen KM. Conducting a user-centered information needs assessment: the Via Christi Libraries' experience. J Med Libr Assoc. Apr 2007;95(2):173-81, e54-5. C E N D I Principles Meeting

29. Needs Assessment Results • Clinicians emphasized the need for • “Just in time” information accessible at the point of care • Information services customized to their professional information needs, preferences, and patterns of use • Organization-specific information needs identified • Library nonusers emphasized the need to market library services and resources. C E N D I Principles Meeting

30. Measure Inputs • What is the size of our staff, budget, space, or collection? • How do we compare to others in benchmarking studies? • Often criticized, but comparisons are sometimes it is the only thing stakeholders understand. • The comparisons answer questions people ask because they do not know what else to ask. C E N D I Principles Meeting

31. Measure Inputs • A casual conversation: • “How many e-journals do you have?” • Answer 1: “350 (or whatever) AND did you know that the process of investigation by our scientists has been fast-tracked by their access to this information at their desktop?” • Answer 2: “1500 (or whatever) AND did you know that the use of these resources by our high school students at home has increased their ability to have more resources cited in their papers?” C E N D I Principles Meeting

32. Measure Inputs • You still need those traditional input counts. • Good management • Budgeting and resource allocation • Comparative Benchmarking C E N D I Principles Meeting

33. Measure Quality Processes • TQM (Total Quality Management) - now part of any management culture. • Quality improvement techniques: • PDCA - Plan-Do-Check-Act • Gemba Visits • Kaizen • Lean, Five Ss • Learning Organization • Sigma Six • Balanced Scorecard C E N D I Principles Meeting

34. Measure Quality Processes • David Orenstein • A quality culture “considers service issues, people, and challenges as simultaneous and interconnected concerns.” • Eleven ways to move your library to adopt a quality culture - • Orenstein DI. Developing quality managers and quality management: the challenge to leadership in library organizations. Libr Adm Manag. 1999;13(1):44-51. C E N D I Principles Meeting

35. Quality Culture • Build a shared vision for the library. • Put the needs of the customers before the politics of the organization. • Build cooperation among all levels of employees. • Communicate. • Emphasize teamwork. C E N D I Principles Meeting

36. Quality Culture • Build trust. • Redesign processes and attitudes. • Train for quality. • Develop leadership skills. • Manage by fact. – (EBLIP4.unc.edu) • Motivate staff by making work enjoyable. C E N D I Principles Meeting

37. Measure Outputs • Answers the question - What do you do? • Use in communications to stakeholders • Identify trends. • Can indicate a re-assignment of staff. • A trend up or down may indicate the necessity of further evaluation of a program or service. C E N D I Principles Meeting

38. By the Numbers • 8,428,033 Items circulated • 417 Full-time equivalent staff members • 3,918,658 Visitors to the Central Library and 22 branch locations • 31,378,982 Transactions conducted online • 99,332 Hours donated by docents and volunteers • 939,393 Reference questions answered system-wide • 22,446 Kids / teens who registered for Summer of Reading (2005 Annual Report) C E N D I Principles Meeting

39. Measure Service Quality • LibQual+™ - from ARL - http://www.libqual.org/ • A suite of services that libraries use to solicit, track, understand, and act upon users’ opinions of service quality. • The program’s centerpiece is a rigorously tested Web-based survey - bundled with training that helps libraries assess and improve library services, change organizational culture, and market the library. C E N D I Principles Meeting

40. Measure Service Quality • LibQual+™ seeks to measure three dimensions of library service quality: • the library as place (utilitarian space, symbol, refuge); • information control (scope, timeliness, convenience, ease of navigation, modern equipment); and • affect of service (empathy, responsiveness, assurance, reliability). C E N D I Principles Meeting

41. Measure Service Quality • Non-ARL libraries • Simple satisfaction surveys • Rate service as to • Satisfaction 1 2 3 4 5 • Importance 1 2 3 4 5 • Used Web-based tools such as SurveyMonkey or Zoomerang C E N D I Principles Meeting

42. Measure Outcomes • Started in 1993 with the Government Performance and Results Act (GPRA) • The GPRA required that most federal agencies develop objective, quantifiable and measurable goals and report how well those goals were achieved. • The influence of the law spread to local governments and philanthropic organizations. • If a program says it will change the lives of a certain population group, how does it prove it actually did? C E N D I Principles Meeting

43. Measure Outcomes • United Way of America: • Measuring Program Outcomes: A Practical Approach, 1996. • The Outcome Measurement Resource Network • W.K. Kellogg Foundation: • W.K. Kellogg Foundation Evaluation Handbook, 1998; • Logic Model Development Guide. 2004. • The Evaluation Toolkit C E N D I Principles Meeting

44. Measure Outcomes • A fiscally conservative trend • “Prove” that the dollars spent helping people really do • Trickle down effect to all types of libraries, as well as museums, colleges, and universities C E N D I Principles Meeting

45. Outcomes - Public Libraries The Institute for Museum and Library Services (IMLS) describes an outcome as - a benefit to people through achievements or changes in skill, knowledge, attitude, behavior, condition, or life status. C E N D I Principles Meeting

46. Outcomes - Public Libraries • How libraries and librarians help: A Guide to Identifying User-Centered Outcomes, by Durrance and Fisher • “Political pressures are very strong for public libraries to prove their worth in competition for tax dollars.” • Information School, University of WashingtonSchool of Information, University of Michigan C E N D I Principles Meeting

47. Outcomes - Public Libraries • Outcomes Tool Kit 2.0 provides “guidance for going beyond reporting outputs and will help you to discover outcomes, or indicators of impact, of your programs that can be shared with others.” C E N D I Principles Meeting

48. Outcomes - Public Libraries • Shaping Outcomes • An outcome is a change in a target audience’s skills, attitudes, knowledge, behaviors, status, or life condition brought about by experiencing a program. • Indiana University Purdue University Indianapolis • http://www.shapingoutcomes.org/course/index.htm C E N D I Principles Meeting

49. Outcomes - Ice Cream Socials • Example: A local Denver dairy is awarding cash grants or ice cream socials to school libraries. • As part of the application, the library needs to: • Describe who will benefit from the grant. • What measure will you use to determine the success of the project? C E N D I Principles Meeting

50. Outcomes - Academic Libraries • Challenges • Is the academic performance of students improved through their contact with the library? • By using the library, do students improve their chances of having a successful career? • Are undergraduates who used the library more likely to succeed in graduate school? • Are students who use the library more likely to lead fuller and more satisfying lives? • ACRL Task Force on Academic Library Outcomes Assessment C E N D I Principles Meeting