LibQUAL+ ® & Beyond: Applying Your Survey Results & Other Performance Measures in Library Practice

1. LibQUAL+® & Beyond: Applying Your Survey Results & Other Performance Measures in Library Practice LibQUAL+® Canada Workshop October 24-25, 2007 Ottawa, Ontario, Canada Martha Kyrillidou, Director, Statistics and Service Quality Programs, ARL Steve Hiller, Director, Assessment and Planning, UW Jim Self, Director, Management and Information Services, UVA

2. Martha Kyrillidou, Director Statistics and Service Quality Programs Association of Research Libraries

3. What’s in a “Library”? A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. --Justice Oliver Wendell Holmes

4. What’s in a word? What makes a qualitylibrary? “Quality much like beauty is in the eye of the beholder”

5. Library Assessment and its Global Dimensions • Markets and people exposed to economic and social frameworks unheard of before • Competing internationally • Library users exposed to global forces • Libraries facing similar challenges • Libraries as the Internet • Libraries as Google • Libraries as Collaborative Spaces

6. Library Assessment Library assessment provides a structured process to learn about our communities, their work and the libraries connection to what they do The information acquired through library assessment is used in an iterative manner to improve library programs and services and make our libraries responsive to the needs of our communities. Academic libraries do not exist in a vacuum but are part of a larger institution. Assessment within the institution may take place in individual areas as well as at the broad institutional level.

7. Thinking Strategically About Library Futures: Some Assessment-Related Questions • What is the central work of the library and how can we do more, differently, and at less cost? • What important services does the library provide that others can’t? • What advantages does the research library possess? • How is customer behavior changing? • How do we add value to our customers work? • What are the essential factors responsible for library success now and in the future?

8. Free speech wall, Charlottesville, Sept 2006

9. IFLA: Measuring Quality • Resources, infrastructure: What services does the library offer? • Use: How are the services accepted? • Efficiency: Are the services offered cost-effectively? • Potentials and Development: Are there sufficient potentials for future development?

10. Assessment at ARL • A gateway to assessment tools: StatsQUAL®: • ARL Statistics -- E-Metrics • LibQUAL+® • DigiQUAL® • MINES for Libraries® • Library Assessment Conferences • Service Quality Evaluation Academy • Library Assessment Blog • Making Library Assessment Work • ESP Assessment • Effective, Sustainable, Practical

11. Assessment at CARL

12. Assessment at SCONUL

13. Assessment at CAUL

14. Assessing the Value of Networked Electronic Services The MINES survey Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries®

15. What Are We Measuring? Reviewing the ARL Statistics October 2005, ARL Board approved a study to: • Determine if there are new ways of describing research library collections. • What is it we are currently measuring • Are they the right data • Develop alternative models • Develop a profile of the characteristics of a contemporary research library • Determine/develop new meaningful measures to augment current ones to support this profile

16. Quantitative Stats(Per Bruce Thompson) • Expenditure Focused Index (EFI) • Current ARL stats that could be used for benchmarking • Collections • User interactions • # Participants in group presentations • # Presentations to library groups • # Reference transactions • Collaborative Activities - Interlibrary loan activities • Borrowed total items • Loaned total items • Set of statistics related to the digital library (from ARL supplementary statistics)

17. Qualitative ProfileDeveloping New Metrics (per Yvonna Lincoln) • Uniqueness of collections • Defining the value of consortia • Administrative and budgetary efficiencies • Student outcomes/student learning/graduate success • Contributions to faculty productivity • Social frameworks/intellectual networks • Generating new knowledge • Creating the collective good with reusable assets

18. What Makes a Research Library? • Breadth and quality of collections and services • Sustained institutional commitment to the library • Distinctive resources in a variety of media • Services to the scholarly community • Preservation of research resources • Contributions of staff to the profession • Effective and innovative use of technology • Engagement of the library in academic planning Association of Research Libraries ‘Principles of Membership’

19. Group discussion • How do you go about developing a profile that is succinct and rich? • Other important areas that should be part of a qualitative profile? • Can LibQUAL+® be used in the profiles?

20. Library of the Future Will Also Need . . . . . . To have it’s own data collection and management personnel, individuals who constantly collect, analyze and prepare reports on data regarding what services are being used, which portions of the collection are getting the highest usage, what materials are being lent through interlibrary loan, and who patrons are. Documenting the libraries contributions to quality teaching, student outcomes, research productivity will become critical.

21. Making Library Assessment Work • ARL project approved in 2004 • Funded by participating libraries • Site visits by Steve and Jim • Presentation • Interviews and meetings • Report to the Library • 24 libraries in U.S. and Canada visited in 2005-06 • Succeeded by Effective, Sustainable and Practical Library Assessment in 2007 • Open to all libraries • 6 libraries participating in 2007

22. What We Found • Strong interest in using assessment to improve customer service and demonstrate value of library • Many libraries uncertain on how to establish, maintain, and sustain effective assessment • Effectiveness of assessment program not dependent on library size or budget • Each library has a unique culture and mission. No “one size fits all” approach works. • Strong customer-focus and leadership support were keys to developing an effective and sustainable assessment

23. What are the lessons learned? • Understanding changes in users approach to information resources. • Service quality improvement is a key factor. • Understanding the impact of e-resources on library services - TRL. • Learning how to compete with Google. • Upfront investment in design and development. • Making the assessment service affordable, practical,&effective. • Assessment needs to be satisfying and fun.

24. User Needs Assessment and Academic Library Performance Steve Hiller Director Assessment and Planning University of Washington Libraries

25. An “Aha” Moment “[Access to online resources] has changed the way I do library research. It used to be a stage process: Initial trip, follow-up trip, fine-tuning trip. Now it’s a continuous interactive thing. I can follow-up anything at any time. While I’m writing I can keep going back and looking up items or verifying information.” Graduate Student, Psychology (2002 UW Libraries focus group)

26. What Do We Need to Know About Our Customers? • Who are our customers (and potential customers)? • What are their teaching, learning, and research interests? • How do they work? What’s important to them? • How do they find information needed for their work? • How do they use library services? What would they change? • How do they differ from each other in library use/needs? How does the library add value to their work? How does the library contribute to their success?

27. How Do We Get Customer Information? • Surveys • Usage statistics • Focus groups • Observation • Usability • Interviews • Embedding • Data mining (local, institutional) • Logged activities

28. University of Washington(Site of the 2008 Library Assessment Conference!) • Located in beautiful Seattle metro population 3.2 million • Comprehensive public research university • 27,000 undergraduate students • 12,000 graduate and professional students (80 doctoral programs) • 4,000 research and teaching faculty • $800 million annually in federal research funds (2nd in U.S.) • Large research library system • $40 million annual budget • 150 librarians on 3 campuses

29. UW Libraries Assessment PrioritiesCustomer Needs, Use and Success • Information seeking behavior and use • Patterns of library use • Value of library • User needs • Library contribution to customer success • User satisfaction with services, collections, overall • Data to make informed and wise decisions that lead to resources and services that contribute to user success

30. UW Libraries:Assessment Methods Used • Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007 • All faculty • Samples of undergraduate and graduate students • Research scientists, Health Sciences fellow/residents 2004- • In-library use surveys every 3 years beginning 1993 • LibQUAL+™ from 2000-2003 • Focus groups/Interviews (annually since 1998) • Observation (guided and non-obtrusive) • Usability • Use statistics/data mining Information about assessment program available at: http://www.lib.washington.edu/assessment/

31. Our Latest Assessment Method

32. The Qualitative Provides the Key • Increasing use of such qualitative methods as, comments interviews, focus groups, usability, observation • Statistics/quantitative data often can’t tell us • Who, how, why • Value, impact, outcomes • Qualitative provides information directly from users • Their language • Their issues • Their work • Qualitative provides understanding

33. Researchers and Libraries:3 Recent Studies with Qualitative Focus • University of Minnesota • Extremely comfortable with electronic sources • Interdisciplinary critical in sciences • Inadequate methods for organizing research materials • New York University • Researchers (all disciplines) no longer tied to physical library • Physical library can play a “community” role • Expectations for info shaped by Web and commercial sector • University of Washington (Biosciences) • Start info search outside library space (virtual and physical) • All digital all the time • Could not come up with “new library services” unprompted

34. Reasons for UW Libraries Biosciences Review • Better understand how bioscientists work • Growing inter/multi/trans disciplinary work • Significant change in use patterns • Libraries responsiveness to these changes • Value of research enterprise to the University • Strengthening library connection to research Ensuring our services and resources support the work of the biosciences community

35. Biosciences Review Process (2006) • Define scope (e.g. what is “bioscience”?) • Identify and mine existing data sources • Extensive library assessment data • Institutional and external data • Acquire new information through a customer-centered qualitative approach • Environmental scan • Interviews • Focus groups • Peer library surveys NO NEW USER SURVEYS

36. Biosciences Faculty Interview Themes • Library seen primarily as E-Journal provider • Physical library used only for items not available online • Start information search with Google and PubMed • Too busy for training, instruction, workshops • Faculty who teach undergrads use libraries differently • Could not come up with “new library services” unprompted

37. Biosciences Focus Group Themes • Content is primary link to the library • Identify library with ejournals; want more titles & backfiles • Provide library-related services and resources in our space not yours • Discovery begins primarily outside of library space with Google and Pub Med; Web of Science also important • Library services/tools seen as overly complex and fragmented • Print is dead, really dead • If not online want digital delivery/too many libraries • Go to physical library only as last resort • Data and reference management important to some • Bioresearcher toolkit, EndNote, JabRef, StatA

38. Biosciences Task Force Recommendations • Integrate search/discovery tools into users workflow • Expand/improve information/service delivery options • Make physical libraries more inviting/easier to use • Consolidate libraries, collections and service points • Reduce print holdings; focus on services • Use an integrated approach to collection allocations • Get librarians to work outside library space • Lead/partner in scholarly communications & E-science • Provide more targeted communication and marketing

39. Biosciences Review Follow-up : 2007 Actions • Appointed a Director, Cyberinfrastructure Initiatives & Special Asst to the Univ Libr for Biosciences & E-Science • Libraries Strategic Plan priorities for 2007 include: • Improve discovery to delivery (WorldCat Local etc.) • Reshape our physical facilities as discovery and learning centers • Strengthen existing delivery services, both physical and digital, while developing new, more rapid delivery services • Enhance and strengthen the Libraries support for UW’s scientific research infrastructure • Do market research before developing & promoting services • Informed development of Libraries 2007 Triennial Survey

40. In God We Trust: All Others Must Bring Data UW Triennial Survey 2007 – Selected Questions Mode of access/physical library uses and users Resource type importance Sources consulted for research Primary reasons for using Libraries Web sites Information literacy Libraries contribution to work and academic success Useful library services (new and/or expanded) Satisfaction

41. UW Triennial Library Survey Number of Respondents and Response Rate 1992-2007

42. I only wish I could reproduce the graduate reading room in my home because I do so much of my reading/research online now. Oh well, at least I can be in my slippers. Associate Professor, Psychology

45. Physical Library Users by Group forSelected Libraries (2005 In-Library Use Survey) Undergrads 70%, Grads 25%, Faculty/Staff 5%

46. Physical Library Use by Academic Area(2005 In-Library Use Survey)

49. Off-Campus Remote Use1998-2007(Percentage using library services/collections at least 2xweek)

50. Importance of Books, Journals, DatabasesAcademic Area (2007, Faculty, Scale of 1 “not important” to 5 “very important)