Service Quality Assessment in a Digital Library Environment

1.  Service Quality Assessment in a Digital Library Environment EDUCAUSE 2002 Atlanta, Georgia October 3, 2002 Duane Webster

2. Overview of Discussions • New Models for Understanding and Describing Library Success • ARL’s New Measures Initiative • LibQUAL+ Project Development • Experience with LibQUAL+ • Conclusions and Next Steps

3. The Association of Research Libraries Mission: Shaping and influencing forces affecting the future of research libraries in the process of scholarly communication. Members: 123 major research libraries in North America. Ratios: 4% of the higher education institutions providing 40% of the information resources. Users: 3 million students and faculty served. Expenditures: $2.35 billion annually, $727 million for acquisitions of which 9% is invested in access to electronic resources. www.arl.org ASSOCIATIONOF RESEARCH LIBRARIES

4. ARL New Measures Initiative • Collaboration among member leaders with strong interest in this area • Specific projects developed with different models for exploration • Intent to make resulting tools and methodologies available to full membership and wider community

5. LibQUAL+™ Description • LibQUAL+TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality-assessment tools for local planning.

6. Project Resources • LibQUAL+TM is an ARL/Texas A&M University joint effort. The project is supported in part by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) and a 3-year grant from the National Science Foundation (NSF)

7. The Imperativefor our Research In an age of accountability, there is a pressing need for an effective and practical process to evaluate and compare research libraries. In the aggregate, among the 122 Association of Research Libraries (ARL) alone, over $2.8 billion dollars were expended in 1999/2000 to satisfy the library and information needs of the research constituencies in North America (Kyrillidou & Young, 2001, p. 5).

8. The Problem of Assessment in Research Libraries • The lack of metrics • ARL Membership Criteria Index variables emphasize inputs, primarily expenditures • No demonstrable relationship between expenditures and service quality • To rise in the ARL Index it is only necessary to spend more

9. Assessment “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” Sarah Pritchard, Library Trends, 1996

10. LibQUAL+™ Project Goals • Establishment of a library service quality assessment program at ARL • Development of web-based tools for assessing library service quality • Development of mechanisms and protocols for evaluating libraries • Identification of best practices in providing library service

11. LibQUAL+TM Participants Year 3 Year 2 164 Participants 43 Participants Year 1 12 Participants Spring 2000 Spring 2001 Spring 2002 For More Information about Participants: Visit the LibQUAL+ web site.

12. Relationships: perceptions, service quality and satisfaction ….only customers judge quality; all other judgments are essentially irrelevant” Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

13. Source: Parasuraman, ARL Symposium on Measuring Service Quality Washington, DC, October 2000

14. 70+ Interviews conducted  • York University • University of Arizona • University of Connecticut • University of Houston • University of Kansas • University of Minnesota • University of Pennsylvania • University of Washington • Smithsonian • Northwestern Medical

15. Atlas TI Dimensions of Analysis

16. Self-reliance “By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” Graduate student

17. Self-reliance “…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.” Faculty member

18. Affect of Service “I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” Faculty member

19. Comprehensive Collections “I think one of the things I love about academic life in the United States is that as a culture…, we tend to appreciate the extraordinary importance of libraries in the life of the mind.” Faculty member

20. Comprehensive Collections “I sense that if I were in an institution that didn’t have the rich collections as this library and the very effective staff members that this library has that I would imperceptibly slip in my discipline….” Faculty member

21. Ubiquity of Access “Over time my own library use has become increasingly electronic. So that the amount of time I actually spend in the library is getting smaller and the amount of time I spend at my desk on the web … is increasing.” Faculty member

22. Reliability “You put a search on a book and it’s just gone; it’s not reacquired. … There’s more of a problem of lost books, of books that are gone and nobody knows why and nobody’s doing anything about it.” Faculty member

23. Reliability “I put something on reserve. And it didn’t show up, and somebody complained. I went back and said ‘I’ve asked for this to be put on reserve’ and they had lost the form. So I had to do it again.” Faculty member

24. Library as Place “I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.” Faculty member

25. Library as Place “The poorer your situation, the more you need the public spaces to work in. When I was an undergraduate, I spent most of my time in the library, just using it as a study space.” Faculty member

26. Library as Place “One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.” Faculty member

27. Dimensions of Library Service Quality

28. Affect of Service • Emerged as the dominant factor early in our work • Absorbed several of the original SERVQUAL questions measuring Responsiveness, Assurance and Empathy • In the current analysis also includes Reliability • All in all: the Human Dimension of Service Quality

29. Library as Place • Transcends the SERVQUAL dimension of Tangibles to include the idea of the library as the campus center of intellectual activity • As long as physical facilities are adequate, library as place may not be an issue

30. Personal Control • How users want to interact with the modern library • Personal control of the information universe in general and web navigation in particular

31. Access to Information • Ubiquity of access: information delivered in the format, location and time of choice • Comprehensive collections

32. LibQUAL+ 2002 Iteration • 42 — ARL Libraries • 35 — Health Sciences Libraries • 36 — State Colleges & Universities (excluding ARL) • 34 — Private Colleges & Universities (excluding ARL) • 15 — Community Colleges • 2 — Special & Public Libraries (Smithsonian & NYPL)

33. The Challenge of Analysis • There are few, if any useful conclusions to be drawn from aggregate data of all institutions, because their missions and subsequent user expectations for service are too diverse. • There are commonalities in service delivery profiles that merit further investigation. • In the long run, information that may be derived from demographic responses of individuals may yield the richest data.

34. Two Interpretation Frameworks • Score Norms • Zone of Tolerance

35. Zone of Tolerance • The area between minimally acceptable and desired service quality ratings • Perception ratings ideally fall within the Zone of Tolerance

36. Aggregate Dimension Summary (n=70,445) Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 1, p. 24

37. Faculty Item Summary Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 2, p. 40

38. Score Norms • Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample. • LibQUAL+TM norms have been created at both the individual and institutional level

39. Overall Mean Scores and Service Adequacy Gap Scores By Cohort Group 2002 LibQUAL+ Iteration (n=162) Community Colleges 7.26 (.55) Private Colleges 6.90 (.49) ARL Top 40 6.84 (.46) AAHSL 7.07 (.56) State Colleges & Universities 6.38 (.30) ARL Other 6.74 (.27)

40. LibQUAL+™ Fundamental Contributions to the Measurement of Effective Delivery of Library Services • Shift the focus of assessment from mechanical expenditure-driven metrics to user-centered measures of quality • Re-ground gap theory for the library sector, especially academic libraries • Grounded questions yield data of sufficient granularity to be of value at the local level • Determine the degree to which information derived from local data can be generalized, providing much needed “best practices” information • Demonstrate the efficacy of large-scale administration of user-centered assessment transparently across the web • Makes little demand of local resources and expertise

41. Recognize the limitations of listening to customers • Customers have a limited frame of reference and tend to offer incremental, rather than bold, suggestions • A better slide rule • The microwave oven, Post-it Notes, Velcro • Innovation is the responsibility of staff Anthony W. Ulwick, Harvard Business Review, January 2002

42. Shift the focus to outcomes • Plan outcome-based customer interviews • Capture desired outcomes • Organize the outcomes • Rate the outcomes for importance and satisfaction • Opportunity algorithm: (Importance+(Importance-Satisfaction)=Opportunity) • Use the outcomes to jump-start innovation Anthony W. Ulwick, Harvard Business Review, January 2002

43. When desired outcomes become the focus of customer research, innovation becomes a manageable, predictable discipline. Anthony W. Ulwick, Harvard Business Review, January 2002

44. Summary • Survey can handle large numbers • Survey can be turned around quickly • Limited local expertise required • Interpretations should be across chosen cohorts • Lots of opportunities for using demographics to discern user behaviors • Q-technique and other tests will provide opportunities to observe how institutions may cluster

45. LibQUAL+ Related Documents • LibQUAL+Web Site http://www.arl.org/libqual/ • LibQUAL+Bibliography • http://www.coe.tamu.edu/~bthompson/servqbib • Survey Participants Procedures Manual • http://www.arl.org/libqual/procedure/lqmanual2.pdf

46. NDSL LibQUAL+ Activities • 120-200 qualitative interviews to contribute to identifying dimensions of digital library service quality • Test and refine dimensions of digital library service quality and self-sufficiency through development of total market survey • Implement survey across variety of organizational and digital library implementations

47. NDSL LibQUAL+ Goals • Define dimensions of digital library service quality from the users’ perspectives • Develop tool for measuring user perceptions and expectations of digital library service quality across NSDL digital library contexts • Identify digital library best practices that permit generalizations across operations and development platforms

48. NSF Grant • Assess service quality in digital libraries • 3 year period • Adopt LibQUAL+ instrument for use in the Science, Math, Engineering and Technology Education Digital Library community (NSDL)

49. LibQUAL+TM Project History Jan2000 Sept. 1999 Sept. 2000 July 2000 Oct. 2000 Jan. 2001 June 2001 9/99 - ARL launches “New Measures Initiative” which includes the study of service effectiveness known as SERVQUAL spearheaded by Texas A&M University. 1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team. 7/00 - LibQUAL+ as a distinct library-based assessment tool is presented. 9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the LibQUAL+ project. 10/00 - The ARL symposium, “New Culture of Assessment in Academic Libraries Measuring Service Quality” attracts a group of 170 people. 1/01 - Representatives from 43 research and university libraries participating in the Spring 2001 Implementation meet in Washington, DC during ALA midwinter. 6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+ for NSDL

50. LibQUAL+TM Team • ARL • Duane Webster • Martha Kyrillidou • Kaylyn Hipps • Julia Blixrud • Jonathan Sousa • Consuella Waller • TAMU • Fred Heath • Colleen Cook • Bruce Thompson • Yvonna Lincoln • Trey Thompson • Julie Guidry