Mary E. Jackson Senior Program Officer for Access Services Association of Research Libraries

1. Next Generation Statistics for Libraries: CAUL Forum20 January 2003Sydney, NSWThe ARL E-Metrics Project Mary E. Jackson Senior Program Officer for Access Services Association of Research Libraries mary@arl.org

2. ARL New Measures Initiative • Collaboration among member leaders with strong interests • Specific projects explore different models • Self-funded projects

3. ARL New Measures Initiative • Make resulting tools and methodologies available to full membership and wider community • Continue to collect, but limit/freeze modifications to existing descriptive measures

4. ARL New Measures Projects • Demonstration project for service effectiveness measures • LibQUAL+ • Identification of measures that demonstrate a library’s contribution to student learning outcomes • Investigation of role libraries play in support of the research process

5. ARL New Measures Projects • Development of tools to address cost effectiveness of library operations • staff allocation, Assessing ILL/DD Services study • Project to define usage measures for electronic information resources • E-metrics Project

6. Electronic Serials Expenditures

7. Percentages of Acquisitions Dollars Devoted to Electronic Resources

8. Average Yearly Increases in Electronic Resources and Total Library Materials Expenditures

9. Project Leaders • Co-Chairs • Rush Miller, University of Pittsburgh, and Sherrie Schmidt, Arizona State University • Information Use Management and Policy Institute, Florida State University • Charles R. McClure, Wonsik “Jeff” Shim, and John Carlo Bertot • ARL • Duane Webster and Martha Kyrillidou

10. Alberta Arizona State Auburn Chicago Connecticut Cornell Illinois - Chicago Library of Congress Manitoba Maryland Massachusetts Nebraska New York Public Notre Dame Pennsylvania Penn State Pittsburgh Purdue Southern California Texas A & M Virginia Tech Western Ontario Wisconsin Yale Project Participants

11. Vendor Statistics Working Group 12 major ARL vendors met with project team in Denver prior to 2000 ACRL Meeting Academic Press/IDEAL * Elsevier/Science Direct Lexis/Nexis Ovid Bell & Howell Gale Group ISI * † netLibrarySilver Platter *EBSCO JSTOROCLC/First Search * Unable to attend Denver Meeting † Nonparticipant in project.

12. Working Definition of Networked Services Electronic information resources and/or services that users access electronically via a computer network: • From on-site in the library • Remote to the library – but from a campus facility • Remote from the library & campus

13. Networked Information Resources • Locally Licensed Databases • Regional or Statewide Consortia Licensed Databases • Aggregated Databases • Publishers Databases • Publicly Available (Web) Resources

14. ARL E-Metrics Project: Phase 1 May-October 2000 What do we know? Inventory of current practices at ARL libraries of statistics, measures, processes, and activities that pertain to networked resources and services.

15. ARL E-Metrics Project: Phase 2 November 2000-June 2001 What can we collect? Identified and field tested an initial draft set of statistics and measures.

16. ARL E-Metrics Project: Phase 3 July 2001-December 2001 What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure.

17. Recommended Statistics & Measures • Patron Accessible Electronic Resources (R1-3) • Use of Networked Resources & Services (U1-5) • Expenditures for Networked Resources & Related Infrastructure (C1-3) • Library Digitization Activities (D1-3) • Performance Measures (P1-3)

18. Patron Accessible Electronic Resources • R1 – Number of electronic full-text journals • R2 – Number of electronic reference sources • R3 – Number of electronic books

19. Use of Networked Resources & Related Infrastructure • U1 – Number of electronic reference transactions • U2 – Number of logins (sessions) to electronic databases (db) • U3 – Number of queries (searches) in electronic db • U4 – Items requested in electronic db • U5 – Virtual visits to library’s website and catalog

20. Expenditures for Networked Resources & Related Infrastructure • C1  Cost of electronic full-text journals • C2  Cost of electronic reference sources • C3  Cost of electronic books • C4  Library expenditures for bibliographic utilities, networks, & consortia • C5  External expenditures for bibliographic utilities, networks, & consortia

21. Library Digitization Activities • D1 – Size of library digital collection • D2 – Use of library digital collection • D3 – Cost of digital collection construction & management (Collecting these data requires staff familiar with the digital environment.)

22. Performance Measures • P1 – Percentage of electronic reference transactions of total reference • P2 – Percentage of virtual visits of all library visits • P3 – Percentage of electronic books to all monographs

23. Characteristics of Each Recommended Measure • Definition • Rationale • Unit of Measure • Data source • Frequency • Process • Related Issues

24. R1 – Number of Electronic Full-text Journals • Definition - Number of electronic full-text journal subscriptions – by individual institution or consortia licensing. • Rationale – Documents degree of expansion of electronic subscriptions available – can be used to show good coverage & need for more funding. • Unit of measure – the journal subscription

25. R1 – Number of Electronic Full-text Journals • Data source – local or vendors. • Frequency – annual, monthly, etc. • Process– parse into database or spreadsheet, update dynamically from local catalog or vendor record.

26. U1 – Number of Electronic Reference Transactions • Definition - number of electronic reference transactions – via e-mail, WWW form, etc. • Rationale – libraries are interested in tracking the development of new electronic services. Attempt to measure reference transactions through new electronic tools and services. • Unit of Measure – request count, time it took.

27. U1 – Number of Electronic Reference Transactions • Data Source – local server, manual tally, e-mail count. • Frequency – daily, monthly, annually, etc. • Process – clarify process, identify activity points, identify collectors of data, consolidate data. • Related Issues –This measure may have to broken down into additional data types – time, type of query, type of interaction, scheduling issues, measures of quality and reliability.

28. Outcomes of the Investigation • Mixed results • Participants wanted more definitive answers • Confirmed how difficult, but not impossible, to come to consensus • Realized how dependent libraries are for vendors to supply data • Recognized need for ongoing effort • Reaffirmed importance of using data for decisions

29. E-Metrics Next Steps Call for participation among members to test proposed measures for 2002/2003 (over 35 participants to date) • Examination of the deliverables from the first phases • Collect FY02 totals • Compilation • Data analysis • Distribution for discussion • Analysis of approaches • Best practices for work processes • From E-metrics project • Locally developed

30. E-Metrics Next Steps • Continued work with vendors through international COUNTER project • Continued work with international standards activities • Workshops and training to develop necessary data analysis skills

31. Project Documents • Measures for Electronic Resources (E-Metrics) Part 1: Project Background and Phase One Report Part 2: Phase Two Report Part 3: E-Metrics Instructional Module Part 4: Data Collection Manual Part 5: Library and Institutional Outcomes • www.arl.org/stats/newmeas/emetrics/

32. Additional Questions? • Please contact: Martha Kyrillidou Senior Program Officer for Statistics and Measurement Association of Research Libraries martha@arl.org