Institutional Background

Systematic Quantitative and Qualitative Reference Transaction Assessment: An Approach for Service Improvements • Ellie Buckley • KorneliaTancheva • Xin Li • August 6, 2008

Institutional Background • Cornell University • Founded in 1865 • Ivy League and land-grant • 13,500 undergraduate and 7,000 graduate/professional students • 3,000 faculty and 12,000 staff • The Library System • 20 distributed libraries • 460 staff • $53.6 million total expenditure • 7.8 million print volumes, 350,000 networked electronic resources

Reference Statistics Reporting System (RSRS) • RSRS was created to bring consistency across the system and, more importantly, to better understand the number and type of reference questions. • The web-based system was developed in collaboration with a group of Cornell Computer Science students. • A 12-week sampling scheme was developed with input from a Cornell statistics professor. • RSRS launched in August 2006.

RSRS Data Collection Location Staff type

RSRS Data Collection Transaction Type(Reference, Directional, Technical Reference, Equipment) Duration Mode Question Content (“What your visitor’s query was about.”) Notes Date-Time Stamp

RSRS Assessment Project • This project analyzed 27,227 reference desk transactions from two very different libraries: • Olin Library desk is staffed ~90 hours per week with information assistants, librarians, and some students. • Uris Library desk is staffed 35-40 hours per week with mostly students.

Guiding Research Questions • What is the distribution of the traffic at the two reference desks? • What is the distribution of the duration of the transactions? • What is the distribution of the transaction types? • What are the major kinds of questions asked at the desk? • How are the questions distributed over the semester? • How might these data be used as a baseline for future analyses and decision-making?

RSRS Assessment Project • The project proceeded in two phases: • Phase One: quantitative data descriptions of traffic patterns, transaction duration, mode, and staff data. • Phase Two: a multi-part exploration into the nature of the questions brought to the reference desks.

Phase One: Quantitative data description • Most transactions are completed within 5 minutes.

Phase One: Quantitative data description • Reference Assistants handle the majority of transactions followed by Reference Librarians.

Phase One: Quantitative data description • Most traffic occurs early in the workweek and during the mid-day hours.

Phase One: Quantitative data description • The desk is busiest during the semester weeks, fall more than spring.

Phase Two: The nature of the questions • We wanted to gain a better understanding of the nature of the reference questions by analyzing the Question Content data: the open ended description of “What your visitor’s query was about.” • Phase Two had 3 parts, we: • looked at issues of reliability of the Transaction Type variable distribution; 2) examined the reference desk questions using principles of qualitative analysis; and 3) performed a systematic content analysis.

Phase Two: Transaction Type reliability • Two “Raters” independently reviewed the Question Content in a random sample of 1,020 transactions and assigned a transaction type, post hoc.

Phase Two: Transaction Type reliability • We used Cohen’s Kappa to measure inter-rater agreement. It evaluates rater agreement against agreement that could have happened by chance.

Phase Two: Qualitative analysis • Open-ended data can be difficult to analyze! • Qualitative analysis allows meaningful themes to emerge, even if those findings cannot be “counted”. • We used principles of Grounded Theory, an inductive approach. Findings are allowed to emerge from the data, rather than imposed on the data from the top down. • patron looking for Collective action and the civil rights movement • print copies of the journal Nature. • stapler • phone number for dspace admin. • hq's • How tall is McGraw Tower? (173 feet!) • problem with access to a Blackwell journal online • is this book here in Olin • Printer jam • catalog check • how does my hair look? (referred patron to the bathroom)

Phase Two: Qualitative analysis • One of the themes that emerged was a functional typology. • The patrons appear to view the reference staff as serving one of three broad functions or roles: • Research-Teaching Assistant • Information Central • Problem Solver

Phase Two: Qualitative analysis • Research-Teaching Assistant • Sample question content: • book by samuelson and nordhaus? • I need help finding two articles for my class. I should know this 'cuz am a senior, but.... • properties of sandpaper • what does the catalog search? how to search beyond cornell? • help with PAM assignment • how do you spell "cataracts?"

Phase Two: Qualitative analysis • Information Central • Sample question content: • directions to 106 CRIO office • Explained stacks layout to new student who could not find books. • overdue fines from summer...help! • clock tower access and chime concert info • closest printer? • college town? closer places to eat? • did you see a family walk by?

Phase Two: Qualitative analysis • Problem Solver • Sample question content: • Help opening a CDROM. • I need a pencil! • computer ate paper (timeout error) This was on friday...how to prove to professor that paper was actually written? • Can’t connect RedRover…“secure” error message • help with a bad citation

Phase Two: Content analysis • Content analysis also relies on intensive and iterative review of the open-ended Question Content data, but it is a more deductive, top-down approach: we also wanted to count things. • We used systematic “coding rules” to classify and categorize (and then count) the transactions based on the Question Content data. • SPSS Text Analysis for Surveys software helped to partly automate the process.

Phase Two: Content analysis

Outcomes • Based on the data, we were able to: • Modify staffing by increasing the student coverage of Uris, and providing double staffing at Olin only between 1 and 5pm. • Validate the sampling method. • Implement RSRS system and interface improvements. • Gain a clearer understanding that significant staffing changes would require additional, multidimensional data collection. • As a “bonus,” the Research & Assessment Unit: • Developed, tested, and revised workflow and protocols. • Learned valuable lessons for subsequent tasks.

Conclusions • This project documented and interpreted the trends in academic library reference beyond the mere numbers of transactions. It helped inform decisions on differentiated reference desk staffing by documenting traffic patterns and also identified areas for potential service improvements. • Additionally, this project will serve as a “jumping off point” for other reference research questions and provide important baseline data for assessments of service improvements in the future.

Thank you for attending! • (Any questions? Discussion?) • Ellie Buckley Xin Li Kornelia Tancheva • ebuckley@mac.comxin.li@cornell.edu kt18@cornell.edu

Institutional Background