Demonstrating excellence: a European Perspective

1. Demonstrating excellence:a European Perspective American Library Association 26th June 2010

2. Summary • Context • Pressures to prove excellence • Responses and experience • Quality • Impact • Value • Where next?

3. Context

4. Trends - Library concept • “Storehouse to service to educational integration” (Lancour, 1951) • “Digitisation increases tempo” but reduces explicit mediation • “Library as place” to “Learning space” • Reinvention, rebranding, repositioning of academic libraries

5. Environmental Trends • European “harmonisation” • Bologna declaration • New nations and national library systems • Economics • Competition (local and international) • Consumerism • Downturn

6. Bologna process • Driven by common European issues • Growth, diversification, skills shortages, graduate skills, life long learning • European Higher Education Area • 29 countries initially; now 49 • Common degree frameworks • “Institutions should routinely monitor, review and improve the effectiveness of the support services available to their students” Bologna QA guidelines

7. Cross pressures on assessment 1 • Quality assurance systems • Customer driven quality • “The student experience” • Deeper customer understanding • Digital and other generational issues • Impact • Teaching and learning • Research library contribution and advocacy

8. Cross pressures on assessment 2 • Statistical aggregation and development • Beyond national boundaries • Evidence-based management • Value • Costs and cost analysis • Cost comparison and benchmarking • Return on investment & contingent valuation • “Real” worth and value

9. RIN Report on academic library challenges • “ … there is a strong feeling among senior librarians that they have failed effectively to communicate the value of their services … there is an increasing risk that much of what libraries actually do may be invisible in a virtual environment. … we believe it is important that libraries should be able to show … that they provide services with demonstrable links to success in achieving institutional goals. Return on investment is thus an increasingly important issue. Libraries need to be more proactive in seeking to understand user behaviour and workflows; and in rigorously demonstrating the value of their activities … . The focus of performance indicators up to now has tended to be on inputs and outputs … rather than addressing the much harder issues relating to impact and value. … we believe it is essential that more work is done to analyse the relationships between library activities … and learning and research outcomes … .”

10. Comparison

11. SPEC Kit 303 in the UK and Ireland

12. Sample & characteristics • 77 libraries (43% of SCONUL membership but 60% of University institutions) • Majority engaged with PM from late ‘80s onwards • User surveys were first assessment activities in most cases • Rationale was internal and user driven

13. PM Activities in use Range of 3-19 of listed methods; median of 10; average of 10.6 • Statistics (96%) • Suggestions (91%) • Data mining (72%) • Outcome evaluation (67%) • Benchmarking (63%) • KPIs (63%)

14. Least used, but … • Value/ROI assessment • Impact assessment • Balanced scorecard • Physical orientation studies • Mystery shopper studies

15. Culture of assessment? • Results used to improve library (75%) • Evaluation for service quality (69%) • Assessment is a library priority (67%) • Staff development is adequate (13%) • Staff have necessary skills (26%) • Staff accept responsibility (34%)

16. Basic comparisons (ARL/SCONUL) SPEC Kit 303 UK & Ireland 77 of 129 (60%) 100% active 84% customer driven 9% accreditation driven Majority survey first Improvement 75% No particular training 51% Preference for surveys • 73 of 123 (60%) • 99% active • 91% customer driven • 29% accreditation driven • Majority survey first • Improvement 76% • No particular training 29% • Preference for interface testing

17. Responses Quality & Deeper Understanding

18. The Student Experience • National Student Survey • UK comparative & competitive data for guidance • Single library question • LibQUAL+ • Benchmarking non-competitive data for improvement and international comparison • SCONUL and other survey tools • “Student first” initiatives

19. National Student Survey

20. LibQUAL+ Consortia outside North America • Belgium • Canada • European Business Schools (EBSLG) • France • Hong Kong (JULAC) • Ireland • Japan • NHS England • Norway • UK & Ireland (SCONUL)

21. Overall SCONUL Sample to 2009 • Full variety of institutions • 50% of institutions* (53% of RAE top 50) • >53% of HE students potentially sampled (>850,000) *Based on a selection of Universities UK membership of 131

22. LibQUAL+ SCONUL results

23. Use for national policy influence • SCONUL LibQUAL+ results contributed to national enquiries on libraries and learning in higher education, with particular reference to learning spaces, and the competitiveness of UK University libraries in comparison to global peers

24. Research Information Network • Studies of research, researchers and information resource use • Impact of collaboration • Response of librarians to difficult economic times • The future academic library, with SCONUL and others • The value of the academic library to researchers, with RLUK

25. Deeper understanding • Space use • SCONUL Working Group on Space Planning • Snapshot survey figure: % responses to the statement ‘There is an increased incidence of students ‘booking’ study spaces by leaving possessions on desks during the exam period’

26. Responses Value & Impact

27. Impact initiative • July 2003-December 2005 • Two cycles • 22 UK Universities • Impact of a specific innovation • Majority focussed on information literacy or e-resource use

28. SCONUL’s ‘Performance Portal’ • The VAMP programme for value and impact measures, plus … • A Wiki of library performance measurement containing a number of ‘approaches’, each (hopefully) with: • A definition • A method or methods • Some experience of their use in libraries (or links to this) • The opportunity to discuss use

31. The Ontology of Performance • ‘Frameworks’ • ‘Impact’ • ‘Quality’ • ‘Statistics’ • ‘Value’

32. Value measurement in libraries “focusing on cost without being able to demonstrate [service] value and quality … leaves the initiative to people whose chief concern is cost-control or profit: the funders and the vendors” Whitehall, T (1995)

33. University influences • Research, Teaching & Reductionism • ‘Mode 1’ Research & impact ‘transcendental’ • ‘Mode 2’ Research & impact ‘instrumental’ • Value, Price & ‘Mandarinisation’ of research and its support • Learning as a set of discreet assessed modules • All of this may damage the idea of Libraries as ‘transcendent’, collective and connective services

34. Guardian editorial on Michael Sandel’s Reith Lectures, 2009 “the credit crunch has exposed myriad mirages, demonstrating how the market can get things badly wrong when it comes to valuing things … when bureaucracies price things which should not be priced, they start trading them off against other objectives, instead of appreciating their absolute obligations.”

35. Libraries and economic value: a review of recent studies A natural history of value initiatives: • Activity based costing for output efficiency • Perceived value based on labour saving • Balanced scorecard pressure for ’hard’ value measurement “demonstrate value by linking to the organisation’s value statements” Missingham, 2005

36. UK cost analyses • UK HE Transparency Initiative (TRAC) • University cost breakdown to identify and separate teaching and research costs • Open University • Business reporting • Process costing and continuous improvement • Service planning • Benchmarking (for international exercise) • ‘to generate real accountability’

37. Financial Benchmarking • International Groups • Commercial • LISU • Tribal Group • National • BIX – the Library Index

38. Contingent valuations and ROI • British Library • £4.40 for every £1 spent • Reviews and commentary • Aabo, 2009 • Missingham, 2005 • White, 2007

39. Intangible assets for academic libraries. • Requires intellectual capital reporting model • Using ROI, contingent valuation etc • Dimensions • Human capital • Structural capital • Relational capital Kostagiolas & Asonitis, 2009

40. Where next? • Comprehensive and holistic measurement • “True worth” • Value and impact • Real understanding • Transcendent valuation • The narrative of worth

42. Valuation to values “Whenever valuation takes place … values must enter in .. in evaluation an indispensable recourse to underlying values is involved” “values cannot be deduced from .. data or logic … they have to be chosen”

43. Values-based value • Competing values • Establishing the link between values and value • Reawakening of the broader common good of libraries • University of York • Values “jam” • User perceptions of value as opposed to quality • Value(s) scorecard

44. Competing values

45. Staff values debate using ideascale

46. User values (Academic Engineer) • Library as a “real tangible physical expression of knowledge” • “Intellectual heart, a collection of knowledge made without fear or favour” • Exaltation of solitary study - deeper understanding by “conquering the stuff alone” • Organisation of knowledge reflected in how things are laid out; browsing and walking through physical objects • Browsing; overview of knowledge by the way it is structured; ‘to steer thinking”; density tells you what’s important • “A real physical existing thing where I can see the celebration of scholarship”

47. Web references • BIX see: http://www.bix-bibliotheksindex.de/index.php?id=115 • Bologna see: http://ec.europa.eu/education/higher-education/doc1290_en.htm • CIBER see: http://www.ucl.ac.uk/infostudies/research/ciber/ • LIBER see: http://www.libereurope.eu/node/457 • Impact initiative (Poll & Payne, 2006) see: http://eprints.bbk.ac.uk/373/ • National Student Survey see: http://unistats.direct.gov.uk/ • Research Information Network see: http://www.rin.ac.uk/ • SCONUL Performance Portal see: http://vamp.diglib.shrivenham.cranfield.ac.uk/ • Performance measurement in the UK & Ireland (Stanley & Killick, 2009): http://www.arl.org/bm~doc/LibraryperformanceUKIreland.pdf

48. J. Stephen Town jst504@york.ac.uk