RAE & REF

1. research HEFCE RAE & REF Presented at Council of Deans, Seminar, Manchester, Feb 2009 By Hugh McKenna Karen Cox

2. BRIEF BACKGROUND • Conducted jointly by the Higher Education Funding Councils for England (HEFCE), Scotland (SHEFC), Wales (HEFCW) and Northern Ireland (DEL) • Purpose: to produce ratings of research quality for the HE funding bodies to determine research funding allocations to HEIs

3. RAE 2008 • Principles of Equity, Diversity, Equality, Expert Review, Clarity, Consistency, Continuity, Credibility, Efficiency, Neutrality, Transparency. • All types of research, including practice-based research, applied research, basic/strategic research, interdisciplinary research. • Assessment based on peer review. • 15 Panels and 67 Sub Panels

4. Sub Panel 11 • Chair • Prof Hugh McKenna, University of Ulster • Panel Secretaries • Mrs Lilian Caras, Brenda Purkiss • Members • Ms Sarah Buckland, INVOLVE • Prof Charlotte Clarke, University of Northumbria • Prof Karen Cox University of Nottingham • Prof Dame Pauline Fielding, • Prof Martin Johnson, University of Salford • Prof William Lauder, University of Dundee • Prof Karen Luker, University of Manchester • Prof Dame Jill Macleod-Clarke, University of Southampton • Dr Maggs Maguire, Department of Nursing, Scottish Executive • Professor Dave Thompson, University of Leicester • Prof Ruth Northway University of Glamorgan. • Prof Anne Marie Rafferty, Kings College London • Dr Kate Seers, University of Warwick • Prof Roger Watson, University of Sheffield • Prof Anne Williams, University of Wales, Cardiff

5. Sub Panel 12 • Chair • Prof Julius Sim, Keele University • Panel Secretaries • Mrs Lilian Caras, Brenda Purkiss • Members • David Billington, Liverpool John Moores • Michael Campbell, University of Sheffield • Aedin Cassidy, University of East Anglia • Stephen Downes, University of Ulster • Pamela Enderby, University of Sheffield • Bernard Gilmartin, Aston University • Martin Griffin, Aston University • Jenny Hewison, University of Leeds • Tracey Howe, Glasgow Caledonian University • Ian Kitchen, University of Surrey • Di Newham, King's College London • Margaret Nicol, Queen Margaret University Edinburgh • David Rogers, University of Portsmouth • Diane Scutt, University of Liverpool • Debbie Sell, Great Ormond Street Hospital • Christina Victor, University of Reading • Keith Wafford, Eli Lilly and Company

6. Main Panel C • Professor Maggie Pearson (Chair) • Professor David Williams (10) • Professor Hugh McKenna (11) • Professor Julius Sim (12) • Professor Robert Hider (13) • Professor Mi Ja Kim • Professor John Stamm • John Stageman • MRC/AMRC/ESRC/DoH/Scot Gov Observers • Secretariat

7. RAE 2008 • January 2005 Guidance to panels issued; • Summer 2005 Guidance on submissions issued; • Autumn 2005 Consultation on criteria and working methods; • February 2006 Final criteria and working methods issued; • 31 July 2007 End of assessment period for research income and research student data • 31 December 2007 End of publication period (cut-off point for publication of research outputs) • 31 October 2007 Census date • 30 November 2007 Closing date for submissions • December 2008 Results published

8. DEFINITION OF RESEARCH • Research - original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce and industry, and to the public and voluntary sectors; scholarship*; the invention and generation of ideas... etc, where these lead to new or substantially improved materials, devices, products and processes... [* The creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databases.] • It excludes routine testing and routine analysis of materials, components and processes (such as for the maintenance of national standards…). • It also excludes the development of teaching materials that do not embody original research”. • It excludes Audit

9. CRITERIA & WORKING METHODSSub Panel 11 • “Research activity relevant to the disciplines of nursing and midwifery, specialist community public health nursing and all the contexts within which they operate, including policy, practice, education and management”. • Proportional weighting given to quality of Research Outputs, Research Environment and Research Esteem; • Opportunity to refer to other Sub Panels and specialist advisors;

10. CRITERIA & WORKING METHODSSub Panel 12 • “The UOA includes (but is not limited to): biomedical sciences; nutrition and dietetics; optometry and orthoptics; radiography; podiatry; occupational therapy; physiotherapy; speech and language therapy; arts therapies; health promotion; psychosocial and ethical aspects of health and healthcare; associated health services research (to include methodological work on quantitative or qualitative procedures)”; • Proportional weighting given to quality of Research Outputs, Research Environment and Research Esteem; • Opportunity to refer to other Sub Panels and specialist advisors.

14. Star Ratings?

15. Overall Overall Overall Overall Overall The overall quality profile The overall quality profile The overall quality profile The overall quality profile quality profile quality profile quality profile quality profile quality profile comprises the aggregate comprises the aggregate comprises the aggregate comprises the aggregate of the weighted profiles of the weighted profiles of the weighted profiles of the weighted profiles produced for outputs, produced for outputs, produced for outputs, produced for research outputs, 4* 4* 4* 4* 4* 4* 3* 3* 3* 3* 3* 2* 2* 2* 2* 2* 1* 1* 1* 1* 1* u/c u/c u/c u/c u/c research environment and research environment and research environment and research environment and Quality level Quality Level Quality Level Quality Level . . . . esteem indicators esteem indicators esteem indicators esteem indicators % of research % of Research % of Research % of Research 20 20 20 20 20 20 25 25 25 25 25 30 30 30 30 30 15 15 15 15 15 15 10 10 10 10 10 10 activity Activity Activity Activity Research environment Research environment Research environment Research environment Research environment Research environment Esteem indicators Esteem indicators Esteem indicators Esteem indicators Esteem indicators Research Esteem Research outputs Research outputs Research outputs Research outputs Research outputs Research outputs 4* 4* 4* 4* 4* 4* 4* 4* 4* 3* 3* 3* 3* 3* 3* 3* 3* 3* 2* 2* 2* 2* 2* 2* 2* 2* 2* 1* 1* 1* 1* 1* 1* 1* 1* 1* u/c u/c u/c u/c u/c u/c u/c u/c u/c 4* 4* 4* 4* 4* 4* 4* 4* 3* 3* 3* 3* 3* 3* 3* 3* 2* 2* 2* 2* 2* 2* 2* 2* 1* 1* 1* 1* 1* 1* 1* 1* u/c u/c u/c u/c u/c u/c u/c u/c 4* 4* 4* 4* 4* 3* 3* 3* 3* 3* 2* 2* 2* 2* 2* 1* 1* 1* 1* 1* u/c u/c u/c u/c u/c 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 25 15 15 15 15 15 15 15 15 15 20 20 20 20 20 20 20 20 20 15 15 15 15 15 15 15 15 15 30 30 30 30 30 30 30 30 20 20 20 20 20 20 20 20 10 10 10 10 10 10 10 10 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 15 15 15 15 15 25 25 25 25 25 35 35 35 35 35 15 15 15 15 15 10 10 10 10 10 eg eg eg eg 70% 70% 70% 70% 70% eg eg eg eg 20% 20% 20% 20% eg eg eg eg 10% 10% 10% 10% (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 5%) (Minimum 50%) (Minimum 50%) (Minimum 50%) (Minimum 50%) How did this work in reality? The overall quality profile comprises the aggregate of the weighted profiles produced for research outputs, 4* 3* 3* 2* 2* 1* 1* u/c u/c research environment and Quality level . esteem indicators % of research 20 25 25 30 30 15 10 activity 4* 4* 3* 3* 2* 2* 1* 1* u/c u/c 15 15 25 25 35 35 15 15 10 10 25% 5%

16. Examples - Quality Profile

17. Overview - Sub Panel 11 • 761 individuals were returned, submitting 2,851 outputs, mostly from peer reviewed journals; • Selectivity in number of staff returned; • Selectivity in type of staff returned; • Selectivity of panel/sub panel; • 2001 – 43 submissions, 2008 - 35 submissions; • All thirty-five submissions were reviewed in detail by every sub-panel member; • All quality profiles represent the consensus of the sub-panel as a whole; • Strict cognisance was taken of equal opportunity issues and individual staff circumstances.

18. RAE 2008: Submissions to Sub Panel 11- THE STRENGTHS • 44th out of the 67 subject areas in the RAE; • Internationally recognised research in all submissions; • Strengths in translational/applied research; • Good service user involvement; • Positive impact on patient/client care; • Good interdisciplinary work – various disciplines; • Good PhD completions in review period (446); • Good international collaborations; • Good methodological and theoretical sophistication; • Strategic research appointments; • Research linked to national and international priorities;

19. RAE 2008: Submissions to Sub Panel 11- THE STRENGTHS • Impact on national and international health care policy; • Substantial competitive research grant capture (> £103 M); • Modern and high quality infrastructure; • Robust staffing policies; • Emphasis on capacity building and sustainability; • Realistic and sustainable research strategies; • Highly focused, strategically sound, financially viable; • Good links between NHS, Voluntary bodies and HEIs; • Esteem indicators - numerous awards, involvement in prestigious national and international committees, funding bodies & editorial boards.

20. RAE 2008: Submissions to Sub Panel 11- THE WEAKNESSES • In some institutions service user/public engagement was very limited; • Few departments were actively engaged in technology and knowledge transfer activities; • Very few links with industry; • Some researchers working in isolation in poorly developed research environments; • Lone researchers and small disparate groups working on the same topic area; • Little evidence of activity around the economic evaluation of what nurses and midwives do or on the evaluation and impact of new roles;

21. RAE 2008: Submissions to Sub Panel 11- THE WEAKNESSES • Fewer outputs on educational and managerial issues; • Research into aspects of care fundamental to nursing and midwifery were not sufficiently evident in submissions; • Over one third of the 1,283 research studentships were supported by the submitting HEIs themselves; • The doctoral degrees awarded per research student FTE were lower than expected; • Only fifty-nine research-active staff were returned as designated ‘early career researchers’; • Insufficient critical mass, inadequate onward investment and inferior succession planning.

22. Overview- Sub Panel 12 • 68 submissions - 18 to this sub-panel for the first time; • 1,622 individuals returned, submitting 6,216 outputs; • External research income totalled £355.7 million; • Varying levels of selectivity exercised by HEIs, in submissions and outputs; • Heterogeneous in substantive and methodological terms; • Very wide distribution of quality across the submissions; • All quality profiles represent the consensus of the sub-panel as a whole; • Strict cognisance was taken of equal opportunity issues and individual staff circumstances;

23. RAE 2008: Submissions to Sub Panel 12- THE STRENGTHS • Clearly-articulated and focused research strategy; • Coherent and synergistic research groupings, with an appropriate ratio of researchers to research foci; • Strong evidence of sustainability; • Well-balanced portfolio of research funding, from competitive peer-review funding sources; • Appropriate and productive links with partners in the NHS and industry, and with service users; • Good evidence of translational research.; • Outputs showed rigorous and innovative methodology; • Research outputs had clear evidence of actual or potential impact on practice or policy.

24. RAE 2008: Submissions to Sub Panel 12- THE STRENGTHS • Evidence of internationally excellent or world-leading research in virtually all the disciplines – especially biomedical science, nutrition and optometry; • Evidence of investigations into the efficacy or effectiveness of therapeutic interventions; • Large body of important work on the underpinning basic sciences; • Studies providing valuable insights in the social and psychological aspects of health, illness, and the delivery of professional care; • Appropriate and well developed collaborative research links • Some Departments were leading important international research collaborations • 50% increase over review period in PhDs awarded (2,096).

25. RAE 2008: Submissions to Sub Panel 12- THE WEAKNESSES • Submissions were heterogeneous, consisting of a number of small and seemingly unrelated areas of work, raising questions as to cohesiveness and future proofing; • Lack of clear evidence of sustainability; • Lack of strategic focus, often with disparate research groupings; • Underdeveloped research infrastructure; • Inappropriately large number of research foci for the number of staff submitted; • Weak funding profile; • Discrepancy between the profiles for outputs and environment;

26. RAE 2008: Submissions to Sub Panel 12- THE WEAKNESSES • A number of outputs did not meet the RAE research definition; • Need for an increased focus on the many areas within AHPs where the evidence base is as yet underdeveloped • 3,426 new research studentships - institutional self-funding (36%); Research Councils (11%); • High quality doctoral work may not have attracted, or been unable to access, Research Council funding; • Research outputs that were of poor methodological quality and/or of little potential impact; • Poorly developed interdisciplinary and/or institutional collaboration.

27. Purpose of QR(points from HEFCE) Creating a sustainable and flexible national baseline capacity which enables the sector to respond strategically to a changing external environment To maintain a research base of world leading quality across the full range of disciplines To encourage institutions to take investment decisions which lead to excellence in research, funded selectively by reference to robust indicators of research quality. Invest in innovative research, including in new fields and opening new lines of enquiry, supporting early-career staff who aspire to grant funding create potential to make connections across subjects

28. REF –current state of play HEFCE suggests reduce the burden on institutions Will be designed to take better account of the impact research makes on the economy and society. A pilot exercise for bibliometric indicators of excellence is now underway HEFCE will set out its plans for all aspects of assessment, including user-focused research and subjects where bibliometrics are not appropriate, by summer 2009.

29. Framework for assessment RAE assesses a body of research activity in terms of: • Quality of outputs • Research environment • Esteem • HEFCE suggests indicators in REF may also capture • How a research environment is supportive of User valued research • Evidence that specific work is esteemed for its quality by particular audiences

30. Tools and approaches for assessment HEFCE suggests assessment is likely to include: • Bibliometric analysis • Citations? • Expert review of outputs • Other available statistical indicators • Students (PG) and completions • Income • Studentships

31. Cunning Plans • Succession Planning; • Increase and support Early Career Researchers (ECR)s; • Ruthless selectivity in all things; • Staff appointments and support for ECRs -Home grow Vs recruit; • Publish for high citations; • Research Grants from prestigious bodies – NIH guidelines; • Network and collaborate for best esteem nationally and Internationally; • NHS-HEI collaborations; • Impact – does the research make a difference? • Service user involvement from design to dissemination; • Interdisciplinarity in the widest sense; esteem is important – who you work with • Translational & policy Research; • Internal Vs External Studentships; • Address policy issues; • Increase the amount and quality of pedagogic research.

32. Conclusion • The sub-panel were impressed with most of the submissions and with the pattern of 3 and 4 star research; • There are many models of good practice from which developing research groupings can learn in terms of research activities, outputs, environment and esteem; • Investment by Governments, funding bodies and universities has increased research capacity and developed research leaders; • These funding streams need to be sustained and enhanced if the upward trajectory and momentum are to continue and if the quality differentials between the strongest and weakest departments are to be addressed.

33. Additional slides added from notes taken by Alan Simpson at meeting of Academy of Nursing, Midwifery and Health Visiting Research (UK) 26/02/2009.

34. Alison Tierney, Scottish Exec • Need a continuous and stable funding stream to maintain and build on progress • In Scotland, over 40% submitted to other panels, e.g. social policy, psychology – some may be best researchers • What do we do with low quality and sub-Rae researchers? • Approx 10% N&M HEI staff submitted to Rae • How do we build structures and processes that best support HEI/Health Services partnerships in research and how best to measure outcomes? • Few joint academic-services authors of papers (7-10%?)

35. Developing HEI/NHS partnerships for research • Sustainable partnerships • Joint targets for research • Capacity building jointly agreed • Jointly managed research programmes • More shared infrastructure (as in medicine) • Joint and Honorary appointments at senior level • Clinical-academic careers (see Post Registration Career Framework for Nurses, Welsh Assembly consultation document, November 2008)

36. Professor Maggie PearsonChair, RAe 2008 Main Panel C • 70 HEIs in nurse education • 1996 36 HEIs and 396.5 staff submitted • 2001 43 HEIs and 578.4 staff • 2008 36 HEIs and 641.7 staff • So, over half did not submit – does this matter?

37. 1996 11.1 staff per submission • 2001 13.5 • 2008 17.8 • 7 HEIs submitted only in 1996 • 5 HEIs submitted only in 2001 • 14 ‘dropped out’ • 9 ‘new entrants’ in 2008 • 12 HEIs submitted to all 3 exercises • 7 high quality

38. Stronger groups Strategic, strong leadership, clear foci Peer reviewed, competitive funding Critical mass (no lone researchers) Institutional investment National/international collaborations Weaker groups Lack of strategy Weak leadership Small, disparate group Lone researchers 2008 C Report