Reflections of an editor on research and practice?

1. Reflections of an editoron research and practice? Richard Smith Editor, BMJ Granada, May 2002 www.bmj.com/talks

2. What I want to talk about • The disconnect between practice and research • A rough history of health research in Britain • A vision of how to improve the connection • The relation between disease burden and volume of research • Setting research priorities

3. What I want to talk about • Where does innovation come from? • Peer review of research • Measuring the value of research • Disseminating research • How to get from research to change? • Conclusions

4. The disconnect between practice and research • Research is usually funded by the Ministry of Education, whereas health care and public health is funded by the Ministry of Health • Research is run by researchers who value basic science, discovery, and original questions, thinking, and methodology • The answering of practical questions is seen as dull, unoriginal, and “unimportant in scientific terms” • Nobel prizes go to the discoverers of molecular mechanisms not those who work out the most cost effective method for treating incontinence

5. The disconnect between practice and research • There is often no mechanism to transmit the questions of practitioners (and patients) to researchers • Scientists are wary of directed research: “only scientists can know what is scientifically important”; “directed research leads nowhere” • The results of research do not seem valuable to practitioners • The idea that doctors are scientists is a myth

6. The disconnect between practice and research • Most practitioners are not competent researchers • Nor are practitioners sophisticated consumers of research • “Practice is one thing; research another. I make decisions based on my experience and what clinical experts advise” • Health policy makers sometimes boast that they don’t use research results • Evidence based practice is a force for change, bringing research and practice together

7. The disconnect between practice and research • It must be “evidence informed” practice not “evidence tyrannised” practice • But we realise that fewer than 5% of studies in medical journals are both valid and relevant to clinicians or policy makers; in most journals it’s less than 1% • We have good evidence on perhaps 10% of treatments and a smaller percentage of questions about diagnosis, symptoms and signs, and prognosis • Evidence needed for health management and policy is even weaker

8. A rough history of health research in Britain • 1900s--independent researchers • 1930s--Medical Research Council (MRC) begins • 1940--Pharmaceutical companies begin to do a great deal of research • 1980--MRC begins some health services research • 1986--House of Lords realises that the National Health Service has almost no research capacity

9. A rough history of health research in Britain • 1990--NHS research and development directorate established • vision is a “knowledge based health service” • aim is to spend 3% of NHS turnover on R&D • programme attracts international interest • 2001--NHS R&D programme still there but is less central than it once was

10. Bringing practice and research closer together: a vision • Patient asks a question to a doctor • Doctors consults databases on what the evidence says (Cochrane Library, Clinical Evidence, or an electronic decision support system) • (Or, increasingly, patient consults the same knowledge sources as the doctor--besttreatments.org) • If there is evidence, patient and doctor discuss best course of action

11. Bringing practice and research closer together: a vision • If there is no evidence, then a systematic review may be needed • Or the patient and doctors consult the meta-register of trials underway • If there is a trial, the patient may enter the trial (knowing that patients treated in trials do better than others no matter whether they get the active treatment) • If there is no trial, then the patient and doctor register the question with a central database

12. Bringing practice and research closer together: a vision • Trials can then be conducted to answer the questions that are most important and arising most commonly • The information sources needed to achieve this vision exist for questions on treatment • (The culture and the infrastructure do not exist) • The information sources do not exist for questions on diagnosis, prognosis, health policy, and much else--but could be created

13. The relationship between disease burden and the amount of research • Examples from the US, Africa, and neurology • The 90:10 rule--90% of research is on diseases affecting 10% of the world’s population • The association is often small: some diseases with a small burden are highly researched, whereas some with a high burden are poorly researched

17. A system for prioritising research • Consider disease burden • Consider questions generated by patients and health care providers • Consider possible “research gain” (what are the chances that an investment could result in real advances?) • Incorporate social and professional values • Britain has had a system along these lines--but only for NHS R&D programme

18. Where does innovation come from? Two models • The linear model: curiosity driven research---applied research---experimental development---innovation • The market pull model: market need---applied research---experimental development--innovation

19. Project Hindsight (1966) • Examined 20 weapon systems (including Polaris) • Researchers identified 686 “research or exploratory development events” that were essential for development of the weapons • Only 9% were “scientific research” (0.3% basic research) • Only 9% of research conducted in universities

20. Project Hindsight (1966) • “Science and technology funds deliberately invested and managed for defence purposes have been about one order of magnitude more efficient in producing useful events than the same amount of funds invested without specific concern for defence needs.”

21. TRACES Study (1968) • Technology in Retrospect and Critical Events in Science • Origins of magnetic ferrites, video recorder, contraceptive pill, electron microscope, and matrix isolation • Looked back 50 not 20 years, as did Project Hindsight

22. TRACES Study (1968) • 340 events • 70% non-mission research, 20% mission oriented, and 10% development and application • Universities did 75% of non-mission and one third of mission oriented research

23. Comroe and Dripps (1976) • Julius Comroe, physiologist, and Robert Dripps, anaesthetist • The top 10 advances in cardiovascular and pulmonary medicine and surgery in the last 30 years • Around 100 specialists selected the top 10

24. Cardiac surgery Vascular surgery Drug treatment of hypertension Medical treatment of myocardial ischaemia Cardiac resuscitation Oral diuretics Intensive care units Antibiotics New diagnostic methods Prevention of polio Top 10 advances

25. Comroe and Dripps (1976) • Went back to the dawn of time • 137 “essential bodies of knowledge” • 500 essential or key articles • 41% not clinically oriented • 37% “basic: not clinically oriented” • 25% “basic: clinically oriented”

26. Conclusions from studies of innovation • The sources of innovation are numerous, varied, and scattered • Both the science push and market pull models of innovation are oversimplified • Research funders should not put all their eggs in one basket • Attempts to force “more relevant” research may backfire

27. Conclusions from studies of innovation • The coming together of different lines of research and and scientists from different disciplines seems to be important • Promoting interdisciplinary research may seed innovations • “Research into research” may be beneficial

28. Peer review of research • Research grants are often given after peer review • Which research will be published is often decided by peer review • But there are problems with peer review

29. Problems with peer review • A lottery • A black box • “Ineffective” • Slow • Expensive • Biased • Easily abused • Can’t detect fraud

30. Peer review • But it is hard to find an alternative to peer review • It’s like democracy--”the least bad system” • The answer seems to be to improve peer review with training, openness, blinding, etc

31. Measuring the value of research • The point of health research is to improve health • But researchers are usually rewarded according to measures of scientific value • These include the impact factor of the journal in which they publish--despite there being little or no correlation for individual authors between the impact factor of the journal in which they publish and citations to their articles • There are many other problems with impact factors--bias towards certain disciplines, US, methodology; data are often unreliable

33. Measuring the value of research • Royal Netherlands Academy of Arts and Sciences is trying to devise a measure of the social impact of research • Might include publications, software, products, press coverage, etc • But it’s not easy to find a reliable measure

34. My suggestions for measuring influence/impact • Level one (the highest): making change happen • Level two: setting the agenda for debate • Level three: leading by example • Level four: being quoted • Level five: being paid attention to • Level six (the lowest): being known about

35. Dissemination of research • There are tens of thousands of journals • Millions of studies are published each year • Most studies are neither valid nor relevant • It’s hard--usually impossible--for clinicians and policy makers to keep up • There is a need to review research results systematically

36. Dissemination of research • Evidence based journals--coverage of one off studies, not put into context • Cochrane Library--treatments only, big gaps, researcher (not clinician) led questions, complex • Clinical Evidence--treatments only, 160 topics, 400 needed

37. Dissemination of research • Guidelines--cover only some topics, sometimes not evidence based, go beyond the evidence, tell people what to do • Appraisals by National Institute of Clinical Excellence (NICE) --cover only a few topics, must incorporate evidence, cost, and “values,” insufficiently transparents

38. From information to change Change Know how Know about Information Data

39. Failures to follow evidence • Aspirin underused in patients with vascular disease • ACE inhibitors underused in patients with heart failure • Inhalational steroids underused in patients with asthma • Antibiotics overused in patients with upper respiratory tract infections and acute otitis media • Enemas, pubic shaving, and episiotomies overused in women in labour

40. From research to change • “We should stop all research for two years and concentrate instead on implementing what we already know.” • Somebody in, I think, the Lancet quite some time ago

41. From information to change • Achieving change is hard • Information on its own rarely changes practice • Combinations of audit and feedback, computerised reminders, educational outreach, and interactive educational sessions will sometimes change practice

42. From information to change • Interactive learning • Improvement methods • Organisational development • Consultancy • “Just in time” information

44. “The thing” that will save us • Able to answer highly complex questions • Connected to a large valid database • Electronic - portable, fast, and easy to use • Prompts doctors - in a helpful rather than demeaning way • Connected to the patient record • A servant of patients as doctors • Responds to the need for psychological support and affirmation

45. Conclusions • Research and practice are currently not well connected • It’s possible to envision how they might be better connected • Some substantial health problems are poorly researched, while some smallish problems are heavily researched • Mechanisms are needed to set priorities in health research

46. Conclusions • Research into sources of innovation suggests that different sorts of research in different circumstances are important • Innovation often comes from interdisciplinary innovation • Peer review has many problems but can probably be improved

47. Conclusions • Better methods are needed for measuring the performance of health researchers • The dissemination of research results is inadequate, but better means are appearing • Moving from research to change is hard, but we can see how to do it better