Evidence-based practice and policy in California: Choices ahead

1. Evidence-based practice and policy in California: Choices ahead Eileen Gambrill School of Social Welfare University of California at Berkeley Presented July 14, 2005 Leadership Symposium on Evidence-based practice in the human services, Sacramento, California

2. My goals: • To describe the philosophy and process of evidence based practice and policy as presented in original sources (e.g. Sackett et al, 1997, 2000; Gray, 1997, 2001). This is a new educational and practice paradigm for closing the gaps between research and practice in order to maximize opportunities to help clients and avoid harm. It attends to ethical, evidentiary and application concerns. It has not “been around for decades” as suggested in Guide for Child Welfare Administrators on Evidence Based Practice.(2005) Chadwick Center, Children’s Hospital of San Diego, p. 5. 2. Describe the origins of evidence-based practice. Gambrill (2005)

3. My goals, cont.: 3. Suggest choices ahead for the state of California regarding what view of evidence-based practice and policy to use. • Will we take advantage of the new and evolving paradigm of EBP described by its originators that: • is systemic; it has implications for educators and researchers as well as for administrators, line staff and clients • attends to ethical issues such as informed consent • attends to application issues • offers a way to handle uncertainty in making decisions in an honest way (sharing ignorance as well as knowledge; there is a willingness to say “I don’t know) Gambrill (2005)

4. My goals, cont. • Will we define this narrowly as basing decisions on practice related research; using practice guidelines. This view leaves out the role of clinical expertise, attention to client values and preferences, and application problems. • Will we define this as business as usual: use the name but not the substance. Gambrill (2005)

5. Ethical Obligations • Beneficence • Avoid Harm • Informed Consent • Autonomy; Self-determination Gambrill (2005)

6. Evidence-based Practice and Care EBP “is the integration of best research evidence with clinical expertise and [client] values (Sackett, Straus, Richardson, Rosenberg & Haynes, 2000, p.1). It is the conscientious, explicit and judicious use of current best evidence in making decision about the care of individual [clients]” (Sackett, et al.,1997, p.2). Evidence-based health care refers to “use of best current knowledge in decision making about groups and populations” (Gray, 2001). The origin of the philosophy and process of EBP is not the Institute of Medicine (IOM). (See for example Sackett et al., 1997, 2002; Gray, 1997, 2001). Gambrill (2005)

7. A Model for Evidence-Based Decisions Client Characteristics and Circumstances Clinical Expertise Client preferences and actions Research Evidence Haynes, R., Devereaux, P. & Guyatt, G. (2002). Editorial: Clinical expertise in the era of evidence- Based medicine and patient choice. ACP Journal Club, 136: A-11-14. Gambrill (2005)

8. Evidence-based Practice, cont. Clinical expertise includes use of effective relationship skills and the experience of individual helpers to rapidly identify each client’s unique circumstances, characteristics and “their individual risks and benefits of potential interventions and their personal values and expectations” (p.1). It is drawn on to integrate information from these varied sources (Haynes, Devereaux & Guyatt, 2002). Client values refer to “unique preferences, concerns and expectations each [client] brings to an…encounter and which must be integrated into…decisions if they are to serve the [client]” (p.1). Gambrill (2005)

9. The process of Evidence-based Practice: • Convert information needs related to practice decisions into answerable questions • Track down, with maximum efficiency, the best evidence with which to answer them • Critically appraise that evidence for its validity, impact (size of effect) and applicability (usefulness in practice) • Apply the results of this appraisal to practice/policy decisions. This involves deciding whether evidence found (if any) applies to the decision at hand (e.g., is a client similar to those studied, is there access to services described) and considering client values and preferences in making decisions as well as other application concerns • Evaluate our effectiveness and efficiency in carrying out steps 1-4 and seek ways to improve them in the future (Sackett et al., 2000, pp.3-4).

10. Different Kinds of Questions • Effectiveness: In elderly clients who are depressed, what method is most effective in reducing depression? • Prevention: In poor, inner-city children, are head start programs effective in decreasing school drop out? • Risk/prognosis: In children who are abused, are actuarial or consensus risk assessment measures most predictive of future abuse? • Description/Assessment: In families in which there is parent-child conflict, is self report or observation most accurate in describing interaction patterns? Gambrill (2005)

11. Different Kinds of Questions, cont. • Other kinds of questions include harm, cost and self-development. • Different questions require different kinds of research methods to critically appraise proposed assumptions. These differences are reflected in the use of different “quality filters” to search for and appraise related research (e.g. see Gibbs, 2003; Sackett et al, 2000). Gambrill (2005)

12. Critical Thinking Values: Integral to EBP • Courage: Critically appraise claims regardless of negative reactions • Curiosity: An interest in deep understanding and learning • Intellectual empathy: Accurately understanding and presenting the views of others • Humility: Awareness of the limits of knowledge including our own; lack of arrogance (e.g. promoting bogus claims of effectiveness) • Integrity: Honoring the same standards of evidence to which we hold others • Persistence: Willingness to struggle with confusion and unsettled questions Based on R. Paul (1993). Critical Thinking: What every person needs to survive in a Rapidly changing world (3rd Ed.) Foundation for Critical Thinking.www.critical thinking.org Gambrill (2005)

13. Examples of Innovative Technology • Evidence cart (Strauss and Sackett, 1998) • Knowledge manager (resource person) (Gray, 1998) • Decision aids (e.g. computer based, client informed choice form) • Systematic Reviews: Cochrane and Campbell databases Gambrill (2005)

14. Evidence-based practice: • Is an alternative to authority-based practice. Appeals to: A famous person Popularity Tradition Consensus “If Freud said it,it must be true.” “Eighty percent of social workers use…I’m going to use it too.” “That’s the way we have always done it.” “We all believe that ____” Gambrill (2005)

16. Evidence-based practice • It is as much about the ethical obligations of educators and researchers to be honest brokers of knowledge and ignorance as it is about the obligations of practitioners and administrators. • Many kinds of “evidence” may be considered in deciding what outcomes to pursue and what services to offer including:1) client preferences, 2) evidentiary status of different services, 3) resources available, 4) cost, 5) legal regulations, 6) codes of ethics Gambrill (2005)

17. Implications of Evidence-Based Practice • Move away from authority-based decision making (appeal to tradition, consensus, popularity, status). • Honor ethical obligations to clients such as informed consent. • Make practices and policies and their outcomes transparent. • Attend to application problems: Encourage a systemic approach to improving services. • Maximize knowledge flow: Honest brokering of knowledge: Sharing ignorance and uncertainty as well as knowledge. Gambrill (2005)

18. Origins of Evidence-Based Decision Making* • Study of variations in service delivery and clinical practice. • Identification of gaps that show that people were not acting systematically or promptly on research findings. • Failure to start services that did more good than harm at reasonable cost. • Failure to stop services shown to be of little value. • Economic pressures. • Key individuals (e.g., David Sackett) • The knowledge revolution: • Increased recognition of harmful side-effects of health care • Increased recognition of misplaced faith in adequacy of peer • review • Flaws in books and editorials • The evolution of the systematic review and web based updates Gambrill (2005)

19. Origins of Evidence-Based Decision Making*, cont. 6. The web revolution. 7. Appeal to clinicians (breaking down the division between research and practice-emphasizing that clinicians have to use their judgment and scientific training to interpret and integrate guidelines). 8. The value of EBP to clients. *Source: Gray, J.A.M.(2001). Evidence-based medicine for professionals. In A. Edwards and G. Elwyn (Eds.), Evidence-based patient choice: Inevitable or impossible? (pp. 19-33). New York: Oxford. Gambrill (2005)

20. The Key Choice: How to view evidence-based practice • Whether to draw on the broad philosophy and evolving process of EBP described by its originators (e.g. Sackett et al, 1997, 2000) as a way to handle the uncertainty in making decisions in an informed, honest manner (sharing ignorance as well as knowledge). • This choice has implications for researchers and educators as well as for clients, clinicians and administrators. It will affect all of the following choices Gambrill (2005)

21. Choices, cont. 1. Who will select the questions on which research efforts are focused? • An elite? (a state or national board) • Administrators? • Clients and line staff (They know what questions arise most often in everyday practice; lets gather this information). • What criteria will be used to select services? • Consensus (e.g. opinions of a state advisory body) • Popularity • What is available • Evidentiary (shown via critical tests to do more good than harm) • Cost/benefit ratio • Ethical concerns Gambrill (2005)

22. Choices, cont. 3. How transparent (honest) to be regarding the evidentiary status of services, (for example acknowledging lack of evidence for services offered), with clients, staff, funding bodies, consumer advocates. Whether to involve clients as informed participants; clearly describe the evidentiary status of services provided including those offered by agencies from which services are purchased. Gambrill (2005)

23. What Hierarchy of Evidence To Use: The one suggested in the Guide for Child Welfare Administrators: (GCWA) l. Well-supported, proven efficacious practice 2 Supported and probable efficacious practice. 3. Supported and acceptable practice. 4. Promising and acceptable practice. 5. Innovative or novel practice. 6. Experimental or concerning practice. Gambrill (2005)

24. Concerns with the GCWA hierarchy 1. Use of justificationary language that encourages confirmation biases and wishful thinking: • Use of the word ‘proven’ • Repeated use of words such as ‘support’ • The word ‘harm’ is not mentioned. 2. Hides the fact that most services are of unknown effectiveness. 3. Use of vague terms such as ‘probable’, ‘efficacious’, 4. In Appendix A we find that ‘clinical-anecdotal literature’ and ‘generally accepted in clinical practice’ are included as indicators of level 1 evidence. Gambrill (2005)

25. A More Transparent Hierarchy (Gray, 2000) • Services that have been critically tested and found to help clients. • Services that have been critically tested and found to harm people. • Service methods are of unknown effectiveness and are not in a research study or are in poor quality research. • Services of unknown effect in a good quality research program. Gambrill (2005)

26. Choices, cont. 4. How rigorous to be in assessing the accuracy of claims. Consider differences between systematic and narrative (traditional) reviews. In a systematic review: • There is a search for all literature related to a question, • in all languages, in both published and unpublished • sources, including hand searches of journals. • The search process is clearly described including the • databases reviewed. Authors describe where they searched, • how they searched. • Rigorous criteria are used to appraise what is found and • these are clearly described. (See Cochrane and Campbell • Collaboration reviews). Gambrill (2005)

27. Choices, cont. (systematic reviews) Systematic reviews differ greatly from incomplete, non- rigorous, non-transparent reviews. Traditional narrative reviews do not control for as many biases and thus overestimate positive effects of services. They are misleading in their conclusions. Compare claims of effectiveness made by the developers of Multisystemic Therapy and conclusions of Littell’s (2005) in Children and Youth Services Review. Gambrill (2005)

28. Choices, cont. 5. Whether to blow the whistle on pseudoscience, quackery, fraud and related propaganda tactics such as bogus claims of what is done and ‘what is known’ Consider this: “The practice of child welfare has long been based on a strong…”(Guide for Child Welfare Administrators on Evidence-based Practice ,2005. p. 5) This is highly misleading if by ‘strong’ we mean based on high quality research and quality of services offered. Consider the poor quality of parent training offered (Gambrill and Goldman, 2005). Bogus claims are not benign. They have resulted in harming in the name of helping and interfere with further exploration; they mislead rather than inform. Gambrill (2005)

29. 6. Whether to purchase services from agencies that • offer ineffective or harmful services. Whether to: • Stop using ineffective and harmful services • Start using effective services • 7. How systemic to be. • Whether to provide the resources and training needed for staff to maximize the quality of services offered such as access to needed databases/knowledge manager • Whether to help all staff to acquire skills in evidence-based practice as well as effective self development skills drawing on evidence informed educational formats (See critiques of traditional continuing education formats such as Thomson O’Brien et al, 2003, Cochrane Library) Choices, cont. Gambrill (2005)

30. Choices, cont. What style of EBP to use:Will all staff be trained in all five steps of EBP: 1. Posing well structured questions 2. Searching efficiently and effectively 3. Critically appraising related research 4. Integrating relevant information 5. Evaluating what happens? • Or, will staff depend on others for steps 2 and 3? • The process of EBP requires all staff to be skilled in carrying out steps 1,4 and 5 Gambrill (2005)

31. Choices, cont. (How systemic to be) • Whether to implement needed organizational changes; e.g. see Evidence-based Health Care (Gray, 2001): • Hire knowledge managers (see Gray, 1998) • Arrange access to relevant databases • Set up a system for identifying errors so avoidable ones can be minimized and educate all staff that errors are typically systemic in cause • Collect questions that arise in everyday practice from line staff and clients. High frequency questions should guide research efforts (e.g. systematic reviews, new studies) Gambrill (2005)

32. Choices, cont. (How systemic to be) • Arrange facilitating incentive systems (e.g. reward staff for blowing the whistle on harmful and ineffective practices) • Administrators and supervisors model critical appraisal of claims; they raise questions regarding current practices and policies and welcome such questions from others; they avoid propaganda strategies such as ad hominem arguments Gambrill (2005)

33. Questionable excuses for not taking a systemic view • Critical appraisal and search skills are too hard for you and your staff to learn Not true; learning key questions to raise regarding different kinds of research including research reviews is easy, especially with the help of user friendly books such as How to Read a Paper (Greenhalgh, 2001). Don’t let the researchers and academics fool you; learning such skills is not that difficult. Len Gibbs teaches them to undergraduate students. Gambrill (2005)

34. Questionable excuses, cont. Regarding research reviews • Is the question addressed clear and relevant? • Do the authors clearly describe how they searched, where they searched, and what criteria they used to appraise studies? • Was a thorough search conducted using relevant databases? • Did the search cover unpublished as well as published work? • Were rigorous criteria used to review research? For effectiveness questions: • Was the sample size adequate? • Was there a comparison group? • Were subjects randomly distributed to different groups? • Were ratings of outcomes blind? • Was there a follow up period? • Was there an intention to treat analysis? Gambrill (2005)

35. You can: • Keep a copy of Carl Sagan’s Baloney Detection Kit available • Be familiar with Hugh Rank’s cues to propaganda pitches: Hi, Trust me, You need, Hurry, Buy • Read Thouless’ Straight and Crooked Thinking (1974) describing informal fallacies such as appeal to popularity and emotional reasoning Gambrill (2005)

36. Questionable excuses, cont. • We researchers and academics don’t have time to prepare systematic reviews Masters degree students in the evidence based social work program at Oxford complete Cochrane Reviews in one year. True, many will take longer and require considerable resources. But others will not. • We don’t need a systemic approach Services are unlikely to improve in quality in a fragmented approach to change. Gambrill (2005)

37. Indicators of choices made • Increase in the percentage of systematic compared to narrative reviews prepared related to questions of direct concern to line staff and clients • Increase in the percentage of clients involved as informed participants • Increase in the percentage of line staff, clients and administrators who have acquired skills in posing well-structured questions related to decisions that must be made and in critically appraising research. Gambrill (2005)

38. Indicators of choices made, cont. • Increase in the percentage of all involved parties who can spot human service propaganda such as inflated claims about “what we know” • Increase in the percentage of all involved parties who ask probing questions regarding the effectiveness of services and who avoid propaganda methods in related discussions (such as distorting the views of others) • Increase in the percentage of services purchased that have been critically tested and found to be effective • Decrease in the use of harmful and ineffective services Gambrill (2005)

39. Obstacles (examples) • Preference for authority-based practices and policies (examples): • A reluctance to be transparent • Inflated claims of effectiveness • Use of the term “evidence-based” to refer to business-as-usual • Prevalence of pseudoscience and propaganda and • underestimating vulnerability to their influence (e.g. see • Science and Pseudoscience in Clinical Psychology, • 2003). Arrogance Gambrill (2005)

40. Obstacles, cont. A justification approach to knowledge in which we search for data to confirm our views.(Use of word ‘proven’.) The symbiotic relationship between clients’ wishes to be helped and professionals’ desire to help. Gambrill (2005)

41. In Summary The philosophy and practice of EBP as described by its originators has implications for: • Professional education • Formats used and skills emphasized. • Research • Who decides on questions pursued • How research is conducted and reported, for example rigorous (likely to answer questions posed), clear descriptions of limitations, accurate rather than inflated claims of “what we know”, involvement of consumers in critiquing research. Gambrill (2005)

42. In Summary, cont. • Practice • Clients are involved as informed participants regarding the evidentiary status of services • Staff are accurately informed regarding the evidentiary status of services • Staff are skilled in carrying out the process of EBP (e.g. posing well structured questions that help them to draw on relevant research findings) • Staff evaluate the effects of their services • Services purchased from agencies have been shown via critical tests to help clients Gambrill(2005)

43. In Summary, cont. • Administration and management • Needed tools are provided (such as access to relevant databases) • Needed training is provided (e.g. in posing well-structured questions) • Needed feedback is provided (including supervisory) to maintain and enhance skills • A system is in place to harvest errors that occur so avoidable ones can be minimized • A user friendly complaint system is in effect and data gathered are used to improve services • Incentive systems are in place that support EBP Gambrill(2005)

44. The Road Ahead • California has an opportunity to take a leadership role: • Involve line staff and clients in identifying vital questions and outcomes and involving them as informed participants. • Prepare systematic reviews related to key questions of direct concern to line staff and clients using Cochrane and Campbell protocols. • Avoid human service propaganda such as bogus claims of effectiveness. • Create a workforce which is skilled in the process of EBP. Gambrill (2005)

45. The Road Ahead, cont. • Create organizational cultures that support EBP (e.g. provide access to needed databases) • Maximize opportunities to honor ethical obligations • To help clients • To avoid harm • To involve clients as informed participants • To maximize self-determination/autonomy Gambrill (2005)

46. The Road Ahead, cont. • Many challenges, especially a preference for authority-based practice (The “trust me” approach) • Many exciting advances, especially in options for integrating research and practice at the line staff level and honoring ethical obligations to clients. Gambrill (2005)