Bridging the Research-Practice Gap: Teacher Preparation and Evidence-based Education

1. Bridging the Research-Practice Gap: Teacher Preparation and Evidence-based Education Ronnie Detrich Wing Institute

2. Goals for Today • Discuss basic characteristics and processes of evidence-based education. • Review the legal and ethical basis for evidence-based education. • Highlight issues for higher education.

3. October 1957 USSR launched Sputnik. U. S. Education quickly blamed. Modern reform efforts began.

4. 1983 A Nation at Risk American students not performing well. Education quickly blamed. The Nations Report Card created.

5. 1994 Goals 2000 All students will start school ready to learn. High school graduation rate ≥ 90%. All students in grades 4, 8, & 12 will demonstrate competency in challenging subjects.

6. 2001 No Child Left Behind By 2014 every student will be at grade level. Instructional methods will be scientifically based. Educators will be held accountable for outcomes.

7. Everybody’s Talking • Teacher accountability often offered as solution for education troubles. • Numerous proposals to pay and evaluate teachers based on student performance. • Proposals usually linked to performance on high stakes tests.

8. Function of Accountability Systems • Assure that all students are benefitting from educational services. • Student performance primary source of feedback to educators about effects of interventions.

9. Scope of the Problem Research to Practice Gap or Chasm? • Gap concern in many disciplines. • Education is not excluded. • Scientist/Practitioner model aimed to close the gap.

10. Scurvy in the British Royal Navy: An Example of the Research to Practice Gap John Lind again experimentally demonstrated the effectiveness of citrus in preventing scurvy. James Lancaster first experiment demonstrating how to prevent scurvy. British Navy adopted policy to have citrus on all ships in the Royal Navy. 1601 1747 1795

11. Research to Practice Issues • The lag time from efficacy research to effectiveness research to dissemination is 10-20 years. (Hoagwood, Burns & Weisz, 2002) • Only 4 of 10 Blueprint Violence Prevention programs had the capacity to disseminate to 10+ sites in a year. (Elliott & Mihalic, 2004)

12. How Big is the Gap? 550 named interventions for children and adolescents Kazdin (2000) Empirically evaluated Cognitive-behavioral Behavioral Evidence-based interventions are less likely to be used than interventions for which there is no evidence or there is evidence about lack of impact.

13. Goals for Evidence-based Practice in Education • At its core the EBP movement is a consumer protection movement. • It is not about science per se. • It is a policy to use science for the benefit of consumers. • “The ultimate goal of the ‘evidence-based movement’ is to make better use of research findings in typical service settings, to benefit consumers and society….”(Fixsen, 2008)

14. How is Education to Increase the Use of Evidence-based Practices? • No Child Left Behind (NCLB) requires that interventions used to improve educational performance are based on scientific research. • In NCLB there are over 100 references to scientific research. • Individuals with Disabilities Education Improvement Act [IDEIA] (2004) requires interventions that are scientifically based instructional practices.

15. How is Education to Increase the Use of Evidence-based Practices? • Specific requirements of IDEIA include: • Pre-service and professional development for all who work with students with disabilities to ensure such personnel have the skills and knowledge necessary to improve the academic achievement and functional performance of children with disabilities, including the use of scientifically based instructional practices, to the maximum extent possible.

16. How is Education to Increase the Use of Evidence-based Practices? • Scientifically based early reading programs, positive behavioral interventions and supports, and early intervention services to reduce the need to label children as disabled in order to address the learning and behavioral needs of such children.

17. How is Education to Increase the Use of Evidence-based Practices? • The Individualized Education Program (IEP) shall include a statement of the special education and related services and supplementary aids and services, based on peer-reviewed research to the extent practicable,to be provided to the child, or on behalf of the child, and a statement of the program modifications or supports for school personnel that will be provided for the child.

18. How is Education to Increase the Use of Evidence-based Practices? • In determining if a child has a specific learning disability, a local education agency may use a process that determines if a child responds to a scientific, research-based intervention as part of the evaluation procedures.

19. The Ethical Basis • Most national psychological and educational organizations have ethical standards requiring science-based practices to address problems. • American Psychological Association Ethical Standard 2.04: • Psychologists’ work is based on the established scientific and professional knowledge of the discipline.

20. The Ethical Basis • National Association of School Psychologists • Standard III F 4. • School psychology faculty members and clinical or field supervisors uphold recognized standards of the profession by providing training related to high quality, responsible, and research-based school psychology services.

21. The Ethical Basis • National Association of School Psychologists • Standard IV 4. • School psychologists use assessment techniques, counseling and therapy procedures, consultation techniques, and other direct and indirect service methods that the profession considers to be responsible, research-based practice.

22. The Ethical Basis • Behavior Analyst Certification Board • Standard 2.09a • The behavior analyst always has the responsibility to recommend scientifically supported, most effective treatment procedures. Effective treatment procedures have been validated as having both long-term and short-term benefits to clients and society. • Standard 2.09b • Clients have a right to effective treatment (i.e., based on the research literature and adapted to the individual client).

23. Another Ethical Responsibility • Education services are largely funded through public dollars (taxpayers dollars). • There is an implicit assumption that the money will be spent for the public good. • A fiduciary responsibility exists when one person or organization is charged with managing another person’s money.

24. Another Ethical Responsibility • The primary responsibility of a fiduciary is to act solely for the benefit of the other party. • Being a fiduciary carries the weight of ethical conduct. • Those of us charged with providing education services have a fiduciary responsibility to assure that the taxpayers are receiving the greatest possible return on their investment.

25. How Do We Meet Our Fiduciary Responsibility? • Interventions that have an evidence base are more likely to produce positive effects for students. • Does not assure positive outcomes but increases the probability.

26. How Do We Meet our Fiduciary Responsibility? • The impact of a non-evidence based intervention is unknown. • Using a non-evidence-based intervention when evidence-based interventions available may constitute unethical practice. • Use a non-evidence based intervention should be considered research • All of the safe-guards afforded research participants and their families should be in place. • Conducting research with tax-dollars provided for education services may constitute a violation of our fiduciary responsibility.

27. Becoming Evidence-based • Clearly, the intent of Congress, the U.S. Department of Education, and Office of Special Education Programs is to rely on interventions that have a scientific basis. • Professional organizations place great value on scientific knowledge. • What does it mean to be evidence-based?

28. What is Evidence-based Practice? • Evidence-based practice has its roots in medicine. • Movement has spread across major disciplines in human services: • Psychology • School Psychology • Social Work • Speech Pathology • Occupational Therapy

29. What Is Evidence-based Practice? • EBP is a decision-making approach that places emphasis on evidence to: • guide decisions about which interventions to use; • evaluate the effects of an intervention. Professional Judgment Best available evidence Client Values Sackett et al (2000) Client Values Best Available Evidence Professional Judgment

30. Phases of Evidence-based Intervention Identify Identify Evidence-based Intervention Implement Evaluate Evaluate Implement

31. Identify What is Evidence-based Education? • The term “evidence-based” has become ubiquitous in last decade. • Often used interchangeably with empirically supported and best practice. • No consensus about what it means. • At issue is what counts as evidence. • Federal definition emphasizes experimental methods. • Preference for randomized trials. • Definition has been criticized as being positivistic.

32. Identify What Counts as Evidence? • Ultimately, this depends on the question being asked. • Qualitative methods are best for answering social validity questions. • In EBP the goal is to identify causal relations between interventions and outcomes. • Experimental methods do this best.

33. Identify What Counts as Evidence? • Even if we accept causal demonstrations to be evidence, we have no consensus. • Randomized Clinical Trials (RCT) have become the “gold standard.” • There is controversy about the status of single subject designs. • Most frequently criticized on the basis of external validity. • WWC has recently established standards for SSDs . • No well established method for calculating effect sizes.

34. Identify Distinguishing Between Evidence-based and Empirically Supported • Evidence-based refers to practices that have been validated through a systematic review. • Can involve meta-analysis. • WWC reviews are systematic reviews. • Empirically supported refers to practices that have received support in peer reviewed journals. • Have not been systematically reviewed to establish strength of evidence.

35. Identify Applying Best Available Evidence Construct • If validated intervention available adopt it. • Assuming it is appropriate to context. • If no validated intervention select empirically-supported intervention. • If no empirically-supported interventions develop intervention based on principles. • Principles of scientific reading. • Principles of behavior.

36. Identify Applying EBP Framework • Professional judgment involved at all levels of selection an intervention. • Interventions must fit competencies of those implementing. • Interventions must be acceptable to consumers and fit values.

37. Identify How Are Evidence-based Interventions Identified? • Identification is more than finding a study to support an intervention. • Identification involves distilling a body of knowledge to determine the strength of evidence.

38. Identify How Are Evidence-based Interventions Identified? • Distillation requires standards of evidence for reviewing the literature. • Standards specify: • the quantity of evidence • the quality of evidence

39. Current “Gold Standard”High QualityRandomized Controlled Trial Meta-analysis (systematic review) Single Case Designs Repeated Systematic Measures Semi-Randomized Trials Single Case Replication (Direct and Parametric) Well-conducted Clinical Studies Threshold of Evidence Convergent Evidence Uncontrolled Studies Expert Opinion Various Investigations General Consensus Personal Observation Single Study Identify Continua of Evidence Quantity of the Evidence Quality of the Evidence Janet Twyman, 2007

40. Identify How Are Evidence-based Interventions Identified? • Two approaches to validating interventions • Threshold approach: • Evidence must be of a specific quantity and quality before an intervention is considered evidence-based. • What Works Clearinghouse • Meets evidence standards. • Meets evidence standards with reservations. • Does not meet standards at this time. • Hierarchy of evidence approach: • Strength of evidence falls along a continuum with each level having differential standards. • National Autism Center • Established • Emerging • Unestablished • Ineffective/Harmful

41. Identify

42. Evidence-based Intervention Evidence-based Intervention Identify

43. Identify No Agreed upon Standards Intervention X Validated Not Validated  Standard 1 Standard 2  How are consumers to decide?

44. Identify Actual Effectiveness Assessed Effectiveness Effective Ineffective Effective Ineffective Effective Effective Most likely with hierarchy approach True False Positive Positive Most likely with threshold approach False True Negative Negative Ineffective Ineffective

45. Identify Choosing Between False Positives and False Negatives • At this stage, it is better to have more false positives than false negatives. FalseNegatives: Effective interventions will not be selected for implementation. • As a consequence, less likely to determine that they are actually effective. FalsePositives: Progress monitoring will identify interventions that are not effective.

46. Role of Higher Ed in Identifying Evidence-based Practices • Conduct systematic reviews to identify effective practices. • Multiple organizations are have processes for validating interventions. • What Works Clearinghouse • Best Evidence Encyclopedia • Campbell Collaboration • Center for Evidence-based Policy • National Autism Center • Promising Practices Network • CEC developing standards for review.

47. Role of Higher Ed in Identifying Evidence-based Practices • Independent reviews for journals: • Evidence-based Communication Assessment and Intervention • Browder, et al., 2006: Research on Reading Instruction for Individuals with Significant Cognitive Disabilities, Exceptional Children. • Expand dissemination efforts. • Specify “audience.” • Different audiences may require different methods. • Expand methods.

48. Diffusion of InnovationRogers, Diffusion of Innovation, 2003 • Diffusion of innovation is a social process, even more than a technical matter. • The adoption rate of innovation is a function of its compatibility with the values, beliefs, and past experiences of the individuals in the social system.

49. Principles for Effective Diffusion:Improving the Odds (Rogers, 2003) • Innovation has to solve a problem that is important for the “client.” • Innovation must have relative advantage over current practice. • It is necessary to gain support of the opinion leaders if adoption is to reach critical mass and become self-sustaining. • Innovation must be compatible with existing values, experiences and needs of the community.

50. Principles of Effective Diffusion:Improving the Odds • Innovation is perceived as being simple to understand and implement. • Innovation can be implemented on a limited basis prior to broad scale adoption. • Results of the innovation are observable to others.