IES Grant Writing Workshop for Efficacy and Replication Projects

1. IES Grant Writing Workshop for Efficacy and Replication Projects Joan E. McLaughlin, Ph.D. Deputy Commissioner National Center for Special Education Research

2. Overview of Presentation • Structure of the Institute of Education Sciences • Research Topics within NCSER & NCER • Goals within Research Topics • Research Narrative for Goal 3 applications • Preparing and Submitting an Application • What Happens Next? • Questions

3. Structure of the Institute of Education Sciences

4. IES Organizational Structure Office of the Director National Board for Education Sciences National Center for Education Research National Center for Education Statistics National Center for Education Evaluation National Center for Special Education Research

5. FY 2012 Research Grant Topics in NCSER and NCER

6. Special Education Research Topics (84.324A) • Early Intervention and Early Learning in Special Education • Reading, Writing, and Language Development • Mathematics and Science Education • Social and Behavioral Outcomes to Support Learning • Transition Outcomes for Special Education Secondary Students • Cognition and Student Learning in Special Education • Professional Development for Teachers and Related Service Providers • Special Education Policy, Finance, and Systems • Autism Spectrum Disorders • Technology for Special Education • Families with Children with Disabilities

7. Education Research Topics (84.305A) • Reading and Writing • Mathematics and Science Education • Cognition and Student Learning • Social and Behavioral Context for Academic Learning • Education Technology • Effective Teachers and Effective Teaching • Improving Education Systems: Policies, Organization, Management, and Leadership • Postsecondary and Adult Education • Early Learning Programs and Policies • English Learners

8. Research Goals within Topics

9. Solution Driven Research Research intended to contribute to the solution of practical education problems: • Exploration Goal One • Development and Innovation Goal Two • Efficacy and Replication Goal Three • Scale-up Goal Four • Measurement Goal Five

10. Exploration • Explore the association between malleable factors and education outcomes • A malleable factor can be changed by the education system be it a characteristic of students (e.g., skills, behaviors), teachers (e.g., credentials, practices) or school (e.g., climate, size), or an education program or policy • Underlying processes that enhance or inhibit learning • Aspects of a school, district,or community associated with beneficial education outcomes • Education interventions associated with beneficial education outcomes (e.g., professional development, curricula, policies) • Explore factors that mediate or moderate the relationship between malleable factors and student outcomes

11. Development and Innovation • Develop new interventions (e.g., instructional practices, curricula, teacher professional development) • Demonstrate the feasibility of the intervention for implementation in an authentic education delivery setting • Collect pilot data on promise of intervention to achieve intended outcomes

12. Efficacy and Replication • Causal test of whether or not a fully developed intervention has a beneficial impact on student outcomes relative to a counterfactual in an authentic educational setting • Takes place under “ideal” conditions • Homogenous sample of students/schools • Extra assistance to support high implementation fidelity

13. Scale-up Evaluation • Independent causal test of whether or not a fully developed efficacious intervention has a beneficial impact on student outcomes relative to a counterfactual in an authentic educational setting under routine implementation • Independent: evaluation team has no financial interest in intervention • Efficacious: evidence of intervention’s efficacy • Routine implementation: as implemented by practitioners with expected level of support if adopted by a school or district

14. Goal 5: Measurement Develop and validate assessments or other measurement tools

15. Overview of Efficacy and Replication Grant Applications

16. Efficacy • Efficacy is the degree to which an intervention has a net positive impact on the outcomes of interest relative to the program/practice to which it is being compared

17. Replication • Replication projects determine if an intervention that has produced a positive impact under one set of conditions will produce a positive impact under different conditions (e.g., the same intervention with different populations of students, different educational settings, or different levels of support)

18. Purpose of Efficacy and Replication Projects • To estimate the strength or potency of the impact of the intervention • To inform the degree to which the intervention can be feasibly or practically implemented • To assess implementation fidelity

19. Appropriate Interventions for Efficacy and Replication Projects • Interventions that are widely used but have not been rigorously tested • Interventions that are fully developed, have evidence of their feasibility and promise, but are not widely used • Interventions that have a positive effect under one set of conditions

20. Efficacy and Replication Projects • Use experimental or quasi-experimental research designs that minimize selection bias or allow it to be modeled • Employ a limited set of conditions • Often involve more support from the intervention developer/researcher than is typically available

21. What percentage of research grants are Efficacy and Replication projects? • 22 to 23 percent of the research projects funded by IES are Efficacy and Replication projects

22. Efficacy and Replication Applications Research Narrative

23. Components of the Research Narrative • Significance • Research Plan • Personnel • Resources

24. Significance • Describe fully-developed intervention • Provide rationale for why testing the efficacy of this intervention is important • Present a clear theory of change to guide the understanding of the intervention and expected outcomes

25. Research Plan (1) • Clearly articulate all research questions • Does the intervention lead to a meaningful improvement in education outcomes? • What are the conditions that support or hinder the implementation of the intervention? • What processes mediate the relation between the intervention and student outcomes?

26. Research Plan (2) • Define sample (include size requirements, exclusion rules, and discussion of attrition issues)

27. Research Plan (3) • Research Design • Use a randomized controlled trial (RCT) whenever possible • Clearly justify the use a quasi-experimental design and address internal validity threats

28. Research Plan (4) • Describe the power analysis - Provide rationale for assumptions - If subgroups are integral to your study, consider whether you have sufficient power to address the relevant research questions

29. Research Plan (5) • Provide detailed description of the chosen measures (include information on appropriateness, reliability, validity) • Don’t forget to detail any researcher-developed measures • Provide clear description of data collection procedures and timing of data collections

30. Research Plan (6) • Describe how fidelity will be assessed • Clearly describe the comparison condition • Identify mediating and moderating variables and how they will be collected • Detail the plan for analysis of qualitative and quantitative data

31. Personnel • Content domain experts • Methodological expertise • Statistical expert • Staff experienced in conducting research in chosen setting and with population

32. Personnel (2) • Qualifications • Roles and Responsibilities • Percent of time devoted to the project • DON’T FORGET to address the issue of the objectivity of the evaluation if the intervention developer is part of the team

33. Resources • Address institutional resources • Document the support of those in the research setting (e.g., letters from school districts or schools, preschools)

34. Efficacy and Replication Typical award: $250,000 to $650,000 per year Maximum of 4 years and $3,500,000

35. Efficacy Follow-up Studies • Examine the sustainability of the impacts of an intervention after the original study on: - students that received the intervention and have moved on - a new group of students in the same setting when additional resources are no longer provided for the intervention

36. Efficacy Follow-up Studies • Emphasis in narrative on previous study, including sample and effects • Typical award: 150,000 to $300,000 per year • Maximum of 3 years and $1,200,000

37. Preparing and Submitting an Application

38. Getting Started • Request for Applications • IES Grants.gov Application Submission Guide • ApplicationPackage

39. Relevant Requests for Applications (RFAs) • Education Research Grant Program (84.305A) • Special Education Research Grant Program (84.324A)

40. Documents on the IES Website FY 2012 Requests for Applications and the Grant Submission Guide are available on: http://ies.ed.gov/funding Sign up for the IES Newsflash: http://ies.ed.gov/newsflash/

41. http://ies.ed.gov

42. Finding Application Packages • FY 2012 Application Packages are available on www.grants.gov • June Application Package available on April 21, 2011 • September Application Package available on July 21, 2011

44. Letters of Intent • For June competition, due 4/21/2011 • For September competition, due 7/21/2011 • Non-binding, but strongly encouraged • Submitted electronically • Instructions at iesreview.ed.gov • Receipt via e-mail

45. IES assistance • Read the Request for Applications carefully. • Call or email IES program officers early in the process. • As they have time, IES program staff will review draft proposals and provide feedback to applicants.

46. Follow Directions and Attend to Details Read Program Announcements and Application Instructions carefully • Eligible Applicants • Special Requirements • Content and Formatting Requirements • Application Submission and Processing • Application Due Dates

47. Application Due Dates in 2011 • Education and Special Education Research: June 23 and September 22 At 4:30:00 p.m. (EASTERN)

48. What Happens Next?

49. Peer Review • Proposal is reviewed for compliance. • Compliant proposals are assigned to a review panel. • For Goal 3, three panel members conduct primary review of each application. • Most competitive applications are reviewed by full panel at the panel meeting.

50. Notification • All applicants will receive email notification of the status of their application. • All applicants receive copies of reviewer comments. • Applicants who do not receive an award should consider revising and resubmitting their application. Applicants are encouraged to contact the program officer about the resubmission.