James Benson, Ph.D. Allen Ruby, Ph.D. National Center for Education Research

1. Continuous Improvement Research in Education ANDEvaluation of State & Local Education Programs & Policies(Topics 2 & 3 of 84.305H) James Benson, Ph.D. Allen Ruby, Ph.D. National Center for Education Research

2. Overview • Overview of IES and its mission • Requirements for both topics • Specifics for each topic • Purpose • The project narrative • Significance • Partnership • Research Plan • The project narrative • Personnel and Resources • Other important sections of the application • Preparing and submitting an application

3. Legislative Mission of IES Describe the condition and progress of education in the United States Identify education practices that improve academic achievement and access to education opportunities Evaluate the effectiveness of Federal and other education programs

4. Organizational Structure of IES Office of the Director National Board for Education Sciences Standards & Review Office National Center for Education Evaluation National Center for Education Research National Center for Education Statistics National Center for Special Education Research

5. IES Grant Programs: Research Objectives • Develop or identify education interventions (i.e., practices, programs, policies, and approaches) • that enhance academic achievement • that can be widely deployed • Identify what does not work and thereby encourage innovation and further research • Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

6. Partnerships & IES Priorities IES seeks to... Encourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies (See http://ies.ed.gov/director/board/priorities.asp)

7. Partnerships & IES Research Grant Programs • Partnerships and Collaborations Focused on Problems of Practice and Policy (84.305H) • To further promote research partnerships between research institutions and State and local education agencies (SEAs/LEAs) • Contains 3 topics • Researcher-Practitioner Partnerships in Education Research • Continuous Improvement Research in Education • Evaluation of State & Local Education Programs & Policies

8. The Two Topics • Continuous Improvement Research in Education (Continuous Improvement) • Well-established partnerships • Goal: To adapt and revise a specific approach, using a continuous improvement strategy, to address a specific education issue or problem of high importance to the education agency that has important implications for improving student education outcomes • Evaluation of State & Local Education Programs & Policies (State/Local Evaluation) • New or established partnerships • Goal: To carry out rigorous evaluations of education programs or policies (programs/policies) that are implemented by state or local education agencies and have important implications for improving student education outcomes

9. Requirements Shared by Both Topics Focus on student education outcomes Partnership between research institutions and SEAs/LEAs

10. Focus on Student Outcomes • IES funds research to improve the quality of education for all students through advancing the understanding of and practices for teaching, learning, and organizing education systems • All research must address education outcomes of students • Academic outcomes • Social and behavioral competencies that support student success in school

11. Student Population • Students from prekindergarten through postsecondary and adult education • Typically developing students • Students with disabilities or at risk for disabilities • Specific requirements for identifying students at risk for disabilities status • see http://ies.ed.gov/ncser/definition.asp

12. Ultimate Outcomes of Interest: Student Outcomes

13. Ultimate Outcomes of Interest: Student Outcomes

14. Applications must be from a Partnership • Partnership must include at least a research institution and a U.S. education agency • Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from an SEA or LEA • PI from research institution:Must have the ability and capacity to conduct scientifically valid research and expertise in the education issue to be addressed • PI from SEA or LEA:Must have decision-making authority for the education issue within his or her agency

15. Partnership • Length of Partnership • Continuous Improvement • document at least 1 year of collaboration and describe products • State/Local Evaluation • may be new or existing partnership • Broad definition of research institution • Ability and capacity to conduct scientifically valid research

16. Partnership: SEA Partner • State education agencies • Examples: education agencies, departments, boards, commissions • Oversee early learning, elementary, secondary, postsecondary/higher, and adult education • Includes education agencies in U.S. territories and tribal education agencies

17. Partnership: LEA Partner • Local education agencies which are primarily public school districts • Community college districts • State and city postsecondary systems • If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner • If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner • A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project

18. Additional Partners Partnerships may include more than one State or local education agency if they share similarities and interests Non-education state and local agencies may be partners as long as an education agency is a partner Partnerships may include more than one research institution if they have shared interests and will make unique contributions Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work

19. Quick Check Check the fit between your research and the topic! • Just because you have a partnership, doesn’t mean the Continuous Improvement topic or the State/Local Evaluation topic is the best grant topic for you • Potentially better fit • Education Research Grants Program (84.305A) or • Special Education Research Grants Program (84.324A)

20. Specifics about the Topics Continuous Improvement Research in Education Evaluation of State & Local Education Programs & Policies

21. Continuous Improvement: General Purpose • Promote joint research by partnerships of research institutions and SEAs/LEAs • Addresses an education issue or problem of key importance to an SEA/LEA • Directly contributes to solving problems faced by an SEA/LEA

22. Continuous Improvement: Specific Purpose Implement, adapt, and revise an educational approach to address the education issue or problem of concern to the SEA/LEA, with the aim of improving student outcomes Increase the agency’s capacity to carry out research, development, and implementation Contribute to our understanding of how approaches can be adopted to address local conditions and wide implementation

23. Continuous Improvement: Expected Products of the Grant Description of the partnership as it developed during the grant Description of the approach in use by the end of the project Description of the process of continuous improvement used to adapt/revise the approach and the measures used in that process Results from an ongoing comparison of student outcomes in sites (e.g., schools/classrooms) where the approach is being adapted and revised, compared to sites that are not trying to implement the approach

24. Continuous Improvement: Expected Products of the Grant Recommendations for how the partnership could be maintained over the longer term Specific and general lessons from the revisions to the approach and changes made in the education system that improved the approach and its implementation Lessons learned from the joint development work performed by the partnership that could benefit other partnerships

25. Education Issue and Approach Applicants may propose to address any education issue or problem of priority to the LEA/SEA An approach is defined as a policy, program, intervention, practice or combination thereof that addresses a problem/issue of high importance to an education agency, and that has a strong theoretical and/or empirical rationale for improving student education outcomes.

26. The Project Narrative Significance Partnership Research Plan Personnel Resources

27. Significance In the Significance section, clearly describe… The education issue/problem The approach to be adapted/revised The education system Current practice in the education system

28. Significance • Describe the education issue to be addressed • Its links to student education outcomes • Its importance to the education agency’s decision making • Its importance to other education agencies, policymakers, and education research

29. Significance • Describe the proposed approach to be implemented, adapted, and revised in order toaddress the education issue • Describe the theory of change for the approach • Provide any empirical evidence that the approach can be successfully implemented • Provide any empirical evidence that the approach can improve student outcomes

30. Significance • Describe the education system • Describe the education system(s) where implementation will occur • E.g., classroom, school, district, multiple districts, state • Discuss why the approach will need adaptation and revision for successful implementation in this system • Discuss how the system(s) may need to change to support successful implementation

31. Significance • Describe current practice in the education agency • Describe how the education agency is currently addressing the education issue • Discuss why current practice is not satisfactory • Describe the current status of approach within the education agency

32. The Project Narrative Significance Partnership Research Plan Personnel Resources

33. Partnership In the Partnership section, clearly describe… The partnership Partnership development Tracking the partnership’s success

34. Partnership Description • Describe the partners • The research institution and the education agency • Any other members of the partnership • The partnership’s previous work (1-year minimum) and resulting products • Partners’ common interests and complementary abilities • How all members contribute to and benefit from the partnership • How the partners decided to propose a Continuous Improvement project • Management structure and procedures to keep the project on track and quality control

35. Partnership Development • Planned activities and processes to further develop the partnership • How will these activities and processes contribute to the research, agency capacity building, and future collaborations? • Partnership decision-making process • How will you determine next steps in research, dissemination, capacity building, and future research? • Building the education agency’s capacity to carry out research, development, and implementation

36. Tracking the Partnership • Monitoring the success of the partnership • During the project: Maintaining the partnership, completing the adaptation and revision of the approach, completing the ongoing comparison • After the project: Opportunities for the partnership to continue and for the agency to be more able to carry out research, development, and implementation • IES encourages projects to propose additional indicators of partnership success

37. The Project Narrative Significance Partnership Research Plan Personnel Resources

38. Research Plan In the Research Plan, clearly describe… A measurement strategy and plan for data collection The continuous improvement process An ongoing comparison study of student outcomes

39. Measurement Strategy • Describe how you will collect data and measure for: • How well the approach is functioning including its: • Usability: Can intended user physically implement the approach as well as understand it and be willing to use it? • Feasibility: Is the approach usable within the constraints of the education system? • Progress toward the desired outcomes (as set out in the theory of change) • Short-term • Intermediate • Final • What attributes of the approach and the education system need revision • Describe measure construction, including any new measures the project will need, AND how the measures will inform the improvement process

40. Continuous Improvement Process • Describe the Improvement Process • Starting Approach: how the approach will be obtained and initially implemented • Analysis Process: how collected data will be studied and interpreted to determine adaptations and revisions • Implementation Process: how the iterative revisions of the approach will be implemented • Describe the Monitoring of the Improvement Process • Infrastructure and processes to keep work on track • Identification of needed changes in the education system

41. Ongoing Comparison Study • Detail a plan for an ongoing comparison of student outcome between sites taking part in adaptation and revision of the approach and those not involved • Comparability of comparison sites • Timing of comparisons • From start of project • Not need to be every improvement cycle but should take place every semester or year) • Student outcomes to be compared (using easily available data) • Analysis plan • Comparison site practice (based on non-intensive study) • Not expected to have the rigor of a pilot study or an efficacy study (as described in 84.305A or 84.324A)

42. Specifics about the Topics Continuous Improvement Research in Education Evaluation of State & Local Education Programs & Policies

43. State & Local Evaluation: Purpose • Promote joint evaluation research by research institutions and SEAs/LEAs • On an education program/policy of key importance to SEAs/LEAs • That will directly contribute to SEA/LEA program and policy decisions • Provide opportunities to develop the partnership through the evaluation • Foster longer-term research partnerships • Provide and support the use of rigorous research-based evidence in decision making • Continue practitioner input into research agenda

44. What should the partnerships do during the grant? Broadly, the partnerships should… • Identify an education program or policy • Implemented by an SEA/LEA • Of high priority to that agency • Intended to improve student education outcomes • Carry out an evaluation of that program/policy

45. Expected Products of the Grant • Causal evidence of the impact of a clearly specified program/policy implemented by an SEA/LEA • Overall impacts • Impacts under a variety of conditions • Conclusions on and revisions to the theory of change that guides the program/policy • Contributions to our theoretical understanding of education processes and procedures

46. Expected Products of the Grant • If a beneficial impact is found… • The organizational supports, tools, and procedures needed for sufficient implementation of the core components of the program/policy under routine practice should be identified • If a beneficial impact is notfound… • A determination should be made whether and what type of further research would be useful to revise the program/policy and/or its implementation • The financial costs of the program/policy

47. The Project Narrative Significance Partnership Research Plan Personnel Resources

48. Significance In the Significance section, clearly describe… • The education program or policy to be evaluated • Components • Processes and materials to support implementation • Evidence it is ready to be or already implemented • How it differs from existing practice

49. Significance • Its implementation • By an SEA or LEA • Target population and sites • End users of the program or policy and how they are to carry it out

50. Significance • The theory of change • How the program or policy is to effect changes that ultimately lead to beneficial impacts to student outcomes • Intermediate outcomes in this process • Rationale for testing its impact on student education outcomes • In widespread use but not well-evaluated • An alternative to common practice that has a theoretical (perhaps empirical as well) justification • Improvement on previous research