Collaborative Program Evaluation to Bridge Theory into Practice

Collaborative Program Evaluation to Bridge Theory into Practice Mary Little, Martha Lue-Stewart, Suzanne Martin , and Jillian Gourwitz University of Central Florida July, 2013 Recruitment - Mentoring - Technology BRIDGE Preservice * Initial Teacher * Professional Development

Abstract Developing data systems that continuously measure student and program success by university and school district educators have the potential to not only bridge the research to practice gap, but also inform programmatic decision-making. Our purpose is to discuss data-based decision making processes among key stakeholders of a 325T project.

Critical Questions The relevant questions posed to the audience include: • How do we actualize our action plans and logic models through implementation, evaluation, and continuous improvement? • How do we bridge the research-to-practice continuum for evidence-based instruction, assessment, and program evaluation? • What procedures, practices, and strategies facilitate continuous, collaborative program evaluation focused on student impact?

Purpose of 325T Project-University of Central Florida To enhance existing, approved initial certification programs to prepare highly-qualified teachers (HQT) of students with high incidence disabilities (HID) within the Masters program. The major content enhancements are: • math and science instruction for students with HID • including Culturally and Linguistically Diverse (CLD) students in co-taught and resource settings • within urban middle schools that have not met Adequate Yearly Progress (AYP) targets.

325T Project Goals-UCF • Improve recruitment to increase the number of highly qualified teachers in ESE, especially from diverse backgrounds; • Enhance and implement an effective preparation program focusing on math and science content instruction at the middle school level within the ESE Masters and certification to assure HQT status and skills to successfully teach in urban settings using culturally responsive teaching (CRT) practices; • Improve mentoring and induction; and, • Implement technology to support overall program effectively.

Establish Bridge Program Team UCF ESE Students prepared to use technology in classroom Increase in teachers’ understanding of math/science concepts Recruit ESE Graduate Students & Gather Baseline Data UCF ESE Students prepared to teach ESE students in Math/Science Increased knowledge of math/science content Assemble Stakeholders & Solicit Input Dissemination& Presentations Model Exceptional Education Graduate Program (multiple pathways-content and delivery) Increase in teachers’ confidence integrating technology in classroom UCF Faculty, Bridges Program Staff & UCF Students Increase in the number of HQT, knowledgeable of research-based practices Bridges Logic ModelResources Activities Outputs Outcomes Impacts Middle/High schools-Districts HQ/NBCT Teachers, Students, Counselors Parents Curriculum modules aligned with state/federal standards for HQT Increase the knowledge and skills in math and science of teachers Review and Revise ESE Graduate Curriculum – Integrate New Technologies State and Community Partners – FLDOE, CEC, NMTE, NSTE, Industry, Volunteers Increase the retention of highly qualified teachers into schools USDOE Cooperative Funding & IRIS, FLDOE and UCF Resources Improved Mentoring and Induction Activities Establish data collection system to analyze/revise program Technology: Recruitment, Instruction, Coaching/ Mentoring (especially CLD) Increase in the technology for program communication and improvement Technology -Lab facilities & equipment FLDOE-ATEN Develop Virtual Activities – Mentoring, Instruction, Collaboration Effective use of technology to support overall program effectively

ACCOMPLISHMENTS TO DATE GOAL1:Improve recruitment: Program applicants increased almost 200% in initial year due to strategic marketing, with increases in diversity. GOAL 2:Enhance and implement an effective preparation program focusing on math and science: M.Ed. includes Mathematics endorsement courses, co-teaching/co-planning experiences in content areas, infusion of evidence-based practices, fully on-line delivery. GOAL 3: Improve mentoring and induction; Currently piloting national e-Mentoring program while collaborating with area school districts to enhance current mentoring activities. GOAL 4: Implement technology to support overall program effectively. Synchronous and asynchronous opportunities enhance recruitment, advisement, course instruction and delivery, mentoring, and communication.

Guiding Question How do we enhance, co-create, and sustain a system of school improvement focused on improving P-12 student results by our teacher candidates that meet the current policies, mandates, and expectations for use of data, continuous progress monitoring, and evidence-based instructional and intervention practices within schools and classrooms?

Changing Focus about Data THAT WAS THEN…… • Decisions were based on judgments of the people in authority. • Multiple data sources were not readily available. • Assessment and program evaluation measures were inputs to be counted, rather than impacts to be measured. THIS IS NOW…… • In this “Information Age”, there is an exponential increase in assessments and data sources. • Accountability is publicly mandated and reported. • However, educators often feel woefully underprepared to engage in data-based problem solving. • Earl & Katz, 2006

Accountability through Informed Professional Judgment (Barber, 2001)

Thinking Differently about Data “Leveraging this data-systems investment into sustainable gains in student achievement requires a parallel investment in strengthening the capacity to use data to improve teaching and learning. Without this capacity, student achievement will continue to stagnate, and we can all expect the next flurry of reports indicating that America’s schools are falling short.” Slotnik & Orland, 2012, pg.13

The Importance of Good Data

Questions about Data • How do you know it is an issue to address? • What do you need to know to tackle the issue? • What are good data? • What data sources do you have? • What else do you need to know? • What other data sources do you need? • Who has the knowledge and expertise to develop, implementation, evaluation, and continuous improve program evaluation systems? • How will the data collection system be sustained?

Goal 1: Improve Recruitment and Retention, especially CLD BARRIERS SOLUTIONS • Time to identify all sources of data related to the identified issue and goal. • Data systems not coordinated by system, office, questions, etc. • Data re: specific question not collected (CLD status) • Knowledge re: issues and data request not understood. • Identified all resources, sources of data • Collected all data in one uniform set of data for program area. • Collaboration among PIs and external evaluator to develop, implement, and report data re: our question. • Developed our procedures as not part of system-wide data collection

UCFExceptional Education Programs

Exceptional Students Education homepage

Bridging the gap. How do we, in universities, improve P-12 student learning? How do we bridge the research-to-practice continuum for evidence-based instruction, assessment, and program evaluation? .

Collaborative Partners for Program Evaluation

Collaborative Partners • IHE-based • External Evaluators • Doctoral Researchers • Doctoral Students (Ph.D. and Ed. D.) in Special Education (Program Planning and Survey Development) • Student Services Personnel • Alumni Staff • Faculty across Colleges • Director of Accreditation • Data-Systems Personnel • District/State-based • Educator-in-Residence • Advisory Committees • Doctoral Students (Ed.D.) • District Administrators • Other 325 T projects • FDOE personnel • School district teachers (former UCF graduates)

Goal 2:Implement EBPs to Impact P-12 Student Learning BARRIERS SOLUTIONS • Time to identify all sources of data across multiple conditions. • Identification of EBPs (isolation; then, integration) • P-12 data systems not coordinated / not available by system, office, questions, etc. • Innovation Configurations (ICs) developed to Internship • Identified all resources, sources of data • NCIPP, other 325T shared EBPs; UCF faculty retreat. • Collaboration among PIs, UCF accreditation, and district personnel to develop, implement, and report data re: our question. • Developed pilot ICs in mathematics and science and extended to P-12 settings.

Overview of Professional Learning and DevelopmentUCF-Masters of Arts Graduate Program UCF Conceptual Framework of a Reflective Practitioner is Foundation of Program.

Overview of Professional Learning and DevelopmentUCF-Masters of Arts Graduate Program-DRAFT 1.18.12

From Program Evaluation to Significance of Research The goal of this current research project within our 325T grant research is to study the knowledge, beliefs, and implementation of evidence-based instructional practices in mathematics related to culturally responsive teaching by graduates of a re-designed teacher preparation graduate program in a large urban university.

Research Questions This research is guided by the following questions: -To what degree are preservice teachers’ beliefs and practices impacted by our work related to CRT? -To what degree and quality are evidence-based practices in mathematics implemented with fidelity in urban middle school classrooms to met instructional goals for all students? -To what degree are graduates knowledgeable in EBP in Response to Intervention within the College of Education? -To what degree are graduates knowledgeable and meeting requirements as a HQ teacher with improved achievement for students with and without disabilities?

Methodology This research study is currently conducted in phases: • Research and program re-design to include and enhance evidence-based practices in mathematics, Response to Intervention, and culturally-relevant teaching practices within preservice courses; • Research and develop logic model for program evaluation, using Concerns-Based Adoption Model (Hord et al., 2006) and Innovation Configuration (Reschly et al., 2009); • Complete research using mixed methods research design within middle schools in a large urban school district; and • Evaluate and disseminate findings related to the final goal leading to the enhanced learning outcomes of students with racial, cultural, ethnic, and linguistic diversity (RCELD) in mathematics.

Explicitly identify EBPs (Regan, 2010) Collected evidence for practices in areas of: Math* Content Learning: History Content Learning: Science* Foundational Reading Secondary Reading Inclusionary Practices Assessment Practices Behavior Practices Writing Instruction

Degree of EBP Knowledge and Use: 45 Evidence-Based Practices identified across Literacy, Instructional Strategies, Inclusionary Practices, Assessment, and Behavior. EBPs then ‘fit into’ the relevant course (s) Course leads of the relevant licensure classes rated the degree of coverage for EBPs: NOTE: adapted scale from Innovation Configuration Tools out of http://www.ncctq.org

Methodology-EBP-Math EVIDENCE-BASED PRACTICES-MATHEMATICS • All syllabi and internships have been enhanced: EBP and CRT. • Using the meta-analyses of effective programs in mathematics conducted by Slavin, Lake, & Groff (2009) (content) and Innovations Configurations (Hord et al, 2006; Reschly et al., (2009), implementation matrices were developed. • Delphi Procedure Analyses were completed for the implementation matrices by representatives from the following groups: parents of students with RCELD, administration, general education teachers, ESE teachers, school psychologists, other instructional support staff. • Multiple sources of data collection in process.

Instruments • Pre/post surveys and Outcome measures • The Levels of Use and Implementation Matrix Survey and Observation Tools will be used with all participants to assess implementation. Both tools have described technical adequacy (e.g., validity and reliability). • Results of state certification exams will be collected (mathematics and exceptional student education) • Review of instructional artifacts / course will be reviewed for evidence of EBP in mathematics and use of CRT practices. • Surveys created to meet goals of Program Evaluations. • Data from university and college outcome data sources have been targeted for collection and aggregation to provide data re: • Satisfaction surveys (first year teachers and principals) • Re-hire surveys (principals)

Sample-IC Rubric-Mathematics

Initial Results-2012 90% of all ESE Graduate interns implementing EBP in content as researched during courses and field placements. 100% of all ESE Graduating students were “satisfied” or “very satisfied” with preparation in all areas of Florida’s Educator Accomplished Practices upon completion of program in year 1 of teaching. Increases of Graduate students passing middle grades math certification test noted. Principal re-hire data reported a 100% rehire rate. Results from RTI Research across COE. What procedures, practices, and strategies facilitate continuous, collaborative program evaluation focused on student impact?

Multiple Opportunities for Performance Learning for Student Impact • Applied action research in multiple courses • EEX 6612: Behavior Change Projects (FBA I) • EEX 6107: Action Research (I) • EEX 6946: Behavior Change Projects and Action Research within Clinical Experiences (FBA II and AR II) • Student impact data collected as part of Graduate Internship “Teacher Work Sample” (M.A.) • Case studies on video dissemination (with permission) • Graduate student artifacts included into Livetext accounts (B.S., M.A., and M.Ed.) • Externally-funded grants

Candidates' Impact on P 12 Student Learning

K-12 Student Impact: Results from Action Research: EEX 6107 Semester: Fall 2011 Number of Candidates: 24 *Grades based upon criteria from rubrics entered into livetext system. The following narratives summarize characteristics from all action research k-12 student impact data including: research question, student grade level(s), Number of students, content Area, description of students, Intervention/assessments, results, and student grades* Grade distribution percentages 2011 *Computed percentages based upon project completions As compared with rubric for action research projects*

Action Research • EEX 6612 Behavior Change Project: 2008 to 2011 • N=298 University Students Participating • Data reflect Percent of Non-Overlapping Data points on Single Case Designs for K-12 Children

Disaggregated Effects • The majority of students demonstrate highly effective outcomes for K-12 children on problem behavior. • These data are similar to Cohen’s D.

Goal 3: Improve Retentionusing E-mentoring BARRIERS SOLUTIONS • Data systems not coordinated by system, office, questions, etc. • Data re: specific question not collected (retention, either within our institution or after graduating) • Difficult to track graduates as “official” email discontinued as of graduation. • Identified all resources, sources of data • Retention data within programs at UCF now being collected. • Collaboration among school district teachers, Technology consultants, and district partners to develop, implement, and report data re: our question. • Developed our procedures as not part of system-wide data collection BEFORE graduation.

Goal 4:Use of Technology BARRIERS SOLUTIONS • Time to identify all sources of data related to the identified issue and goal. • Data systems not coordinated by system, office, questions, etc. • Data re: specific question not collected (Use of technology) • Collected all data in one uniform set of data for program area. • Collaboration among PIs and external evaluator to develop, implement, and report data re: our question. • Developed our procedures as not part of system-wide data collection.

Student Preference for Use of Technologies

Next Steps Continue data collection, analyses, and reporting to multiple accrediting agencies; Continue development of collaborative school sites; Finalize all instrumentation, designs, methods, etc. Complete research and OSEP Final Report; Disseminate findings to multiple stakeholders (e.g., ESE Program faculty, CED, OSEP, and others) Extend process to additional content areas and schools to enhance and sustain process, and Comprehensively align entire Graduate Program to meet multiple accountability requirements and to assure sustainability after grant funding ends.

Comprehensive Program Evaluation

Opportunities for Special Education “Schools require a leveraging system that targets resources toward children at vulnerable moments; giving special educators a major role in this new system can redirect their efforts and create new roles for them as mediators and collaborators, who working with others, facilitate children’s entry (and successes) in the school community.”

Teacher Preparation Robust teacher preparation for special education teachers in the educational curriculum and working experiential knowledge of the best classroom pedagogies develop special education teachers that are well-prepared to work alongside general educators to assure that general education contexts become places where the majority of our children has strong, successful educational experiences. Their unique, explicit work as special educators would follow from this rich base of teacher preparation. • Seidl, B. & Pugach, M. (2009). Support and teaching in the vulnerable moments: Preparing special educators for diversity. Multiple voices, 11(2), Spring, 2009.

References Cross, B. (2003). Learning or unlearning racism: Transferring teacher education curriculum to classroom practices. Theory into Practice, 42(3), 203-209. Fieldler, C.R., Chiang, B., Van Haren, B., Jorgensen, J., Halberg, S. & Bereson, L. (2008). Culturally responsive practices in schools: A checklist to address disproportionality in special education. Teaching Exceptional Children (40)5, 52-59. Freire, P. (2007/1997). Pedagogy of the oppressed. New York: The continuum international publishing group, Inc. Freire, P. (2007/1997). Human rights and liberating education. In A. M. A. F. (Ed.), Daring to dream: Toward a Pedagogy of the Unfinished. (pp. 57-71). Boulder: Paradigm Publishers. Garcia, S. & Guerra, C. (2004). Deconstructing deficit thinking: Working with educators to create more equitable learning environments. Education and Urban Society, 36(2), 150-168. Gardner, D. (2007). Confronting the achievement gap in In K. Ryan & J. M. Cooper (eds.) Kaleidoscope: Contemporary and Classic Readings in Education. (pp. 154-159). Belmont, CA: Wadsworth, Cengage Learning. Gay, G. (2000). Culturally Responsive Teaching: Theory, Research, & Practice. New York: Teachers College. Hall, S. M., Stiegelgelbauer, S.M., Hall, G. E., & George, A.A. (2006). Measuring implementation in schools: Innovation configurations. Southwest Educational Development laboratory. Univeristy of Texas at Austin; Author. Regan, K. (2010.) Special Education Pre-Service Training Improvement ProjectImproving Course Content and Syllabi Revision. Office of Special Education Programs. Washington, DC: Author. Reschly, D., Oliver, R., Schumaker, J., Smartt, S., & Holdheide, L. (2009). Innovation configurations: Implementing evidence-based practices in teacher preparation. Retrieved from the National Comprehensive Center for Teacher Quality: http://www.tqsource.org Siedl, B. & Pugach, M. (2009). Support and Teaching in the Vulnerable Moments: Preparing Special Educators for Diversity, Multiple Voices, 11(2), Spring 2009 Slavin, R.E., lake, C., & Groff, C. (2009). Effective programs in middle and high school mathematics: A best-evidence synthesis. Review of Educational Research, 79(2), 839-911.

Collaborative Program Evaluation to Bridge Theory into Practice