Children First Focus on Accountability

1. CONFIDENTIAL Children FirstFocus on Accountability Accountability Initiative July 19, 2006

2. CONTENTS • Accountability Initiative • Vision and Overview • Achievement Reporting and Innovation System (ARIS) • Accountability Initiative Implementation Timeline

3. Number of Schools = 52 Number of Schools = 82 Number of Schools = 113 Number of Schools = 189 Number of Schools = 205 THE PROBLEM

4. THE SOLUTION: A CYCLE OF CONTINUOUS IMPROVEMENT • Accountabilities / • Measures • Leadership / clear goals • Progress Reports • Quality Reviews • SCHOOLS • Enablers / Tools • Innovation / evaluation • Continuous learning and support (e.g. Empowerment Schools Intensive) • Periodic assessments • ARIS • Adjustment

5. INNOVATIONS IN ACCOUNTABILITY Old accountability measurements New accountability measurements • Snapshot of performance • Movement only across single proficiency drivers- ignores outcomes on high/ low ends • Only grades schools with challenging populations: SINI, SURR • Incentives to focus on students least in need • Progress: what the school adds • All amounts of improvement in student achievement • Universal application: All schools are graded: A, B, C, D, F • Extra credit for gains by students most in need 4 Year Cycle of Consequences • School improvement plan and target setting • Leadership change • School restructuring • School closure

6. ACCOUNTABILITIES / MEASURES: SCATTER CHART

7. Gather data • Plan and • set goals • Monitor and revise • Align • instruction • Build and align capacity ACCOUNTABILITIES / MEASURES: QUALITY REVIEWS • Understand what each student knows and is able to do monitor, and • Measure the student’s progress over time. • Set each student’s next learning step. • Collaborate with student / parents to set goals for improving teaching practice and accelerating student learning • Update documentation / best practices on effectiveness of its improvement plans and flexibility of change processes • Continuous data-driven improvement • Align school instructional activity and resources around focused plans for accelerating individual student learning • Align instructional skills and capacity development around established goals for accelerating individual student learning

8. ENABLERS / TOOLS:CREATE A CYCLE OF CONTINUOUS LEARNING Start by analyzing school’s performance data to ID areas that need improvement Assess progress and make mid-course corrections Publish Progress Reports Align instructional practice and resources, access external resources to support change Create urgency with publicized results Identify high leverage points for intervention to improve student performance Select a point of focus Deeply informed interventions Top-down qualitative analysis from Quality Review Analyze student- level data, instructional practices, and adult learning needs Bottom-up quantitative view from Assessments

9. ENABLERS / TOOLS:A SET OF DIAGNOSTIC TOOLS-PERIODIC PROGRESS MEASURES

10. ENABLERS / TOOLS: DIAGNOSTIC ASSESSMENT METHODOLOGIES

11. Periodic assessments ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION • School customized assessments • Customized version provided to align with different curriculum, sequence or desired outcome (more predictive) • DOE and vendor provided assessments • School created assessments • Standards-based alternative assessment system that.. • Is developed by school, network or region in collaboration with a partner • Ensures equal rigor, measures progress and allows for cross-group comparison • Default system based on citywide ELA and Math curricula • Math #1 • Math #2 • At every grade level, a set of diagnostic assessment tools to help inform interventions and instruction to ensure an individualized instructional plan for every student • Math #7 • ELA #1 • ELA #5

12. ENABLERS / TOOLS: PERIODIC ASSESSMENT – A MENU OF OPTIONS THAT SUIT THE NEEDS OF SCHOOL AND STUDENT POPULATION • School customized assessments • Customized version provided to align with different curriculum, sequence or desired outcome (more predictive) • DOE and vendor provided assessments • School created assessments • Standards-based alternative assessment system that.. • Is developed by school, network or region in collaboration with a partner • Ensures equal rigor, measures progress and allows for cross-group comparison • Default system based on citywide ELA and Math curricula • Math #1 • Math #2 • ARIS • Math #7 • ELA #1 • ELA #5

13. CONTENTS • Accountability Initiative • Vision and Overview • Achievement Reporting and Innovation System (ARIS) • Accountability Initiative Implementation Timeline

14. OBJECTIVES AND SCOPE OF ARIS • Scope of the ARIS • Objectives • Provide principals, teachers and parentswith online information on student achievement, including periodic assessments • Support development of longitudinal and detailed analysis • Develop an integrated portal, including a dashboard to analyze key environment factors and achievement metrics • Enable real-time prediction of school performance against year-end targets • Generate standard reports for specific end-users • Develop scorecards to enable drill-downs by student, assessment, strand and sub-strand analysis • Provide knowledge management tools that capture teaching and assessment content generated at school level with an ability to approve, publish and share with the networks or the broader district. As well as capturing new quality review data • Ensure access to the information and tools necessary to enable • Longitudinal and detailed analysis and reporting of achievement and performance data • Best-in-class assessment design and implementation processes to extract data from multiple sources, including designing own reports and the capability to slice and dice the data • Tools to improve learning of students with a range of identified needs; • Sharing best practices and collaborating across schools to encourage cultures of continuous school improvement and professional learning

15. WHAT ARIS IS AND WHAT IT IS NOT • ARIS is… • ARIS is not… • A solution that… • Analyzes and reports • Assesses • Innovates (assessments, responsive interventions) • Collaborates and personalizes • Shares knowledge (sharing) and manages content • A solution that may not be available off-the-shelf • A solution that may require a partnership or consortium of Bidders to offer, not by a single vendor or a service provider • A collaborative effort between the Vendor and NYCDOE to innovate, develop and execute • A system that will replace all of the current applications now in operation at the DOE (e.g. ATS, HSST). Users will still need to update and print reports very much in the same way they do today but the access to all of the key systems will now all be well integrated through a single interface • An Enterprise Data Warehouse (EDW) that will have all the current available data in DOE (assessment, Financial, HR, etc.). ARIS may be the foundation for an EDW in the long run, but initially the data warehouse will only contain student achievement data and the necessary information to analyze by several key criteria (e.g., age, school, grade, background, etc.) • An opportunity to replace desktops and/or network infrastructure

16. Use best-in class assessments to improve student achievement • Generate reports and analyze achievement/performance data • Innovate the professional learning / share best practices across all schools OVERVIEW OF REQUIRED FUNCTIONALITY 1 2 3 • Periodic assessments • For school- or center-created • Library of templates/idea bank for school-created tests (e.g., rubric-based, standardized multiple choice) • Test creation and upload • For 3rd party-created • Assessment content upload into online testing, reporting and analysis • For all • Online testing and scoring • Score data upload into reporting and analysis • Knowledge and plan management • Library of templates for different document types (e.g., lesson plans, curriculums) • Document creation and upload to repository • Document tagging • Collaborative document creation • Publishing approval • Document linking to reporting (surveys) • Analysis + Reporting • Canned and customized reports • Drill-down, roll-up, filtering • Ad-hoc queries • User-generated databases • Adding new data to central repository • Data viewing in 3rd party tools (e.g., Excel) • Advanced analysis • (for selected users) • Advanced statistics • Pattern detection for target setting Knowledge and plan management (continued) • Searching documents in the repositories • Document viewing and download • Pattern analysis across text documents • Collaboration • Posting announcements; setting alerts • Online discussions; online messaging • Custom workspaces, integration w/email • Personalization • Pre-defined home page based on user role • Personalization of home page (HP) by user • Adding new tools to HP from a tool catalog Auditing (ie.,usage instances, frequency, modification history tracking)

17. KEY TECHNICAL AND SYSTEM REQUIREMENTS Implications for ARIS RFP / requirements Context and challenges • Data issues • Significant archive size • Uncertain data quality • Critical data element gaps • Legacy applications • Significant number, fragmented functionality • Diverse but emerging preferences for platforms • Goal of having a single Enterprise Data Warehouse • Infrastructure • Uneven / unclear quality of... • Desktop standards / availability • Network access • Stringent performance requirements • Funding constraints • Data issues Vendors, based on their experience, to work with DOE on developing a processes and tools for… • Data cleaning and migration • Filling critical data element gaps • Improving data entry quality • Legacy applications Bidders to.. • Develop new data sources for new accountability functions (e.g., QR) • Offer a scalable and evolvable data model • Comply with strongly preferred, but not mandatory standards • Infrastructure • Bidders have the option to host externally, or suggest hardware for DOE-hosted solutions • Optimized for mobile/ existing infrastructure • Funding for infrastructure outside of ARIS contract scope

18. DOE ARIS PREPAREDNESS TASK FORCE FOR KEY TECHNICAL AND SYSTEM REQUIREMENTS • Objectives • The task force will… • Data Integrity: Create solutions and define processes to ensure and maintain confidence in the quality of data and validity of ARIS outputs • Infrastructure: Define the infrastructural improvements for maximizing usage and effectiveness of ARIS • Data Source Inventory: Identify alternate types of approaches/ applications schools use to host data • Communications: Craft messaging tailored to user experience • Identify level of cleanliness of existing data1 • Identify key causes for data inaccuracy or incompleteness • Design and pilot solutions to clean existing source systems feeding ARIS2 • Recommend processes for maintaining data quality going forward • Concentrate on transactional systems (ATS, HSST, etc.) • Context • NYCDOE has set up a Task Force to set ARIS for success by preparing internal DOE systems and processes to complement vendor solution

19. Phase 1 (Sep. 2007) • Phase 2 (Sep. 2008) • ( _ ) Segment size OVERVIEW OF TRAINING APPROACH User segments Role-based view • Central/DAA (50) • Network / Regions (280) • Principals (1,400) • Other school achievement leaders1 (5,600) • Teachers (87,000) • Support staff / DIIT (?) • Parents (~100s K) Skill-based view • Power (352) • Advanced (~4,3503) • Standard (~90,0004) • Standard (~ 100s K) Training approach • Initial training (role-based, project- and site-based) • One-on-one (for power and advanced) • Train-the-trainer (for standard) - school achievement leaders train other teachers • Ongoing need-based training • Phone support service • Online simulation exercises • Initial training • Train-the-trainer – select volunteers to train others • Ongoing need-based training • Phone support service • Online simulation exercises • RFP requires the vendors to submit plans for measuring training effectiveness 1 Includes select admin staff, select teachers 2 Includes 25 Central Staff/Accountability Office; 10 power users drawn from schools 3 Includes 150 network/region; 4200 principals and other school achievement leaders 4 Primarily teachers

20. CONTENTS • Accountability Initiative • Vision and Overview • Achievement Reporting and Innovation System (ARIS) • Accountability Initiative Implementation Timeline

21. ACCOUNTABILITY INITIATIVE IMPLEMENTATION TIMELINE