Presentation Transcript

1. The ViewOctober 4, 2012

   Dr. Valorie Foy valorie.foy@nebraska.gov
2. Dr. Valorie Foy, Director Jeremy Heneger, Assistant Director John Moon, NeSA Project Manager Dr. Ed Foy, Writing Coordinator Dr. Ted Larson, NAEP Coordinator Jackie Naber, Administrative Assistant Carol Bom, Administrative Assistant The Assessment Team
3. NSSRS C4L Update Writing NeSA Nebraska Performance Accountability System [NePAS ] Today’s Agenda

4. NSSRS

5. NeSA results ARE an important data source! When combined with other information, these data can support curricular, instructional, and learning support decision making. For NeSA... every student will be tested!
6. Commit to data analysis as a continuous process, not an event. (Reeves, 2009)
7. Nebraska schools should use NeSA data to . . . Provide feedback to students, parents and the community Inform instructional decisions. Inform curriculum development and revision. Measure program success and effectiveness. Promote accountability to meet state and federal requirements.
8. Ensuring Quality Data Online Assessments Make sure students use their own Test Session Ticket Test sessions can be taken in any order Have students exit NeSA completely after testing Install NeSA software early and verify it works using Practice Tests Paper/Pencil Assessments Make sure the student ID is correct and complete (page 2) Affix labels on front of every answer document Separate answer sheets and test booklets Leave all test booklets and answer sheets intact Inventory test materials early and order more if needed Bubble answers on correct part of answer sheet
9. Ensuring Quality Data Code Not Tested (NT) Reasons Use Management Tools to enter NT codes for the online assessments Bubble NT codes on answer sheets for pencil/paper assessments Tested Codes include: No Longer Enrolled (NLE) Parent Refusal (PAR) Emergency Medical Waiver (EMW) Recently Arrived LEP (RAL) Student Absent Entire Window (SAE) Invalid (INV) Other (OTH)
10. Ensuring Quality Data Score Corrections (ZEROS and NT codes) June Review by NDE Staff New Student No Attempted Documents Returned Duplicate State ID Multiple Documents Returned Online Demographics Mismatch Alt Assessment Flag Changed Late June districts are contacted for NT Reasons for DRC reports August corrections DRC reports revised in NSSRS for SOSR October 2012 SOSR District Preview for Final Corrections (Two weeks from release to make corrections)

11. C4L

12. Timeline for System Updates October 1st System Management Update Improved method for uploading students October 29th Test Creation Update Improved process for building reading tests New Items Added Throughout Fall Check 4 Learning
13. Currently Districts Can Upload students to the system Place students into classes Use old tests Create and use new math and science tests Please note: System will be down: Sept. 28 & Oct. 26 Check 4 Learning
14. WebEx for System Update October 30 2:00 p.m. October 31 10:00 a.m. Recorded and posted on NDE website Check 4 Learning
15. Ideas for Use: Point of Instruction Assessment Curriculum Unit Assessment Standard Review District/Building Level Assessment New Student Skill Assessment L to J Model Check 4 Learning http://www.education.ne.gov/Assessment/C4Learning.htm
16. Moving Forward Currently Review and Approval of Items Seeking Feedback Determine Needs Minimize Burden Earlier MOU Maintain System that is Useful Check 4 Learning
17. Statistics Over 4000 items in the system Over 1500 ready to be approved Last year from Feb. until the end of the year-135,000 assessments were administered online Approximately 180 Districts take part in C4L Check 4 Learning

18. NeSA Writing

19. Prompt Creation Field Testing Prompt Selection Rangefinding Score Verification (formally Writing Appeals) Continued training and communication Nebraska Educators continue to be involved in these aspects of NeSA-W:
20. Nebraska Director of Statewide Writing Includes 7-10 Nebraska educators per grade level from a variety of educational backgrounds. Two DRC reps facilitate the work. The Rangefinding Process:
21. Analytically score, discuss, and come to consensus on 90-100 papers from the field testing of the test prompt. DRC uses the papers selected and scored by this group to establish the anchors and train the raters prior to the scoring of the assessment. The Rangefinding Process:
22. On-line Writing Times
23. When time on task was plotted with the scale score the following general conclusions seem plausible: 1. Student scores start to level off after a 75-90 minute period so students should be encouraged to spend a minimum of 75 minutes. Grade 8 and 11 Testing Time
24. 2. With SPED students, the trend is somewhat upward for the entire time tracked, so SPED students could be encouraged to spend as much time as they need as long as they are productive. 3. However, keep in mind students cannot be controlled for time during testing. Training and instruction must come in the classroom before the test. Grade 8 and 11 Testing Time
25. In April NDE conducted a standard setting for grades 8 and 11. The results of the process were presented to the State Board, resulting in new cut scores for these grades. The quality of writing did not decline but rather the board--after discussion over a two day period- raised the requirements for Meets the Standards. Scoring Shift – Grades 8 and 11
26. Under the old holistic grading system, a student at grade 8 and grade 11 would reach Met the Standard with a 2 and a 2+ (4.33). Descriptions in the 2 column included: Digressions Limited unrelated details Occasionally forced or contrived Errors may district
27. With new analytic system, a student must have a score that falls in the 3 range of the rubric (if equivocated to the old rubric the student needs a 3 and a 3 -). 3 Column: Generally Adequate Logical Functional Paragraphing successful Conventions usually correct and do not distract
28. When looking at the rubrics (old and new), the 3 column is the acceptable standard. The Nebraska PLD’s (Performance Learning Descriptor) also clearly reflect the 3’s column on each rubric as the state writing goal. The new cut scores will allow some 2’s, but most likely in the domains of word choice and conventions. Scoring Shift – Grades 8 and 11
29. September 13 – New prompts were written by Nebraska teachers at grades 4, 8, 11. The new prompts need to be field tested. Field Testing at Grade 4 will remain the same as in the past. Testing will be in the spring from the end of February to middle of May using paper/penciltests. Field Testing at 4, 8, 11
30. Grades 8 and 11 will be on-line through the NeSA testing site. Prompts are assigned randomly to students. November 12 - tickets are available and new download if needed. New button on NeSA icon will say NeSA-W field testing. The field testing window will be November 14 to December 21. (5 1/2 weeks) Field Testing WebExes November 6-8 New Process - Field Testing Grades 8 and 11
31. This is an opportunity for students to practice taking the test under the same conditions as the regular NeSA-W test. As a field test of future prompts for NeSA–W, the test results cannot be used in any other way. (For example: no local scoring) NDE appreciates schools supporting this project as we want to keep prompt writing in the hands of Nebraska educators. Field Testing
32. Designed for teachers. November 8 - Scottsbluff November 14 - Kearney November 16 – Lincoln Times: 8:30 – 11:30 Materials from the training will be posted to our website at: http://www.education.ne.gov/Assement Writing.htm Analytic Scoring Training Dates Grade 4
33. Will remain in the narrative mode. Will continue to be two timed 40-minutes sessions Will still be tested in paper/pencil format. (No online administration planned) Will allow access to commercial dictionary and thesaurus. Statewide Writing AssessmentGrade 4 – 2012-2013
34. Will be scored analytically in 2013 Will have a new cut score set in April 2013 and approved by the State Board of Education in May. Will have test results released publicly in the spring of 2013 along with grades 8 and 11. Statewide Writing AssessmentGrade 4 – 2012-2013
35. Will include a new self-assessment tool for analytic scoring. Information about the modes of writing and student writing samples are found on the assessment website in the writing section: http://www.educatin.ne.gov/Assessment/NeSAWriting.htm Statewide Writing AssessmentGrade 4 – 2012-2013
36. ON New at grade 4 ON the Test – Self-Assessment Tool
37. Analytic scoring provides five scores for each student: a score for content, a score for organization, a score for word choice and voice, a score for sentence fluency and conventions (punctuation, grammar, spelling), and a total overall composite score. After the standard setting process, the composite score will be converted to a scale score of 0-70. Students in grade 4 will receive analytic scoring information in 2013. Analytic Scoring – Grade 4Starting 2013
38. The four domains of writing to be used in analytic scoring: Scoring domains were weighted as follows: Content and ideas 35% Organization 25% Word choice and Voice 20% Conventions (grammar, punctuation, spelling) 20% Analytic Scoring – Grade 4 Starting 2013
39. In April of 2013, a cut score process will be conducted for the writing assessments in grade 4 so that the following classifications of proficiency determinations can be made: Exceeds the standards Meets the standards Below the standards Grade 4 - Cut Score
40. In December of 2011, NDE conducted a session to create the Performance Level Descriptors for the expectations of Nebraska students based on the analytic rubric. The PLD’s are available on the Assessment web page: http://www.education.ne.gov/Assessment/NeSA_Writing.htm Grade 4 - Cut Score
41. The State Board of Education will approve the scores at their May meeting. Following the cut score process and approval in May, the conversion to the scale score (0-70) will occur, and the public release will follow. Grade 4 - Cut Scores
42. Based on what happened at Grades 8/11, we can anticipate a significant shift at Grade 4 in the requirements for Meets the Standard. Grade 4 – Cut Score
43. For any paper/pencil tests, students must write in #2 PENCIL. Papers written in ink will not be scored and will result in zero scores. In 2012 the papers/pencil booklet submitted in ink, therefore, not scorable, were for the most part administered in make-up sessions by someone other than the regular classroom teacher. Lessons Learned from 2012
44. In paper/pencil testing, the student response to the writing prompt must be written in the actual test booklet. If a student with an IEP is word processing a paper or using a scribe, the student response must be transcribed into the test booklet in pencil. Originals of transcribed papers will be sent to DRC in an envelope provided with the shipping materials. Lessons Learned from 2012
45. This score verification process is appropriate only for very specific, not general, scoring issues. Only the total score will be submitted. The score verification process is designed for student scores that fall below the state cut score. Statewide Writing Appeals Process in the future under analytic score will be called Score Verification
46. Districts will need to complete the score verification form, on the website or from the SAA update. Districts must provide a written explanation indicating the reason for the score verification change and a copy of each paper in question. The writer of the score verification should focus on the particular domain or domains in question. Score verifications must reach the Assessment Office by the deadline. Score Verification is completed by Nebraska educators who have been involved in the writing grading for years. Appeals – Score Verification (cont)
47. Questions contact: edward.foy@nebraska.gov Phone: 402-471-2947

48. NeSA

49. NeSA News2013 – NeSA Testing Window January 21 – February 8 February 11 – March 22 March 25 – May 3 NeSA Writing ELDA NeSA – Reading Mathematics Science NeSA-AA Alternate Tests
50. All paper/pencil tests must be ordered October 8-19 through eDirect – writing, reading, mathematics, and science. Enrollment form Online Testing 2013
51. Students whose IEP plans or 504 plans specify an accommodation that would require a paper/pencil. ELL students who need an accommodation that would require a paper/pencil. Students responding in a language other than English or Spanish (NeSA-W). Students contracted to institutions where online access is not allowed. Students who take the alternate assessments-R/M/S 2013 – Online TestingRequirement in reading, mathematics, science, writing (Grades 8 and 11) for all students except:
52. Review the revised Accommodations document-On Assessment Page-NDE District-Accommodations that are best for the student A student who takes one content area paper/pencil does not necessarily need paper/pencil in all Students who qualify for speech may not need any paper/pencil Inform the student Research for the October Paper/Pencil order
53. October 2-4, 2012 – “Invitation” is posted Enrollment WebExes

54. NePAS

55. Federal Accountability – AYP/PLAS State Accountability – NePAS Tied to LB # 870 Nebraska Performance Accountability System - NePAS
56. 1. Three NePAS Grade-Level Configurations 2. School District Section I-Scores with Rankings
57. Section I: 4 charts for each school district Elementary Grade-Level Configuration Grades 3-5 Middle Grade-Level Configuration Grades 6-8 Secondary Grade-Level Configuration Grades 9-12 District Grades 3-12
58. Status—Scale Score Improvement—Scale Score Growth—Scale Score Participation--Met/Not Met or Percent Graduation Rate--Percent Components
59. Reading E M S D Math E M S D Science E M S D Writing E M S D Scale Scores and RankingsStatus=Average Scale Score
60. Reading E M S D Math E M S D Science* E M S D Writing* E M S D Scale Scores and RankingsImprovement= Difference of Average Scale ScoresCompares scores of groups of students
61. Reading E M D Math E M D Scale Scores and RankingsGrowth= Average of Differences in Scale ScoresGrowth based on matched student data in the same district-consecutive years
62. Will be accompanied by Enrollment- Grades 9-12 4 Year 2012 (Secondary & District) 6 Year* Available 2013 (Secondary & District) Graduation RatePercent and Ranking
63. Reading E M S D Math E M S D Science E M S D Writing E M S D Participation (95%)Met/Not Met-No Rankings
64. Reading, Math, Science 8th and 11th Grade Writing Cut Scores Charts are on NePAS screens
65. Section 1--Anywhere District Nebraska Performance Accountability (NePAS)Elementary Grade-Level Configuration (Grades 3-5) DRAFT
66. Ties will receive the same ranking The next school will receive the next lower number. Example: If 20 districts are ranked # 1 in a component,the next district will receive a ranking of 21. Rankings
67. Appropriate masking (under 10) Any groups 10 and above to be included In any criterion where 100% of the students place in a single rating, the scores must be masked—by law. Example 100% graduation rate NePAS business rules- still being determined
68. With inclusion of ten or more-Number of districts possible in reading and math
69. By Grade Level-Building (Actual physical building) By Grade Level-District No Rankings Status (SS) Improvement (SS) Growth (SS) Participation (%) NePAS Part II
70. Example Grade 4-Reading Status (SS) Improvement (SS) Growth (SS) Participation (%) For eachBuilding in a district for each Grade-Scale Score
71. Example Grade 4-Reading Status (SS) Improvement (SS) Growth (SS) Participation (%) For eachDistrict-for each Grade- Scale Score
72. Section 2.1 School Building Accountability By Grade Choose Grade: (Drop Down Menu) Grade 05
73. State of the Schools Report October State of the Schools Report--available to districts Disaggregated data AYP/PLAS NePAS 15 days—then rankings added November State of the Schools Report Public Release Disaggregated data AYP/PLAS NePAS
74. The month of October gives districts the opportunity to prepare a message Others-Focus on the district chart Your Story
75. Anywhere School District School District (Grades 3-12)
76. Many rankings—focus on strengths If there are scores that are weak, provide information that informs the scores Use the counterbalance of information—the story behind the rankings Our community wanted to know what we were doing to improve Your Story

77. Test Security

78. Caveon Security Report United State Department of Education: Office of the Inspector General Two Security Events
79. Equity Security
80. Allowing word walls, personal dictionaries, graphic organizers on writing test Editing student first drafts on writing test Reading the reading passages to students Allowing students to carry tests from one teacher to another unmonitored Providing accommodations not allowed by an IEP Allowing dictionaries on reading test. Examples of Score Invalidation
81. Examples of Score Invalidation Student accessed help on cell phone Student used inappropriate math document from notebook
82. New Teacher New Principal New District Assessment Contact Substitute Teacher Paraprofessional Make-up Test Reasons for breach in security- that called for score invalidations
83. District information Date of incident Description of incident Action taken District response (if known) Actions taken to assure situation will not be repeated Security Breach Form
84. List of missing materials Action district took regarding the breach (missing materials) District Outcome/Resolution Missing Materials
85. Enrollment for Paper/Pencil Orders October 2 and 3 “Invitations” posted on Assessment Page C4L Update October 30 and 31 “Invitations” posted on Assessment Page Technology Update/Writing Field Test 8 & 11 November 6-8 Dates Posted WebExes Galore
86. Questions? http://www.education.ne.gov/Assessment/Index.html