THE MAP TEST -or- “You can attach a meaningless number to anything. Even a child.”

1. THE MAP TEST-or-“You can attach a meaningless number to anything. Even a child.”

2. The graphs & statistical analysis for this presentation came from http://conceptualmath.org/misc/MAPtest.htm#stddev This graph shows the most precise resultsobtainedfrom MAP tests in the classroom of an expert teacher. Yet, 10 out of 38 of the growth scores (26%) fall outside a reasonable range. Further, scores did not match the student performance that the teacher was observing in the classroom.

3. MAP is based on the assumption that all students learn information in the same order, and all schools teach information in the same order. • The RIT scores (RIT score = the final number assigned to each student by NWEA’s MAP test) are no more accurate than these assumptions.

4. What’s RIT ? That’s a loaded question… IRT is Item Response Theory. NWEA uses a specific IRT model conceived by Danish mathematician, Georg Rasch, (1901-1980). They’ve apparently rebranded “IRT” as “RIT”. It’s proprietary…. • NWEA claims that • Characteristics of the RIT Scale include: • It is an achievement scale. • It is an accurate scale. • It is an equal interval scale. • It helps to measure growth over time. • It has the same meaning regardless of grade or age of the student.

5. MAP levels are based on academic levels (curriculum sequencing), not cognitive levels (Bloom's Taxonomy). • A high RIT score can represent low level cognition in advanced curriculum material. • Thus, MAP is not a good measure for gifted performance, and should not be used to guide decisions for gifted programs. (as it has been repurposedin Seattle Public Schools)

6. MAP does not provide information as to why the student is having difficulty with specific material. • MAP does not even provide specific information as to what material students need work on. • As a result, MAP does not provide useful information that can guide teachers in meeting student's real needs. • Teachers don’t see the test, and the test isn’t tied to their classes or curricula

7. All decisions based on test results should consider the 2-sigma range. (Greek letter sigma is a standard statistical result which describes the size of measurement error within a set of measurements. A low sigma represents more believable data. As astronomers say, ”The road to hell is paved with 3-sigma.”) • NWEA reports sigma to be about 3 (except for high RIT scores where sigma is even larger.) • Thus, all decisions should consider the possibility that each reported score could have been 6 point higher, or 6 points lower. • Many schools that have retested with MAP have seen evidence that sigma is actually larger than 3.

8. RIT scores are actually determined by averaging together strand scores. • Since a typical MAP test has 50 questions and 5 strands, only 10 questions were asked for each strand. • Uncertainty for a 10 question multiple choice test is rather large. • This is where the dream of the statistician meets the reality of the young student. Students can’t do tests for hours, and schools don’t have computer resources for students to do tests all the time. So you ask just a few questions, and you get results with a lot of, well, sigma.

9. One means of checking the reliability of a test is to compare its results to another test. This is particularly important if the goal is to increase the scores on the other test. Such would be the case for high stakes testing mandated by NCLB. Below is a comparison of the results of a MAP test to the results of an End of Grade Test (EOG) required bythestate.Reliability was low. MAP ranked about 5 students a year higher in performance, and 4 students a year lower in performance, (23% total) than the EOG.

10. NWEA's technical manual uses r-values to estimate the reliability of the tests. The r-values they report range from 0.76 to 0.93 with most of the values being between 0.80 and 0.89. But what does this mean for those wishing to use the tests to guide instruction?   One can simulate the r-values to estimate what percentage of scores will lie within a reliable range for given r-values. An r-value of 0.92 could easily mean that over 25% of the data lies more than 6 points away from true. However, 6 points constitutes a year's normal growth for over half of the tests. • As simulated in the graph above, even with an r-value of 0.92, 29% of the data is in error by more than a year's normal growth. This imprecision would lead to serious errors in identifying student needs, and then tracking the students incorrectly.

11. For high RIT scores (score that are normal for 8th grade and higher) the SEM (standard error measurement) is larger than a typical year's growth • Thus, MAP is not only unable to identify what skills a student may need, MAP is unable to precisely determine the grade level at which student is performing.

12. For very high RIT scores, negative growth is more common that positive growth. • This implies that either MAP is totally unreliable for high performers, or that MAP testing reinforces educational strategies that are counterproductive for high achievers.

13. So, if the MAP test is not so useful, what should teachers look for ? • If your school is considered adopting a standardized test, quiz the salesperson to demonstrate how the score reporting will help you. • Give specific common examples from your experience. "I have students who can solve two-step equations with minimal difficulty, but frequently confuse the properties for negatives. How will the test report let me know what specific help these students need?" • Be sure the testing company can give good answers to these questions.

14. Ask the test salesperson to demonstrate how the test distinguishesbetween high level learning and accelerated knowledge.

15. Recommendformative assessments that identifyspecific strengthsand weaknesses,as opposed to tests that rank for tracking purposes.

16. Parents & Teachers: Learn more about testing. • Do not accept the “results” of any single type of assessment at face value. All tests have measurement errors, nowhere more so than in testing students, particularly when those students have no stake in the outcome. • Test results are commonly used inappropriately by administration for invalid purposes like student placement and teacher evaluation. • If you get a bad review based on test results alone, file a grievance.Insistthe administrator demonstrate that the test was both precise and accurate enough to justify the results.