Ren é Reitsma 1 , Anne Diekema 2 1 Oregon State University 2 Utah State University

1. René Reitsma1, Anne Diekema2 1Oregon State University 2Utah State University Network Visualization of Human and Machine-based Educational Standard Assignment IV10 – London, UK

2. Problem: need for automated educational standard assignment. Question 1: how are machine-based assignments (e.g., CAT) different from human (cataloger) assignments? Question 2: how do humans assign these standards in the first place? (Inductive) method: Generate standard networks of both the human and machine-based assignments. Explore the structure and visual layout of both networks for indications of differences. Network Visualization of Human and Machine-based Educational Standard Assignments – Overview

3. The Standard Assignment Challenge

4. In USA, K-12 teaching & learning is increasingly standards based (‘No Child Left Behind’ Act – 2001) Educational standards (K-12) are governed by the 50 states. States have different views and opinions  50,000+ K-12 Science, Technology, Engineering & Math (STEM) standards: MT: 20, AK: 50, IA: 60, ..., NY: 2,309, IL: 2,881, TN: 3,410 Teachers must find curriculum to teach to the standards. Curriculum providers must assign standards to their curriculum in order to assist the teachers. Available curriculum grows quickly; states regularly change/update their standards. The Standard Assignment Challenge… cont.’d

5. Manual assignment is (very) time consuming  Automatic assignment tools have been under development for some time. Tool of choice: Content Assignment Tool (CAT): Center for Natural Language Processing (CNLP) – Syracuse University Part of NSF’s National Science Digital Library (www.nsdl.org) NLP technology – part-of-speech tags (verbs, nouns, etc.), TF/IDF, etc. CAT web service: Point CAT to an XML/HTML version of a curricular document. Tell it which standard body to assign from. Returns a list of assigned standards. The Standard Assignment Challenge… cont.’d

6. How good are those CAT assignments? What do we really mean when we say that a standard and a curricular item line up? (Reitsma, Marshall, Zarske (IPM – 2010)) If the human and CAT assignments are different, are these differences systematic? Do the human assignments provide suggestions on how to improve the machine-based ones? That’s All Very Nice, but…

7. Notion: visualize both machine-based and human assignments and compare and contrast the visualizations for clues. Approach: build networks of standards; layout the networks, interpret their spatial arrangements: Networks are based on how standards have been assigned to curriculum. Any two jointly assigned standards are considered ‘linked.’ Data: TeachEngineering collection (www.teachengineering.org) – Jan. 2009. CO 2007 Science standards. CAT & human standard assignments. (Inductive) Method & Data

8. CO 2007 Science Standard Assignments

9. CO 2007 Science Standard Assignments... cont.’d – CAT recall = 25 / 324 = .077 – CAT precision = 25 / 139 = .18

10. ‘Curricular units’ – Human network is denser and more clustered. – Human curricular units are clusters –Human clusters link through common standards. – CAT: open structure; less clustering. Has no sense of curricular units.

11. FR diagrams consider the network unweighted; i.e., all links have equal value/weight. Two weights: TF/IDF-like: weigh a standard link inversely proportional to the size of its company. ‘Fidelity:’ weigh a link between standards proportional to their mutual fidelity across the collection. Compute the KK network layouts Weighted or unweighted?

12. Resulting KK diagrams showed essentially the same properties as the FR diagrams (hierarchical cluster analysis of two-dimensional positions)

13. CO Science standards can be grouped and subdivided in two categories: World standards (W): express facts and principles about the empirical world. E.g., S103EC87: Light and sound waves have distinct properties: frequency, wavelength and amplitude. Methodstandards (M!): express ways and means of conducting science. E.g., S103ECE9: A controlled experiment must have comparable results when repeated. Some method standards ‘contaminated’ with world terms/examples (M): E.g., S103ECD4: Technology is needed to explore space (for example: telescopes, spectroscopes, spacecraft, life support systems). Question: How do CAT and human catalogers compare on World vs. Method? Standards: ‘Method’ vs. ‘World’

14. Standards: ‘Method’ vs. ‘World…’ cont.’d

15. W = world M! = (pure) method M = method with world examples – CAT under-assigns method. – Humans: method standards as curricular hubs – CAT central method hub: S103EC77: “physical properties of solids, liquids, gases and the plasma state and their changes can be explained using the particulate nature of matter model“

16. Visualization of standard assignment networks reveals important differences between CAT-based and human-based classification. Tools such as CAT can benefit from contextual knowledge; e.g., that certain lessons are part of a larger set of lessons or a curricular unit. Humans organize curriculum around both World and Method standards. Hence, tools such as CAT MUST!! become better at recognizing Method standards. In USA strong need for automated standard assignment. CAT is a recognized and well-used automated classifier. CAT could use improvement, particularly in the Method segment. Conclusions & Recommendations