Assessing the development of student ideas over time

1. Assessing the development of student ideas over time Karen Draney University of California, Berkeley

2. What is meant by student growth? • Change in average score over time? • Groups of students • Pretest/posttest model • T-test or ANOVA • Maintaining steady level each year in standardized testing? • Normative definition • Staying at the same percentile level

3. What is meant by student growth, continued • Meeting some defined set of criteria? • Reaching a set percentage (e.g. 80% correct) on series of tests • Piagetian-style developmental levels? • Strong theoretical background • Certain kinds of performances on specified sets of tasks • Elimination of misconceptions? • E.g. Minstrell’s work

4. Our approach • Wilson’s BEAR Assessment System • 4 “building blocks” • Progress variables • Items design • Outcome space • Measurement model

5. Connection to NRC Assessment Triangle

6. Progress map • The heart of the system • Big ideas, central concepts, organizing principles… • It is critical to pick the right ones -- but not too many (or teachers will not be able to use) • Developmental in nature

7. “Developmental” can mean… Following a (perhaps) necessarily ordered development of more complex ideas: Progress map for speed

8. Or, displaying increasing amounts of complexity: 5. Generation: Students use the models to generate new knowledge and to extend models. (~graduate school) 4. Construction: Students integrate scientific understanding into full working models of the domain. (~upper division) 3. Formulation: Students combine unirelational ideas, building more complex knowledge structures in the domain. (~lower division) 2. Recognition: Students begin to recognize normative scientific ideas, attaching meaning to unirelational concepts. (~high school) 1. Notions: Students bring real-world ideas, observation, logic and reasoning to explore scientific problem-solving. (~middle-school)

9. Or even, the order in which a curriculum is arranged: Why things sink and float:

10. Items design • Each item directly linked to one or more of the progress variables • Each item displays a clearly defined relationship to two or more levels of a progress variables • The levels of a progress variable remain consistent across the set of items (and their responses) • Relationship between items and progress variables is empirically tested

11. Outcome space • Specifies the relationship between every class of responses to given items and the levels of the progress variables • For every item, these must be ordered, finite, and exhaustive • This takes work. Lots of work. Looking at student responses, and ideally, talking with students, is critical here

12. ChemQuery Examples of items from our instrument: Both of the solutions have the same molecular formulas, but butyric acid smells bad and putrid while ethyl acetate smells good and sweet. Explain why these two solutions smell differently. C4H8O4 ethyl acetate C4H8O4 butyric acid

14. Does the mass of the solution change after the two solutions are mixed and a solid forms?

15. Please answer the following question. Write as much information as you need to explain your answer. Use evidence, examples and what you have learned to support your explanations. Why do things sink and float?

17. The measurement model • Provides statistical evidence of assessment quality (reliability, validity) • Provides assessment of student growth, even across non-identical groups of items • Provides empirical evidence on the fit of the items & outcome space to theory of progress variable • Provides evidence of the performance of particular items and persons (including fit) • Provides the ability to develop a variety of graphical summaries of student development • The key to making sure the system is working; drives our process of revision

18. Dotted line indicates expected performance; blue line is actual performance

19. Gains from Pre- to Post-test

20. The process Construct Map Measurement Model Items Design Outcome Space

21. LBC initial constructs Atomic View 5 Integration across variables 4 Bonding (ionic, covalent molecules, perhaps collections of molecules 3 Model of the atom (including elements and periodicity 2 Particulate view (definition of matter as particulate) 1 Macroscopic observations complex building matter simple

22. Second round Quantitative vs Qualitative 5 Integrate the domain 4 Predict Scientific models 3 Relate Patterns and Equations 2 Represent Scientific definitions 1 Describe Observations & experience 0 E Evaluating Energies Energy transfer used to analyze tendency for change F Quantizing Energy Interaction of light with matter elucidates structure A Characterizing Matter Matter is composed of atoms arranged in various ways B Measuring Matter Mass is used to account for matter C Characterizing Change Change is associated with rearrangements of atoms D Quantifying Change Mass is used to keep track of change

23. models & evidence evidence about things we can’t observe directly limitations of models examining evidence assumptions measured amounts of matter density, grams per mole,molarity mass with a particulate view atoms, isotopes, moles amounts of matter mass, weight, volume, pressure kinetics & changes in bonding rxn mechanisms, activation energy products of reaction solubilities, relative acid strengths change & reaction types precipitation, acid-base, redox change with chemical symbols balanced equations, phys, vs chem change attributes of change mixing, dissolving, color change, change in form stoichiometry & equilibrium weak acids & bases, solubility of salts, gases amounts of products limiting reagents, strong acid/base titrations, % yield amounts of reactants & products reaction stoichiometry, pH change with a conservation view conservation of mass in chemical reactions changes in amount changes in mass, weight, volume spectroscopy & structure group theory, transition probabilities electronic structure quantum model, atomic & molecular orbitals, ionization energy color with light absorption absorption & emission spectra energies associated with light frequency, speed, Planck’s constant light production of light, color bonding & reactivity advanced bonding models, nucleophiles, electrophiles phase & composition bond strengths, intermolecular attractions, polarity properties & atomic views octet rule, ionic and covalent bonds matter with chemical symbols elements, compounds, valence electrons, periodic trends properties of matter solids, liquids, gases, mixtures particle & energy views statisical mechanics KE & temperature degrees of change entropy, free energy & equilibrium energy transfer & change enthalpy changes Hess’s law, bond breaking heats & temperature heat capacity, calorimetry, exo(endo)thermic measures of energy temperature, scales measures of energy 5 Integrate the domain 4 Predict Scientific models 3 Relate Patterns and Equations 2 Represent Scientific definitions 1 Describe Observations & experience 0 A Characterizing Matter Matter is composed of atoms arranged in various ways B Measuring Matter Mass is used to account for matter C Characterizing Change Change is associated with rearrangements of atoms D Quantifying Change Mass is used to keep track of change E Evaluating Energies Energy transfer used to analyze tendency for change F Quantizing Energy Interaction of light with matter elucidates structure

24. Final version Focus on Student Understanding expert 5 Generation: Research 4 Construction: Examining assumptions, relating models 3 Formulation: Relating ideas and concepts, simple models 2 Recognition: Language, definitions, symbols algorithms 1 Notions: Everyday experience, logical reasoning student understanding novice