Are there “Hidden Variables” in Students’ Initial Knowledge State Which Correlate with Learning Gains?

1. Are there “Hidden Variables” in Students’ Initial Knowledge State Which Correlate with Learning Gains? David E. Meltzer Department of Physics and Astronomy Iowa State University AAPT Winter Meeting January 12, 1999 Anaheim, California

2. What Affects Learning Gains? “Normalized Learning Gain” [Hake’s “g”] is a widely used measure of conceptual learning. [g=(posttest-pretest)/(100%-pretest)] • g is not correlated with pretest scores. • g is virtually invariant among “traditional” instructors. • gis correlated with instructional method (higher g found for “Interactive Engagement” methods). ____________________________________________ Many studies assert that correlations exist between students’ performance in physics and various preinstruction parameters (e.g., mathematical skill, reasoning ability, etc.) Is it possible that such a correlation might also be reflected in learning gains as measured by g on the FCI, CSE, or similar conceptual diagnostic instruments?

3. What if there are “Hidden” Variables Correlated with g? If g is correlated with any precourse measure (such as mathematical ability), this would have to be taken into account when comparing learning-gain data. It is usually assumed that [pretest score + instructional method] together determine posttest score. However, if precourse measures are correlated with g, then: • other “hidden” variables — besidespretest score — would be required to fully characterize a student’s preinstruction “mental state-function.” • could no longer assume that, e.g., equal FCI pretest scores necessarily imply equal posttest scores for courses taught with identical instructional methods.

4. Previous Studies on Factors Influencing Students’ Achievement in Physics • More than a dozen studies report that mathematical knowledge is significantly correlated with students’ grades. • Several studies suggest that “logical reasoning ability” is an independent factor as well. However: • These studies almost all use traditional, quantitative “end-of-chapter” problems as their performance criterion. • All of these studies focus on students’ scores on course exams, which are not necessarily the same (nor even necessarily correlated) with how much the student actually learned in the course.

5. Guiding Themes of This Work • Research has shown that success on traditional problems is not necessarily indicative of students’ conceptual knowledge. • Students who perform well on exams may have learned little, and students who have lower exam scores may have learned much (if they started with little or no previous knowledge). Here we report investigations of factors related to learning gains, as measured by pre/posttests of conceptual understanding.

6. How Can We Search for Possible “Hidden Variables” in Initial Knowledge State? Study relationship between learning gains and: • ACT Math score (two samples) • Math Skills Pretest (algebra & trig) (one sample) • Pattern of “wrong answers” on conceptual diagnostic pretest (three samples)

7. Could a Math Skills Pretest be a Predictor of Performance? • H.T. Hudson and others have found significant correlation between performance on math skills pretest and student performance on traditional, quantitative exams. • Here we examine possible correlation with learning gain on a qualitative, conceptual diagnostic test (CSE). • Previous study by Hake et al. (1994): Students with high learning gains on FCI scored 19% higher (than low gainers) on math skills test taken when entering university. Fall 98 data sample: 59 students enrolled in second semester of non-calculus general physics course; 63% female. Math pretest taken within previous 18 months (before taking first semester course).

8. Diagnostic Instruments • Conceptual Survey of Electricity (23-item abridged version), by Hieggelke, Maloney, O’Kuma, and Van Heuvelen. It contains qualitative questions and answers, virtually no quantitative calculations. Given both as pretest and posttest. • Diagnostic Math Skills Test (38 items) by H.T. Hudson. Algebraic manipulations, simultaneous equations, word problems, trigonometry, graphical calculations, unit conversions, exponential notation. Not a “mathematical reasoning” test. Given as pretest only.

9. Learning Gains vs. Math Pretest Scores[Fall 98]

10. Math Pretest Score for High and Low Gainers[Fall 98]

11. Does this imply that improving algebraic skills will lead to increased conceptual learning gains? Probably not. More likely, performance on a math skills test is related to other relevant parameters. (Reasoning ability? Learning rate? Motivation?)

12. Learning Gains vs. “Wrong Answer” Pretest[Fall 98]

13. “Wrong Answer” Pretest for High and Low Gainers[Fall 98]

14. “Wrong Answer” Pretest for High and Low Gainers[Fall 97]

15. “Wrong Answer” Pretest for High and Low Gainers[Spring 98]

16. Analysis of “Wrong Answer” Pattern on Conceptual Pretest • “Wrong Answers” on 11 (out of 23) items on CSE pretest analyzed; • Certain specific answer options are identified as “favored” (though incorrect), perhaps representing “transitional states” of knowledge. • Percentage of “favored” options selected is assigned as “WA” score. (Correctly answered questions are ignored.)

17. Summary • There is significant evidence that precourse measures may be correlated with students’ individual learning gains (even “normalized” gains). • Purely quantitative skills may be (indirectly) related to conceptual learning ability. • Patterns of “wrong answer” choices may provide evidence of students’ initial knowledge state (and of their probable learning gains in a course).