Measuring Learning Effectiveness:

1. Measuring Learning Effectiveness: A New Look at No Significant Difference Findings in The Literature by Ernest H. Joy II, Ph.D.

2. Abstract • Hundreds of media comparison studies claim no significant difference • Much of this research is seriously flawed • No significant difference findings can result simply due to faulty research designs • Misinterpretation can easily occur • Pedagogy is the key factor in learning effectiveness regardless of delivery medium

3. Background • Media comparison studies are difficult to design due to many behavioral variables • Nevertheless, the literature is replete with media comparison studies • Current focus is to compare distributed learning systems, i.e., ALN to traditional classroom learning

4. Media are merely conveyances Instructional methods, not media, influence learning effectiveness Predicated on widely accepted learning theory (Gagne, Briggs, and Wager) Media & methods are interconnected technology interacts with learners who are similarly engaged learning from media is a complementary process between the learner and the media The Great Debate:Do Media Alone Influence Learning Effectiveness? Clark vs Kozma

5. Pedagogy is Key to Learning • Events necessary for learning comprise both internal, or cognitive information processing such as temporary storage, encoding, and retrieval; and external events such as gaining attention, drill and practice, and feedback . (Gagne, Briggs, and Wager, 1992.) • Instructional method is any way to shape information that activates, supplants, or compensates for the cognitive processes necessary for learning to occur. (Saloman cited in Clark, 1994.)

6. Clark’s Replaceability Challenge • As long as there is another medium capable of conveying similar methods of instruction resulting in similar learning outcomes, the cause must be something other than the media itself. • Until such time as a single medium becomes necessary for learning, media will continue to serve only as a conveyor of processes which contribute to learning.

7. Common Flaws in Media Comparison StudiesA. Specification Error • Lack of control group • Unspecified size of control group • Non-random selection/assignment of participants • Assessment instrument bias

8. Common Flaws in Media Comparison StudiesB. Failure to control for key variables • Instructional methods • Ability • Prior knowledge • Learning style • Teacher effects • Time on task • Familiarity with media

9. Consequences It is less likely that significant differences will be found since there is no single difference affecting only the treatment group. On the other hand, when a significant difference is found, that difference is most likely the result of some uncontrolled treatment (s) affecting one group more so than the other.

10. Conclusions • Exercise caution in interpreting results of media comparison studies • Improvement in design of media comparison studies is problematic. • Pedagogy is widely accepted in the literature as key to learning effectiveness regardless of the delivery medium.

11. Recommendations • Do not rely on media comparison research alone to determine the effectiveness of an instructional delivery system • Focus on pedagogy to improve learning effectiveness • Shift the research inquiry: “How best to employ the most appropriate delivery medium?”