Inquiry into effective teaching with one hand tied behind your back

1. Inquiry into effective teaching with one hand tied behind your back 22 November 2006 UBC Institute for the Scholarship of Teaching and Learning Dan Bernstein, University of Kansas djb@ku.edu

2. Overview of session Invited to discuss my research methods Among methodological issues in research on teaching, design and measurement are key Look at examples of some of my methods Embed them in larger framework as I go Inform some practices by research literature Recommend a different path from my own Propose a big tent, even for research done without the best possible methods

3. Research Methods Issues Design and Analysis Measurement

4. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study

5. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment

6. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment EXAMPLE

7. Experimental Procedures • Commercial and home-made multi-media • Used video and live lecture with same content as multi-media • Created text samples with same content • Assessment of levels of understanding with mixed methods of responding • Individual instruction with unlimited time • Participants from introductory psychology

8. Lecture, Reading, and Computer

9. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment Analyze components

10. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment Analyze components EXAMPLE

11. Mean Instructional Time by Instructional Mode

12. Time on task (in minutes)

13. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment EXAMPLE

14. Upgraded the lecture component • Live lecture to portion of introductory class • Topic is reactive measurement and quantum mechanics • Matched to locally authored multi-media and text -- also irrelevant text control • Motivation sustained by possibility of test • Assessment layered by type of understanding and format of question

15. Distribution of total test performance resulting from four methods of instruction

16. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment Statistical comparison

17. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment Statistical comparison EXAMPLE

18. Pushed envelope on assessment • Looked for performance assessment • Topic was searching a jail cell • Verbal assessment is recognizing and stating guidelines for complete search • Authentic assessment is use of guidelines in searching a jail cell assembled for testing • High end interactive program developed by NETV for Nebraska Sheriff’s Association

19. Authentic v. Verbal Assessment

20. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Control Treatment ¿Ethical questions? ¿Permission?

21. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Benchmarks of achievement

22. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Benchmarks of achievement EXAMPLE

23. Criteria for oral interview

24. Criteria for oral interview

25. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Benchmarks of achievement Movement of achievement

26. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study Benchmarks of achievement Movement of achievement EXAMPLE

27. [Universal] Design for Success • Presume students can learn • Discount need to sort or differentiate • Maximize overall course performance

28. Benjamin Bloom promoted mastery • Based on practice and feedback • Divide course into many smaller units • Take examinations and get results • Require taking exam again until high score • IFF 95% correct => study next unit

29. Fred Keller promoted mastery • IFF 95% correct => study next unit • Course grade is number of units passed • No penalty for repeating and learning • All who pass 12 units => grade of A • Do A work on 10 units => grade of C

30. Mastery Class 95% contingency No penalty for learning Immediate feedback No lecture required Lecture class Same exam questions Two attempts per test In class feedback No contingency Also taught conventional lecture

31. Total amount learned • Nearly twice as many at the high end of learning • Virtually no one failed to learn • Maximized learning for many students

32. Showed in amount and accuracy • Many more questions answered • Took 12 15-item tests • Lecture was three tests of 20 items each • Certify more learning • Overall percent correct also higher

33. No magic -- students studied better • They put in more time to their learning • There was more work asked for by the course • Note that they report doing the reading more • Preparation for class is key issue (later also) • Guideline in US -- 2 hours outside for every 1 hour in class

34. Research Methods Issues Design and Analysis Measurement Random Assignment Experiment Descriptive Study What do you think about the relative utility of these approaches?

35. Research Methods Issues Design and Analysis Measurement Direct Indirect

36. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Cognitive Affective

37. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Cognitive Affective ¿EXAMPLES?

38. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Affect Contextual application Repeat knowledge Abstract understanding EXAMPLE

39. Added online questions on reading

40. Feedback until mastery of topics

41. Addition of EDU Out of Class

42. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Affect Contextual application Repeat knowledge Abstract understanding YOUR EXAMPLES

43. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Affect Contextual application Repeat knowledge Abstract understanding EXAMPLE

44. Reasonably Conceptual Items • Describe the results of [Razran’s] second experiment on semantically mediated generalization (numbers) done with a 13-year old boy. Be sure to identify the respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than physically mediated generalization? • What do chicken scratches, raccoon rubbings and pig rooting have in common? What stimuli have pre-existing relations these bits of behavior? What does the “misbehavior of organisms” have to do with the generality of principles of learning?

45. Transform into Problems • Old question: • What were the reinforcing consequences in the Welsh, Bernstein & Luthans (restaurant) study? How were the consequences identified? • Problem-based Assessment : • Suppose you were asked to implement a motivational program in a business with 25 employees engaged in the following activities: production planning, inventory delivery, direct production, packaging, and marketing. Based on your understanding of the restaurant study, how would you improve the quality of the employees work by using access to activities as a motivator? Describe the costs and benefits of the program and make a recommendation about implementation.

46. Old question: • The behavioral context into which punishment is added must be considered in evaluating the likely effect of a given stimulus. What conditions are essential in this analysis? • New question: • Suppose it is your task to decrease the frequency of college students engaging in binge drinking or games involving high levels of intoxication. You have been asked to set up a punishment program to eliminate this program before someone is hurt. What context for binge drinking would you identify first? What punishing consequence would you use? What characteristics would you include in your punishing system to maximize effectiveness? Please give an example of an additional element you would need to include to make the plan maximally effective.

47. Old question: • Describe the results of the experiment on semantically mediated generalization done with a 13-year-old Soviet boy. Be sure to identify the respondent behavior, the training stimuli, and the test stimuli. Why is this example semantically rather than physically mediated generalization? • New question: • Generate your own example of classical conditioning using the neutral stimuli “idea” and “rock” along with the eliciting relation between a very loud noise and increased heart rate. Your example should include all of the following components: a description of a conditioning procedure that would produce differing reactions to the two stimuli, a description of the procedure that tests for the direct effects of conditioning, a description of a procedure that would test for physically mediated generalization, a description of a procedure that would test for semantically mediated generalization, and the likely results of the three test procedures.

48. Distribution on 1st Attempt

49. Change of Questions

50. Research Methods Issues Design and Analysis Measurement Direct Indirect Verbal surrogate Authentic Affect Contextual application Repeat knowledge Abstract understanding General interest Intent to act