1. Observational Methods January 20, 2010

2. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

3. Survey Results • Only 3 responses  • I will be switching to a better technology for the next survey • Mostly positive  • I’ve read through your comments, and will try to take them into account • Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better • The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

4. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

5. HOLISTIC EXISTENTIALIST ESSENTIALIST ENTITATIVE

6. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

7. Observational methods: top level • You don’t manipulate the research setting in any way • You just look at what is happening, or what has happened, and you note it • Later, you analyze the data

8. Observational Methods • This class we will focus on quantitative observational methods • Where observations are noted in terms of a defined set of categories • Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

9. Observational Methods • This week we will focus on quantitative observational methods • Next week we will focus on qualitative methods • Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

10. What is the research setting? • Where the learning (or other phenomenon you want to study) is taking place

11. What is the research setting? • Usually a real learning setting • Classroom • University computer lab • Dorm room • Workplace • Any others? • Occasionally a lab setting • Needs to be justification for why the lab setting is valid

12. Types of observation • Enhanced video coding • Video coding • Field observation • Screen replay coding • Text replay coding • (and more)

13. High-level notes • High-level methodological similarity • Can be used for many of the same research goals • Advantages and disadvantages to each

14. Types of observation(that we’ll focus on today) • Video coding • Field observation • Text replay coding

15. Types of observationthat we’ll focus on today • Video coding • Field observation • Text replay coding

16. Let’s watch a video • http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related • From 0:00 to 1:15

17. What student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

18. List of behaviors • Let’s briefly write down a list of behaviors seen

19. OK, let’s watch it again • Please write every behavior you see that did not fit into the original coding scheme

20. What additional student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

21. What is the total list of behaviors?

22. Now let’s make a coding scheme • A list of “interesting” behaviors, how they are defined, and a code letter • Example: • Hint abuse (a type of gaming the system) • Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

23. Now let’s make a coding scheme • Should not be infinitely long – eliminate • “Uninteresting” behaviors – be guided by • your intuition • prior theory and empirical results • “Extremely rare” behaviors – under approximately 1 in 200 • Behaviors occuring between 1-5% of the time are often very interesting behaviors! • Effects on learning are often disproportionate • Gaming the system occurs 3-5% of the time, and correlates to learning at 0.3-0.4

24. Which behaviors in your list • Are “uninteresting”? • Are “extremely rare”?

25. Now let’s make a coding scheme • And combine related behaviors • For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together • common • antecedents (poorly known skill) • characteristics (very fast responses) • results (poor learning gains) • differences are no longer considered theoretically interesting (some debate on this still) • tend to co-occur

26. Which behaviors • Should be combined?

27. Our Coding Scheme

28. OK, let’s watch it for the 3rd time • Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) • Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds • If 2 behaviors observed, pick the dominant one • If you are not sure, put down a “?” • Do NOT talk to your partner

29. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed • There are better ways to assess inter-rater reliability, we’ll discuss these next class

30. Each group’s % error

31. What categories did you disagree on?

32. OK, now let’s watch • Same video, but times 1:06 to 2:10 • Observe the same student • Follow the same procedure as before

33. Were the behaviors the same in this segment? • It is important to: • Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) • Note behaviors that do not fit in your coding scheme

34. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed

35. Across sessions • Did your agreement go up or go down? • By how much?

36. A good strategy • Repeatedly code the same data separately and compare • Until your inter-rater reliability is “high enough” (more next time) • And then code different data separately

37. Incidentally • There is nothing “magic” about the video coding method we used • It was largely selected for tractability in class

38. Other things we could have done • Watch the same time segment multiple times • Advantages • Disadvantages

39. Other things we could have done • Watch the same time segment multiple times • Advantages: Greater certainty in coding • Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

40. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages • Disadvantages

41. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages: Get better assessment of the duration of each behavior • Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

42. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages • Disadvantages

43. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages: Less ambiguity and decision-making • Disadvantages: May be less representative; very brief behaviors may be missed

44. Other things we could have done • What else?

45. A limitation of one camera

46. What is she doing?

47. Other ways to use video • Multiple cameras • Addresses issue of the student off-screen • Synchronization becomes a challenge • Time to code increases with each camera you add

48. Other ways to use video • Individual student video • Often from web-cams on top of machines

49. Example

50. Example • https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H