1 / 116

Observational Methods

Observational Methods. January 20, 2010. Today’s Class. Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri, Jan. 22 Survey. Survey Results. Only 3 responses  I will be switching to a better technology for the next survey Mostly positive 

eilis
Télécharger la présentation

Observational Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Observational Methods January 20, 2010

  2. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  3. Survey Results • Only 3 responses  • I will be switching to a better technology for the next survey • Mostly positive  • I’ve read through your comments, and will try to take them into account • Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better • The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

  4. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  5. HOLISTIC EXISTENTIALIST ESSENTIALIST ENTITATIVE

  6. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  7. Observational methods: top level • You don’t manipulate the research setting in any way • You just look at what is happening, or what has happened, and you note it • Later, you analyze the data

  8. Observational Methods • This class we will focus on quantitative observational methods • Where observations are noted in terms of a defined set of categories • Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

  9. Observational Methods • This week we will focus on quantitative observational methods • Next week we will focus on qualitative methods • Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

  10. What is the research setting? • Where the learning (or other phenomenon you want to study) is taking place

  11. What is the research setting? • Usually a real learning setting • Classroom • University computer lab • Dorm room • Workplace • Any others? • Occasionally a lab setting • Needs to be justification for why the lab setting is valid

  12. Types of observation • Enhanced video coding • Video coding • Field observation • Screen replay coding • Text replay coding • (and more)

  13. High-level notes • High-level methodological similarity • Can be used for many of the same research goals • Advantages and disadvantages to each

  14. Types of observation(that we’ll focus on today) • Video coding • Field observation • Text replay coding

  15. Types of observationthat we’ll focus on today • Video coding • Field observation • Text replay coding

  16. Let’s watch a video • http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related • From 0:00 to 1:15

  17. What student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

  18. List of behaviors • Let’s briefly write down a list of behaviors seen

  19. OK, let’s watch it again • Please write every behavior you see that did not fit into the original coding scheme

  20. What additional student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

  21. What is the total list of behaviors?

  22. Now let’s make a coding scheme • A list of “interesting” behaviors, how they are defined, and a code letter • Example: • Hint abuse (a type of gaming the system) • Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

  23. Now let’s make a coding scheme • Should not be infinitely long – eliminate • “Uninteresting” behaviors – be guided by • your intuition • prior theory and empirical results • “Extremely rare” behaviors – under approximately 1 in 200 • Behaviors occuring between 1-5% of the time are often very interesting behaviors! • Effects on learning are often disproportionate • Gaming the system occurs 3-5% of the time, and correlates to learning at 0.3-0.4

  24. Which behaviors in your list • Are “uninteresting”? • Are “extremely rare”?

  25. Now let’s make a coding scheme • And combine related behaviors • For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together • common • antecedents (poorly known skill) • characteristics (very fast responses) • results (poor learning gains) • differences are no longer considered theoretically interesting (some debate on this still) • tend to co-occur

  26. Which behaviors • Should be combined?

  27. Our Coding Scheme

  28. OK, let’s watch it for the 3rd time • Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) • Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds • If 2 behaviors observed, pick the dominant one • If you are not sure, put down a “?” • Do NOT talk to your partner

  29. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed • There are better ways to assess inter-rater reliability, we’ll discuss these next class

  30. Each group’s % error

  31. What categories did you disagree on?

  32. OK, now let’s watch • Same video, but times 1:06 to 2:10 • Observe the same student • Follow the same procedure as before

  33. Were the behaviors the same in this segment? • It is important to: • Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) • Note behaviors that do not fit in your coding scheme

  34. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed

  35. Across sessions • Did your agreement go up or go down? • By how much?

  36. A good strategy • Repeatedly code the same data separately and compare • Until your inter-rater reliability is “high enough” (more next time) • And then code different data separately

  37. Incidentally • There is nothing “magic” about the video coding method we used • It was largely selected for tractability in class

  38. Other things we could have done • Watch the same time segment multiple times • Advantages • Disadvantages

  39. Other things we could have done • Watch the same time segment multiple times • Advantages: Greater certainty in coding • Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

  40. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages • Disadvantages

  41. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages: Get better assessment of the duration of each behavior • Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

  42. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages • Disadvantages

  43. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages: Less ambiguity and decision-making • Disadvantages: May be less representative; very brief behaviors may be missed

  44. Other things we could have done • What else?

  45. A limitation of one camera

  46. What is she doing?

  47. Other ways to use video • Multiple cameras • Addresses issue of the student off-screen • Synchronization becomes a challenge • Time to code increases with each camera you add

  48. Other ways to use video • Individual student video • Often from web-cams on top of machines

  49. Example

  50. Example • https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H

More Related