Observational Methods

Observational Methods January 20, 2010

Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

Survey Results • Only 3 responses  • I will be switching to a better technology for the next survey • Mostly positive  • I’ve read through your comments, and will try to take them into account • Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better • The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

HOLISTIC EXISTENTIALIST ESSENTIALIST ENTITATIVE

Observational methods: top level • You don’t manipulate the research setting in any way • You just look at what is happening, or what has happened, and you note it • Later, you analyze the data

Observational Methods • This class we will focus on quantitative observational methods • Where observations are noted in terms of a defined set of categories • Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

Observational Methods • This week we will focus on quantitative observational methods • Next week we will focus on qualitative methods • Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

What is the research setting? • Where the learning (or other phenomenon you want to study) is taking place

What is the research setting? • Usually a real learning setting • Classroom • University computer lab • Dorm room • Workplace • Any others? • Occasionally a lab setting • Needs to be justification for why the lab setting is valid

Types of observation • Enhanced video coding • Video coding • Field observation • Screen replay coding • Text replay coding • (and more)

High-level notes • High-level methodological similarity • Can be used for many of the same research goals • Advantages and disadvantages to each

Types of observation(that we’ll focus on today) • Video coding • Field observation • Text replay coding

Types of observationthat we’ll focus on today • Video coding • Field observation • Text replay coding

Let’s watch a video • http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related • From 0:00 to 1:15

What student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

List of behaviors • Let’s briefly write down a list of behaviors seen

OK, let’s watch it again • Please write every behavior you see that did not fit into the original coding scheme

What additional student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

What is the total list of behaviors?

Now let’s make a coding scheme • A list of “interesting” behaviors, how they are defined, and a code letter • Example: • Hint abuse (a type of gaming the system) • Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

Now let’s make a coding scheme • Should not be infinitely long – eliminate • “Uninteresting” behaviors – be guided by • your intuition • prior theory and empirical results • “Extremely rare” behaviors – under approximately 1 in 200 • Behaviors occuring between 1-5% of the time are often very interesting behaviors! • Effects on learning are often disproportionate • Gaming the system occurs 3-5% of the time, and correlates to learning at 0.3-0.4

Which behaviors in your list • Are “uninteresting”? • Are “extremely rare”?

Now let’s make a coding scheme • And combine related behaviors • For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together • common • antecedents (poorly known skill) • characteristics (very fast responses) • results (poor learning gains) • differences are no longer considered theoretically interesting (some debate on this still) • tend to co-occur

Which behaviors • Should be combined?

Our Coding Scheme

OK, let’s watch it for the 3rd time • Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) • Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds • If 2 behaviors observed, pick the dominant one • If you are not sure, put down a “?” • Do NOT talk to your partner

OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed • There are better ways to assess inter-rater reliability, we’ll discuss these next class

Each group’s % error

What categories did you disagree on?

OK, now let’s watch • Same video, but times 1:06 to 2:10 • Observe the same student • Follow the same procedure as before

Were the behaviors the same in this segment? • It is important to: • Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) • Note behaviors that do not fit in your coding scheme

OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed

Across sessions • Did your agreement go up or go down? • By how much?

A good strategy • Repeatedly code the same data separately and compare • Until your inter-rater reliability is “high enough” (more next time) • And then code different data separately

Incidentally • There is nothing “magic” about the video coding method we used • It was largely selected for tractability in class

Other things we could have done • Watch the same time segment multiple times • Advantages • Disadvantages

Other things we could have done • Watch the same time segment multiple times • Advantages: Greater certainty in coding • Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages • Disadvantages

Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages: Get better assessment of the duration of each behavior • Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

Other things we could have done • Code the first action in a time window, not the predominant one • Advantages • Disadvantages

Other things we could have done • Code the first action in a time window, not the predominant one • Advantages: Less ambiguity and decision-making • Disadvantages: May be less representative; very brief behaviors may be missed

Other things we could have done • What else?

A limitation of one camera

What is she doing?

Other ways to use video • Multiple cameras • Addresses issue of the student off-screen • Synchronization becomes a challenge • Time to code increases with each camera you add

Other ways to use video • Individual student video • Often from web-cams on top of machines

Example

Example • https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H

Observational Methods