Methodology Matters: Doing Research in the Behavioral and Social Sciences

1. Methodology Matters:Doing Research in the Behavioral and Social Sciences ICS 205 Ha Nguyen Chad Ata

2. The way we do research matters • Because the meaning of the discovered evidence is based on the way it is obtained • 3 domains are always involved when doing research • Substantive content of interest • Conceptual  ideas that give meaning to the content • Methodological  techniques for conducting research • 3 important keywords • Relations which are between elements within a context or embedding system

3. The Substantive & Conceptual Domains • Substantive Domain • Elements = Phenomena • Relations = Patterns of phenomena • Studying the “actors behaving towards objects in context” • Conceptual Domain • Elements = Properties of the actors’ behaviors... • Relations = Interconnections between elements

4. The Methodological Domain • Elements = methods = Modes of Treatment • Includes techniques for • Measuring • Manipulating • Giving instruction to participants • Imposing constraints on the environment • Selecting materials for use • Giving feedback about prior performances • Using experimental confederates • Controlling • Distributing impact

5. … Methodological Domain… • Relations = Comparison Techniques • Involve 3 features of the system under study • Dependent variables or features • These are measured • Independent variables or features • These are manipulated • All of the rest of the system’s features • These are relevant but not the focus of the study • Comparison performed to assess covariation or association between dependents and independents

6. The Domains

7. Research Methods Are Like Tools • Each method offers opportunities not available by other means • Each method also has inherent limitations • Therefore you should use multiple methods in order to counterbalance the flaws in each individual method • Make sure to chose methods such that one strength can compensate for another’s weakness

8. The Research Methods Abstract Precision Maximum II III I Obtrusive Unobtrusive IV Realism Maximum Generalizability Maximum Concrete

9. Quadrant I: Field Strategies Abstract Precision Maximum II III Obtrusive Unobtrusive IV Realism Maximum Generalizability Maximum Concrete

10. Quadrant I: Field Strategies • Field Study • Natural & unobtrusive • Field Experiment • Manipulating one variable in an otherwise natural system  decrease in realism • A bit obtrusive

11. Quadrant II: Experimental Strategies Abstract Precision Maximum III I Obtrusive Unobtrusive IV Realism Maximum Generalizability Maximum Concrete

12. Quadrant II: Experimental Strategies • Laboratory Experiment • Concocted system • Increased precision • Decreased realism & generalizability • Experimental Simulation • Similar to lab experiment, but • strives to increase realism

13. Quadrant III: Respondent Strategies Abstract Precision Maximum II I Obtrusive Unobtrusive IV Realism Maximum Generalizability Maximum Concrete

14. Quadrant III: Respondent Strategies • Sample Survey • Sampling of the population  high generalizability • Low precision • Judgment Study • Here the focus is on the stimulus material, rather than respondents’ attributes  High precision • “actors of convenience”  Low generalizability over the population • Both often done in neutral settings  low realism

15. Quadrant IV: Theoretical Strategies Abstract Precision Maximum II III I Obtrusive Unobtrusive Realism Maximum Generalizability Maximum Concrete

16. Quadrant IV: Theoretical Strategies • Formal Theory • Formulating general relations between variables of interest  high generalizability • No concrete system involved  low in realism • No observation  low precision • Computer Simulation • Like experimental simulation, except human participants are not included • Modeling the real world  high realism • Outcomes are predictable  low precision • Modeling a particular system  low generalizability

17. Questions to keep in mind… • Does the presented material coincide with the strengths & weaknesses of the research strategies that were used? • Is the research evidence well supported by a single or by multiple research strategies? • Are the strategies’ weaknesses and strengths offset such that they counterbalance one another?

18. Study Design • For every empirical study, observations have to be gathered, combined, and compared • Ex: Given a pair of properties, X and Y, can we tell if X caused Y? • X is independent variable, Y is dependent variable • What techniques are available for comparison? • Baserates • Correlation • Difference

19. Comparison Technique #1 • Baserates • How often (at what rate, or what proportion of time) does Y occur? • If I do not know how often Y occurs in the general case, then I cannot decide whether the rate of Y in the study is significant (particularly high or low).

20. Comparison Technique #2 • Difference Question • Is Y present under conditions where X is present? Is Y absent when X is absent?

21. Comparison Technique #3 • Correlational Question • Do the values of property X covary with the values of property Y? • If X is high, is it likely that Y will also be high? • If X is low, is it likely that Y will also be low? • High positive correlation, high negative correlation, little or no correlation • This technique tells you whether X and Y go together but it does not help in determining causality.

22. Randomization • Randomization – using a random assignment procedure to allocate “cases” to “conditions” • Each case must be equally likely to end up in any given condition • This procedure does not guarantee to rule out all other factors that could affect the results.

23. Validity of Findings • 4 types of validity • Internal validity • How close can you come to asserting that the presence of X caused the altered level of Y? • Statistical conclusion validity • Is the given result due to chance? • Construct validity • How well defined are the theoretical ideas in the study? • External validity • How well will the results hold up for replication and how generalizable are the results?

24. Measure Techniques • Categorized by two questions • Who makes the record of the behavior? • Is the participant aware that his/her behavior is being recorded for research purposes?

25. 6 Major Types of Measures

26. Strengths and Weaknesses of the Measures • Self-reports • Strength - Low costs, does not require a lot of time to create and use • Weakness - Reactive • Trace measures • Strength – unobtrusive • Weakness – not very versatile, time consuming to gather

27. Strengths and Weaknesses of the Measures • Observations • Strength – relatively versatile • Weakness – can only be used on overt behavior, vulnerable to observer errors, can be reactive for visible observers, possible ethical issues for hidden observers • Archival records • Strength – good for researching the past, studies for extensive periods of time • Weakness – public records can be reactive, versatility is limited

28. Techniques for Manipulating Variables • Selection – selecting cases with desired values and assigning them to appropriate conditions • Direct Intervention – manipulating the specific variable directly

29. Techniques for Manipulating Variables • Inductions • 3 types • Misleading instructions • False feedback • Experimental confederates • Ethical issues

30. Concluding Comments • Results depend on methods. All methods have limitations. Therefore, any set of results is limited. • It is not possible to maximize all desirable features of different methods in any one study. • Each set of results must be interpreted in relation to other evidence relating to the same questions.