Methodology Matters: Doing Research in the Social and Behavioral Sciences Joseph E. McGrath

1. Methodology Matters: Doing Research in the Social and Behavioral SciencesJoseph E. McGrath Gary Suh Vesna Memisevic Informatics 231 Human-Computer Interaction

2. Outline • Introduction • Some Basic Features of the Research Process • Substantive Domain • Conceptual Domain • Methodological Domain • Research Methods as Opportunities and Limitations • Research Strategies: Choosing a Setting For a Study • Quadrant I: The Field Strategies • Quadrant II: The Experimental Strategies • Quadrant III: The Respondent Strategies • Quadrant IV: The Theoretical Strategies • Some Strategic Issues Informatics 231 Human-Computer Interaction

3. Outline • Study design • Comparison techniques • Baserates • The Correlation Question • The Difference Question • Randomization and “True experiments” • Sampling, allocation and statistical inference • Validity of findings • Potential classes of measures in social psychology • Strengths and Weaknesses of Types of Measures • Techniques for Manipulating Variables • Selection, direct intervention, induction • Conclusion Informatics 231 Human-Computer Interaction

4. Introduction • Doing research – the systematic use of some set of theoretical and empirical tools to try to increase our understanding of some set of phenomena or events • Some of the tools with which researchers in the social and behavioral sciences go about doing research (strategy, tactics, and operations) Informatics 231 Human-Computer Interaction

5. Some Basic Features of the Research Process Always involves bringing together three sets of things: • some content that is of interest, • some ideas that give meaning to that content, and • some techniques or procedures by means of which those ideas and content can be studied. Informatics 231 Human-Computer Interaction

6. Some Basic Features of the Research Process These three sets of things more formally, as three distinct, though interrelated domains: • The Substantive domain, from which we draw contents that seem worthy of our study and attention; • The Conceptual domain, from which we draw ideas that seem likely to give meaning to our results; and • The Methodological domain, from which we draw techniques that seem useful in conducting that research. Informatics 231 Human-Computer Interaction

7. Some Basic Features of the Research Process Informatics 231 Human-Computer Interaction

8. Substantive Domain • Phenomena and Patterns of phenomena are the object of our study • The phenomena of interest involve the states and actions of some human systems and the conditions and processes that give rise to and follow from those states and actions. Informatics 231 Human-Computer Interaction

9. Substantive Domain • Example – an individual casting a vote in a county election Informatics 231 Human-Computer Interaction

10. Conceptual Domain • Properties of the states and actions of those human systems that are the focus of study • Relations refer to any of a variety of possible ways in which two or more elements can be connected • Examples – causal connections, logical relations, chronological relations Informatics 231 Human-Computer Interaction

11. Conceptual Domain • Examples – attitude, cohesiveness, power, social pressure, status Informatics 231 Human-Computer Interaction

12. Methodological Domain Basic sets of elements or “tools” by which social and behavioral scientists systematically gather empirical information: • Measuring • Manipulating • Controlling • Distributing Impact Informatics 231 Human-Computer Interaction

13. Methodological Domain • Measuring • For assessing the state or magnitude of some property of some actors-behaving-in-context so that the researcher can determine what value or level that feature has for each “case” to be studied • Examples – questionnaire, rating scale, personality test, instruments for observing and recording communications, techniques for assessing the quality of some products resulting from individual or group task performance Informatics 231 Human-Computer Interaction

14. Methodological Domain • Manipulating • Techniques for manipulating some property of an actor-behavior-context • Experimental manipulation – making a feature have one particular predetermined value or level for certain “cases” to be studied and another specific preordained value or level for certain other “cases,” so that the effect of differences in that property can be assessed by comparing those two sets of “cases” Informatics 231 Human-Computer Interaction

15. Methodological Domain • Manipulating • Social psychologists have tried to manipulate features of the systems they study by a number of techniques, such as: (a) giving instruction to participants (b) imposing constraints on features of the environment (c) selecting materials for use (d) giving feedback about prior performances (e) using experimental confederates • (More is said about techniques for manipulating variables near the end of this chapter) Informatics 231 Human-Computer Interaction

16. Methodological Domain • Controlling • A set of techniques for controlling the impact of features that are important but that you are not going to measure or manipulate in a particular study • These include: • Techniques for experimental control – you make certain features take the same predetermined value for all cases in the study • Techniques for statistical control – you try to nullify the effects of variations in a given property within a study by “removing” those variations by statistical means • Techniques for distributing the impact – so that such impact can be taken into account in interpretation of results Informatics 231 Human-Computer Interaction

17. Methodological Domain • Distributing Impact • Techniques for distributing the impact of a number of features of the system and its context-without directly manipulating or controlling any one of them-so that such impact can be taken into account in interpretation of results • Randomization – the most prominent means; refers to procedures for the allocation of “cases” among various conditions within the study Informatics 231 Human-Computer Interaction

18. Research Methods as Opportunities and Limitations • Methods enable but also limit evidence. • All methods are valuable, but all have weaknesses or limitations. • You can offset the different weaknesses of various methods by using multiple methods. • You can choose such multiple methods so that they have patterned diversity; that is so that strengths of some methods offset weaknesses of others. Informatics 231 Human-Computer Interaction

19. Research Methods as Opportunities and Limitations The fundamental principle, in behavioral and social science is that credible empirical knowledge requires consistency or convergence of evidence across studies based on different methods. Informatics 231 Human-Computer Interaction

20. Research Strategies: Choosing a Setting For a Study • When you gather a batch of research evidence, you are always trying to maximize three desireable features or criteria: A. Generalizability of the evidence over the populations of Actors. B. Precision of the measurement of the behaviors that are being studied (and precision of control over extraneous factors that are not being studied). C. Realism of the situation or Context within which the evidence is gathered, in relation to the contexts to which you want your evidence to apply. Informatics 231 Human-Computer Interaction

21. Research Strategies: Choosing a Setting For a Study • Although you always want to maximize all three of these criteria, A, B and C simultaneously, you cannot do so. • This is the fundamental dilemma of the research process. Informatics 231 Human-Computer Interaction

22. Research Strategies: Choosing a Setting For a Study Informatics 231 Human-Computer Interaction

23. Quadrant I: The Field Strategies Informatics 231 Human-Computer Interaction

24. Quadrant I: The Field Strategies • The two research strategies in quadrant I are the Field Study and the Field Experiment. • Field study – the researcher sets out to make direct observations of “natural”, ongoing systems, while intruding on and disturbing those systems as little as possible. • Field experiment – also works within an ongoing natural system as unobtrusively as possible, except for intruding on that system by manipulating one major feature of that system. Informatics 231 Human-Computer Interaction

25. Quadrant I: The Field Strategies • The essence of both of the strategies in quadrant I, the field study and the field experiment, is that the behavior system under study is “natural”, in the sense that it would occur whether or not the researcher were there and whether or not it were being observed as part of a study. Informatics 231 Human-Computer Interaction

26. Quadrant II: The Experimental Strategies Informatics 231 Human-Computer Interaction

27. Quadrant II: The Experimental Strategies • Laboratory experiment – the investigator deliberately concocts a situation or behavior setting or context, defines the rules for its operation, and then induces some individuals or groups to enter the concocted system and engage in the behaviors called for by its rules and circumstances. • Experimental simulation – the researcher attempts to achieve much of the precision and control of the laboratory experiment but to gain some of the realism (or apparent realism) of field studies. Informatics 231 Human-Computer Interaction

28. Quadrant II: The Experimental Strategies • The two strategies in Quadrant II, in contrast to those of Quadrant I, involve concocted rather than natural settings. • The laboratory experiment and the experimental simulation are strategies that involve “actor-behavior-context” systems that would not exist at all were it not for the researcher’s interest in doing the study. Informatics 231 Human-Computer Interaction

29. Quadrant III: The Respondent Strategies Informatics 231 Human-Computer Interaction

30. Quadrant III: The Respondent Strategies • Sample survey – the investigator tries to obtain evidence that will permit him or her to estimate the distribution of some variables, and/or some relationships among them, within a specified population • Examples – public opinion surveys on voting intentions, political preferences, buying intentions Informatics 231 Human-Computer Interaction

31. Quadrant III: The Respondent Strategies • The strategies of Quadrant III concentrate on the systematic gathering of responses of the participants to questions or stimuli formulated by the experimenter, in contrast to the observation of behaviors of the participants within an ongoing behavior system • Studies are usually done under “neutral” conditions of room temperature, lighting, chair comfort to nullify any effects of the behavior setting or context on the judgments that are the topic of study. Informatics 231 Human-Computer Interaction

32. Quadrant IV: The Theoretical Strategies Informatics 231 Human-Computer Interaction

33. Quadrant IV: The Theoretical Strategies • Formal theory – the researcher focuses on formulating general relations among a number of variables of interest • Computer simulation – a complete and closed system that models the operation of the concrete system without any behavior by any system participants Informatics 231 Human-Computer Interaction

34. Quadrant IV: The Theoretical Strategies • The inclusion of these two strategies reminds us of the importance of the theoretical side of the research process. • One of the more powerful general strategies for research is the simultaneous use of one of the theoretical strategies (say, the formulation of a general theory) and one of the empirical strategies (for example, a laboratory experiment). Informatics 231 Human-Computer Interaction

35. Some Strategic Issues Does the material, as presented, properly reckon with the strengths and weaknesses of the research strategies it encompasses? To what extent is the research evidence on each problem or topic based on use of only a single research strategy, and therefore limited by the weaknesses of that strategy; and to what extent is that body of evidence based on use of multiple, complementary strategies, with agreement or convergence among the findings attained via the different strategies? 35 Fall 2007 Informatics 231 Human-Computer Interaction

36. Study design • Gathering observations • Aggregation and partitioning • Comparison on the data set Informatics 231 Human-Computer Interaction

37. Study design • Comparison depends on: • What’s included in the study (what phenomena, what properties, what model of treatment for variables) • What system works (which substantive system we study, paradigms, strategies) • What conceptual relations have been posted (which properties are linked) • What comparison techniques are available Informatics 231 Human-Computer Interaction

38. Comparison Techniques • Three base forms: • Baserates (how often?) • Correlations (are properties related; do they occur together?) • Differences (comparison or difference) Informatics 231 Human-Computer Interaction

39. Base rates • How often Y occurs in the general case, as a basis for deciding whether the rate of Y in some particular case is “notably” high or low • Problems - difference in interpretation - various political, economic and social issues Informatics 231 Human-Computer Interaction

40. The Correlation Question • Is there covariation in the values of two properties or features of system? • Correlations: High – Low; Positive – Negative; Zero • Linear or nonlinear relation between two or more variables • Can asses conceptual relations that imply covariation between two or more variables, but cannot asses any conceptual relations that are causal in their implications Informatics 231 Human-Computer Interaction

41. The Difference Question • Whether Y is present under condition where X is present and whether Y is absent when X is absent? • Samples are separated in two groups, one with members which have performance X, and another with members who don’t have performance X. • Some set of tasks is given to both of groups, and average task performance is compared for both groups. • Groups must be comparable on factors that might affect task performances! Informatics 231 Human-Computer Interaction

42. Randomization and True Experiments • Randomization = using random assignment procedure to allocate cases to conditions (in order to remove artifacts and observations that happen by chance) • In order for the study to be called a “true experiment”, study must include some randomization • Random allocation procedure doesn’t guarantee an equal distribution of any of the potential factors among the conditions being compared Informatics 231 Human-Computer Interaction

43. Randomization and True Experiments • Possible problems: • Reduce the scope of study, as some variables are hold constant, and therefore experimental variables will occur only at a few levels • Reduce the realism of context of your study, designed the tasks serves us - not the participants' - purposes. Informatics 231 Human-Computer Interaction

44. Sampling, Allocation and Statistical Inference • The way we choose cases which will be included in our study (from larger population of potential cases) effect credibility of the evidence resulting from the study • How to choose right nature of the “random sample” population? Informatics 231 Human-Computer Interaction

45. Sampling, Allocation and Statistical Inference • We need to use random sample in that way that we can apply results of the study to the population of which cases constitute a random sample. Informatics 231 Human-Computer Interaction

46. Sampling, Allocation and Statistical Inference • Sampling case procedure: which cases from larger population will be included in study. • Allocation case procedure: which condition every given cases (already selected as a part of the study) will be assigned to • Random refers to procedure, not outcome! Informatics 231 Human-Computer Interaction

47. Sampling, Allocation and Statistical Inference • Size of the samples? • The larger the number the more distributed those cases will approach the idealized random distribution • Uneven distribution doesn’t occur often if only chance is operating. Informatics 231 Human-Computer Interaction

48. Validity of Findings • Four different types of validity (Cook & Campbell): • Statistical conclusion validity • Internal validity • Construct validity • External validity Informatics 231 Human-Computer Interaction

49. Validity of Findings • Statistical conclusion validity: • Difference arisen just by chance? • Relationship between cause and effect • Internal validity: • How close can you come to asserting that the present of X caused the altered level of Y values? • Difference in Y associated with a difference in X does not necessarily imply a causal role for X • Are there other factors which may have been covary with X and they, rather than X, might have produced the change in Y Informatics 231 Human-Computer Interaction

50. Validity of Findings • Construct validity: • How well defined are the theoretical ideas in our study? • How clearly understood are the conceptual relations being explored? • How well are mapping of concepts and relations • External validity: • How confident you can be that your findings will hold upon replication or how general are your our findings. • How confident you can be that you can make predictions ? Informatics 231 Human-Computer Interaction