Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab

1. Quantitative Research Methods (II) Dr Chen Wenli Learning Sciences and Technologies AG Learning Sciences Lab National Institute of Education

2. Outline • Logic of quantitative research • Constructing hypothesis • Types of quantitative research methods • Survey research • Experimental research • Single-subject research • Casual-comparative research • Quantitative content analysis • Validity and reliability in quantitative research

3. Experimental Research • Characteristics of experimental research • Experimental research design • Experimental design • Quasi-experimental design • Factorial design • Validity of experimental research • Control of extraneous variables

4. Experimental Research • Researcher applies some treatments to subjects for an appropriate length of time and then observes the effect of the treatments on the subjects by measuring response variables • IV (experimental or treatment variable) • a condition or set of conditions applied to subjects • DV (response, criterion or outcome Variable) • results or outcome on the subjects

5. Examples • Quality of learning with an active versus passive motivational set (Benware & Deci, 1984) • Comparison of computer-assisted cooperative, competitive, and individualistic learning (Johnson, Johnson, & Stanne, 1986) • The effect of a computer simulation activity versus a hands-on activity on product creativity in technology education (Kurt, 2001) • The effect of language course taught with online supplement material (Shimazu, 2005)

6. Characteristics • The only type of research that directly attempts to influence a particular variable • The only type that, when used properly, can really test hypotheses about cause-and-effect relationships. • Enable researchers to go beyond description and the identification of relationships, to at least a partial determination of what causes them • 3 characteristics of experimental research

7. Manipulation of IV • Researcher manipulate the IV • Decide the nature of treatment/intervention (what is going to happen to the subjects of the study) • To whom it is to be applied • To what extent • When, where and how

8. Comparison of Groups • At least 2 conditions are compared to assess the effect(s) of particular conditions or “treatments” (IV) • Experimental group (receive a treatment of some sort) • Control group (no treatment) or comparison group (receive different treatment) • IV may be established in several ways: • Presence VS absence of a particular form • One form of variable VS another • Varying degrees of the same form

9. Randomization • Random assignment of subjects to groups • an important ingredient in the best kinds of experiments • every individual who is participating in the experiment has an equal chance of being assigned to any of the experimental or control conditions being compared • It takes place before the experiment begins • Allows the researcher to form groups that are equivalent • Eliminate the threat of extraneous, or additional variables that might affect the outcome of the study

10. Commonly Used Notation • X1 = treatment group • X2 = control/comparison group • O = observation (pretest, posttest, etc.) • R = random assignment

11. Weak Experimental Designs • One-shot case study design • a single group is exposed to a treatment or event, and its effects assessed. • One-group pretest-posttest design • a single group is measured or observed both before and after exposure to a treatment. X O Technology Attitude scale to measure interest O X O Pretest Treatment Post test

12. True Experimental Designs • Randomized posttest-only control group design • involves two groups formed by random assignment and receiving different treatments • Randomized pretest-posttest control group design • differs from the randomized posttest-only control group only in the use of a pretest Treatment group R X1 O Control group R X2 O Treatment group R O X1 O Control group R O X2O

13. True Experimental Designs • Randomized Solomon four-group design • involves random assignment of subjects to four groups, with two being pretested and two not. Treatment group R O X1 O Control group R O X2 O Treatment group R X1 O Control group R X2 O • Better control the threat to internal validity • Drawback—requires twice as many participants

14. Quasi-Experimental Designs • Used in place of experimental research when random assignment to groups is not feasible • Posttest-only design with nonequivalent groups • Pretest-posttest design with nonequivalent groups: Treatment group X1 O Control group X2 O Treatment group O X1O Control group O X2O

15. Quasi-Experimental Designs • Counterbalanced design: all groups are exposed to all treatments, but in a different order • the order in which the groups receive the treatments should be determined randomly • the number of groups and treatments must be equal • Comparing the average scores fro all groups on the posttest for each treatment Group I X1O X2 O X3 O Group II X3O X1 O X2 O Group III X2O X3 O X1 O

16. Quasi-Experimental Designs • Time-series design: involves repeated measurements or observations over time (until scores are stable ), both before and after treatment. O OOO X O OOO • Uses a single group of participants • Examines possible changes • over time Study B Study A X

17. Factorial Designs • Factorial designs extend the number of relationships that may be examined in an experimental study. Treatment R O X1g1 O Control R O X2g1 O Treatment RO X1g2 O Control RO X2g2 O • Incorporates two or more factors • The additional factor could be treatment variable or subject characteristics • Enables researcher to detect differential differences (effects apparent only on certain combinations of levels of IVs)

18. A 2 X 2 factorial design… Boy Girl Group 2 Group 1 Traditional Game-based learning Group 3 Group 4

19. A 2 X 2 factorial design No interaction between factors Interacting factors Game -based Attitudes toward learning Game -based Attitudes toward learning Traditional Traditional Girl Boy Girl Boy

20. Validity • Validity: the experiment tests the variable(s) that it purports to test • If threats are not controlled for, they may introduce error into the study, which will lead to misleading conclusions • Threats to validity… • Internal: factors other than the IV that affect the DV • External: factors that affect the generalizability of the study to groups and settings beyond those of the experiment

21. Threats to Internal Validity • History • Uncontrolled event that occur during the study that may have an influence on the observed effect other than the IV • Maturation • Factors that influence a participant's performance because of time passing rather than specific incidents (e.g., the physical, intellectual, and emotional changes that occur naturally) • Test practice • The effects of participants taking a test that influence how they score on a subsequent test • Instrumentation • Influences on scores due to calibration changes in any instrument that is used to measure participant performance • Statistical regression • Problem that occurs when participants have been assigned to particular group on the basis of atypical or incorrect scores.

22. Threats to Internal Validity • Bias in group composition • Systematic differences between the composition of groups in addition to the treatment under study. • Experimental mortality • A differential loss of participants • Hawthorne effect • Change in the sensitivity or performance by the participants that may occur merely as a function of being a part of the study • Novelty effect • Participant interest, motivation, or engagement increases simply because they are doing something different • Placebo effect • The participants receive no treatment but believe they are

23. Threats to External Validity • Population-sample differences • The degree to which the participants in a study are representative of the population to which generalization is desired • Artificial research arrangements • The degree that a research setting deviates from the participant's usual routine • Multiple-treatment interference • More than one treatment is administered to the same participants and results in cumulative effects that may not be similar to the outside world and may threaten generalization of the results • Treatment diffusion • The situation when different treatment groups communicate with and learn from each other

24. Validity of Different Experimental Designs

25. Control of Extraneous Variables • Confounding: • the fact that the effects of the IV may intertwine with extraneous variables, such that it is difficult to determine the unique effects of each variable • Common ways to control for extraneous variables • Randomization • Holding certain variables constant • Matching • Comparing homogeneous groups or subgroups • Analysis of covariance (ANCOVA)

26. Single-Subject Research • Most commonly used to study the changes in behavior an individual exhibits after exposure to a treatment or intervention of some sort. • Can be applied in settings where group designs are difficult to put into play. • Involves extensive collection of data on one subject at a time. • Primarily use line graphs to present their data and to illustrate the effects of a particular intervention or treatment. • Adaptations of the basic time-series design

27. Single-Subject Research • A-B design • …baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation O OO X O X O X O baseline treatment phase phase A | B

28. Single-Subject Research • Reversal (A-B-A) design • …baseline measurements (O) are repeatedly made until stability is established, then the treatment (X) is introduced and an appropriate number of measurements (O) are made during treatment implementation, followed by an appropriate number of baseline measurements (O) to determine stability of treatment (X) O OO X O X O X O OO baseline treatment baseline phase phasephase A | B | A

29. Other Single-Subject Research Designs • A-B-A-B design • 2wo baseline periods are combined with two treatment periods • B-A-B design • Used when an individual's behavior is so severe or disturbing that a researcher cannot wait for a baseline to be established • A-B-C-B design: • "C" condition refers to a variation of the intervention in the "B" condition. The intervention is changed during the "C" phase typically to control for any extra attention the subject may have received during the "B" phase.

30. Threats to Validity in Single Subject Research • Internal Validity • length of the baseline and intervention conditions • the number of variables changed when moving from one condition to another • the degree and speed of any change that occurs • whether or not the behavior returns to baseline levels • the independence of behaviors • the number of baselines • External Validity • weak when it comes to generalizability • It is important to replicate single-subject studies to determine whether they are worthy of generalization.

31. Controlling Threats in Single-Subject Studies • Single-subject designs are most effective in controlling for subject characteristics, mortality testing, and history threats. • They are less effective with location, data collector characteristics, maturation, and regression threats. • They are especially weak when it comes to instrument decay, data-collector bias, attitude, and implementation threats.

32. Causal-Comparative Research • Explores the possibility of cause-and-effect relationships when experimental and quasi-experimental approaches are not feasible • Differs from experimental and quasi-experimental research • IV is not manipulated (not ethical or not possible) • Focuses first on the effect, then tries to determine possible • Relationships can be identified in causal-comparative study, but causation cannot be fully established.

33. Steps in Causal-Comparative Research • Formulating a problem • Identify and define the particular phenomena of interest, and then to consider possible causes for, or consequences of, these phenomena. • Selecting a sample • Define carefully the characteristic to be studied and then to select groups that differ in this characteristic. • Instrumentation • No limits to the kinds of instruments that can be used • Design • Select two groups that differ on a particular variable of interest and then comparing them on another variable or variables.

34. Threats to Internal Validity in Causal-Comparative Research • Weaknesses : • lack of randomization • Inability to manipulate an IV • A major threat: the possibility of a subject selection bias. • The procedures used to reduce this threat • matching subjects on a related variable • creating homogeneous subgroups • the technique of statistical matching. • Other threats to internal validity • Location • Instrumentation • Loss of subjects.

35. Data Analysis in Causal-Comparative Studies • The first step: construct frequency polygons. • Means and SDare usually calculated if the variables involved are quantitative. • The most commonly used test is a t-test for differences between means. • ANCOVA is particularly useful in causal-comparative studies. • The results of causal-comparative studies should always be interpreted with caution, because they do not prove cause and effect.

36. Common quantitative measure in learning and education • Learning gain • Post-pre • (post-pre)/(1-pre) (Hake’s gain) • Adjusted post score (through ANCOVA) • Learning efficacy • Does it help reduce time spent for problem solving? • User’s attitude • Teachbacks • How well learner can teach back?

37. Quantitative Content Analysis • Content analysis is a quantitative research instrument for a systematical and intersubjective description of content • A form of textual analysis *usually* • Categorizes chunks of text according to Code • Based on the principles of social science of “measuring and counting” • Reduces the complexity of content as it brings out the central patterns of the coverage • One objective is to examine large amounts of content with statistic methods

38. Rough History • Classical Content Analysis • Used as early as the 30’s in military intelligence • Analyzed items such as communist propaganda, and military speeches for themes • Created matrices searching for the number of occurrences of particular words/phrases • (New) Content Analysis • Moved into Social Science Research • Study trends in Media, Politics, and provides method for analyzing open ended questions • Can include visual documents as well as texts • More of a focus on phrasal/categorical entities than simple word counting

39. Procedure

40. The Sample • The sample • Which types of content? • Which period? • Which characteristics? • Elements of the research instrument • Sampling units • Units of analysis: unit of the content on which our “measurements” are based. The categories describe the properties of the media content which is relevant to our research question

41. Validity in Quantitative Research • Definition: the extent to which any measuring instrument measures what it is intended to measure • Types of validity • Construct Validity: examines the fit between the conceptual definitions & operational definitions of the variables • Content Validity : verifies that the method of measurement actually measures the expected outcomes. • Predictive Validity : determines the effectiveness of the instrument as a predictor of a future event • Statistical Conclusion Validity: concerned with whether the conclusions about relationships and/or differences drawn from statistical analysis are an accurate reflection of the real world

42. Reliability in Quantitative Research • Definition: refers to the accuracy and consistency of information obtained in a study; important in interpreting the results of statistical analyses; and refers to the probability that the same results would be obtained with different samples (generalizability) • 3 common methods to check reliability • test-retest method • administering the same instrument twice to the same group of individuals after a certain time interval has elapsed. • equivalent-forms method • administering two different, but equivalent, forms of an instrument to the same group of individuals at the same time. • internal-consistency method • comparing responses to different sets of items that are part of an instrument.

43. Summary • Logic of quantitative research • Constructing hypothesis • Types of quantitative research methods • Survey research • Experimental research • Single-subject research • Casual-comparative research • Others • Validity and reliability in quantitative research

44. Thank You 