Variables and Control

1. Variables and Control Basic Principles of Research Design

2. Overview of Tonight’s Lecture • Variables and Control • 2 variables we can control (subject + situation) • Research designs are rules about controls and variables • evaluated in terms of Validity (answers the question set to address) • Internal Validity (relevant variables are controlled to answer question); 8 internal validity factors • External Validity (findings can be generalized to other settings and populations); 4 external validity factors • Observations are evaluated in terms of Reliability

3. Basic Principles of Research Design • One of the most important aspects of research in CDIS is the careful selection of the variables that are to be controlled or systematically varied. • After a problem has been identified and the purpose specified, there must be careful consideration of which variables are most relevant to the problem and the purpose.

4. 2 Purposes of Research Designs • Answer the research question • Reduce or eliminate contamination of extraneous variables

5. In experimental designs, this means …. • Manipulate the IV in order to answer the question • Arrange the experiment so that extraneous variables are controlled

6. In descriptive designs, this means …. • Select the variables for observation in order to answer the question • make the observations in a systematic and unobtrusive manner so that the criterion variables are not confounded by the extraneous variables

7. RESEARCH DESIGNS are the rules for observation - research designs are rules about controls and variables and controlled variation

8. Research design is described in terms of: • systematic variation of the IV • observation of the effects of the IV on the DV • systematic control of other variables that might produce variation in the DV

9. Researchers systematically vary the IV by controlling its variation • a detailed explanation of controls and variables is essential to understanding the basic principles or rules of research design

10. Control and Variability • Control is a restriction of natural variation • represent opposite extremes of ORDER and DISORDER • both are essential to research • IV is controlled variation

11. 2 Controlled Variables in Human Communication Research • Subject (participant) variables • characteristics of subjects • Situation variables • characteristics of the situation in which the observations are made

12. In carefully controlled studies, subject and situation variables are controlled. - only the phenomena of interest are allowed to vary. - Control of the subject and situation variables permits researchers to study the effects of the variables in which they are interested.

13. Controls are essential for achieving the purposes of research. - if the variables under study are not isolated by controlling the relevant variables, alternative explanations of the findings cannot be ruled out. -the extent to which each kind of variation is controlled depends on the research design.

14. Control of Subject Variables • The type of communication disorder to be studied is controlled by the selection of subjects who have that type of disorder • Subject variables (e.g., age, gender, history of tx, severity) are controlled by holding them constant • If these variables were not controlled, the effects of the disorder might be confounded by the effects of the uncontrolled variables

15. Control of Situation Variables • The research design also controls the situation variables • These include variables such as time of day and instructions given to subjects • The need for situational controls varies with the type of design • EX: If you’re interested in determining how sounds are detected in real-life situations, you will not control background noise.

16. Criteria for Evaluating Research Designs • A research design is evaluated in its ability to achieve its 2 purposes (answer the question AND reduce/eliminate contamination of extraneous variables) in terms of INTERNAL and EXTERNAL VALIDITY • Therefore, research designs are evaluated in terms of VALIDITY (internal and external)

17. Internal Validity • Ability to isolate effects of the IV • means that we can believe that the manipulation of the IV did cause the observed change in the DV • a study is internally valid when relevant variables have been controlled and the only variables that affect the DV’s are the IV’s

18. Internal Validity (con’t) • When uncontrolled variables can affect the DV, the effects of the IV are confounded with the effects of the uncontrolled variables • internal validity indicates that the research design accomplished what it was suppose to accomplish, i.e., answer the question.

19. External Validity • Means that the findings from the study can be generalized to other populations and settings • external validity is the extent to which the effects of the IVs on the DVs in the research situation apply to the natural settings. • Indicates the degree to which the findings from the study can be generalized • if findings from a study do not external validity, they may not be of any practical use or value.

20. The primary thing in evaluating experimental research is whether you can believe with enough confidence that the change in the DV was caused by manipulation of the IV - the fewer the alternativeexplanations for the observed change in the DV, the greater the internal validity of the study.

21. 8 Factors that Affect Internal Validity • history • maturation • test-practice effects • instrumentation • statistical regression • differential selection of subjects • mortality • Hawthorne effect

22. History • Events that occur between the first and subsequent measurements in addition to the experimental variable (“in the gap”) • these are the events that occur outside the experimental setting and therefore are outside the control of the experimenter • EX: prior treatment received in another setting; history of ear infections

23. History (con’t) • Descriptive studies: longitudinal studies more susceptible to history effects than cross-sectional studies • Experimental studies: one group pretest-posttest design is susceptible to history effects

24. Maturation • Changes in the subjects that cannot be controlled by the experimenter (“time-tied”) • these changes may cause effects that are attributed incorrectly to the experimental variable • EX: age changes, changes in biological or psychological processes that take place over time; even short-term changes, such as changes in motivation, interest, attention

25. Solution to maturational effects in studies: • Keep experimental sessions relatively short • randomize or counterbalance conditions • randomly assign subjects to experimental and control groups

26. Test-Practice Effects • The effect of taking a test may influence the subjects’ performance when they retake the test • practice with the test, familiarity with the test items, reduction of test anxiety, etc. • pretest-posttest designs are especially vulnerable to this problem • must be aware of the effect of testing on behavior when evaluating findings from research

27. Instrumentation • Changes in the calibration of an instrument or changes in the observers or scorers may produce changes in the observed measurements. • EX: (a) faulty or changing calibration of equipment; (b) rating scales, questionnaires, attitude inventories that have not been standardized or are not reliable; (c) biases of judges or observers used to collect data for the study; (d) poor test environment which may contaminate observations due to distractions, noise, interruptions, etc.

28. Statistical Regression • The migration of extreme scores toward the mean • atypical scores become more typical (elastic band) • did change in performance occur because of tx or because of regression to the mean? • occurs because subjects elected on basis of their extreme scores. • Solution: random selection of subjects and random assignment of subjects to groups

29. Differential Selection of Subjects • Inherent differences between subjects assigned to groups may account for the treatment effects rather than the treatment itself. • EX: SES, IQ • Solution: subjects should be equal on important dimensions prior to the study and systematic differences between comparison groups must be minimized with the exception of the criterion variable under study. Further, there should be no overlap between groups on the variable that presumably distinguishes the groups from one another.

30. Mortality • Differential loss of subjects between experimental and control groups or between comparison groups • EX: subjects who drop out may be different in important respects from those subjects who continue to participate in the study.

31. Hawthorne Effect • Changes in a subject’s behavior that occur because the subject knows he/she is participating in a study (“on stage”) • Solution: apply Hawthorne effect to control group; begin sessions with a brief period of acclimation to establish rapport.

32. Finally, you can have an interaction of any one or combination of the above factors that affect internal validity.

33. External Validity (Generalizability) • External validity is qualitatively different from internal validity. • If a study is internally invalid, then the results are meaningless. • If, however, the study is externally invalid, the results are only limited to the degree to which the internally valid results can be generalized • A study can be internally valid, but externally invalid. However, a study cannot be externally valid, but internally invalid

34. 4 Factors that Affect External Validity • Subject selection • reactive or interactive effects of pretesting • reactive arrangements • multiple treatment interference

35. Subject Selection • The representativeness of subjects selected from the population to which the researcher hopes to generalize the findings. • The relevant characteristics of the subjects that determine their degree of representativeness should be specified in the article to allow readers to evaluate the generality of the results to other subjects

36. Subject Selection (con’t) • Investigators should report what the relevant characteristics were in selecting the subjects AND what the characteristics were of the subjects on those relevant characteristics (e.g., age, gender, severity, etc.) • this factor affects the generalizability of findings to other subjects

37. Reactive or Interactive Effects of Pretesting • The degree to which a reactive pretest may interact with an IV in determining the subjects’ performance on the DV • that is, subjects who take a reactive pretest may react to an experimental treatment in a different way from people who have not been exposed to the pretest. • The test itself may change the subjects’ responses.

38. Reactive or Interactive Effects of Pretesting (con’t) • Therefore, the effect of pretesting may limit the generalization that can be made to people who have not been pretested • this factor affects the generalizability of findings to other measures and subjects

39. Reactive Arrangements • The degree to which the setting of the research is reactive or interacts with the IV in determining the subjects’ performance on the DV. • For example, the child may respond in a certain way with the clinician in the therapy room, but not in their classroom • this factor affects the generalizability of findings to other settings.

40. Multiple Treatment Interference • The degree to which various parts of multiple treatment interact with each other in determining the subjects’ performance on the DV. • This is likely to occur when more than one treatment is administered to the same subjects • this factor affects the generalizability of findings to other treatments. • The results of a multiple treatment study can only be generalized to people who would receive the same sequence and number of treatments.

41. The 4 factors that affect external validity can affect generalization in 1 of 4 ways: • Generalizability to other subjects • generalizability to other settings • generalizability to other measurements • generalizability to other treatments

42. In sum…. • Research designs are evaluated in terms of validity (internal and external) • Observations are evaluated in terms of reliability • a measure is valid to the extent that it measures what it claims to measure, and reliable to the extent that it is consistent and repeatable.

43. Reliability • Refers to the precision of measurement and can be assessed by examining the consistency or stability of a test or measure. • Measurements of DVs are reliable to the extent that the same measurements would be obtained if the study were repeated.

44. Reliability of measurements can be estimated in several ways • Test-retest reliability • split-half reliability • alternate form reliability • observer reliability

45. Test-Retest Reliability • Assessed by obtaining the same measurement twice and comparing the two measurements

46. Split-Half Reliability • Assessed when measurements consist of a series of items • the items are split in half and the resulting measurements are compared.

47. Alternate Form Reliability • Assessed when alternate forms of measurement are compared

48. Observer reliability • Assessed when the judgement of two or more observers are compared.

49. The design is evaluated in terms of validity and the observations are evaluated in terms of reliability.