Experimental Design

1. Experimental Design Week 9 Lecture 1

2. Agenda • Purpose of experimental design • Key elements in experimental design • Various types of experimental design • Causal-comparative design • Field and controlled laboratory experiment ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

3. Purpose of experimental design • Examine the possible influences that one factor or condition may have on another factor or condition • Build on positivist approach • Research questions appropriate for an experiment • Issues that have a narrow scope or small scale • Can rarely address question that may require looking at conditions across an entire society or across decade • Small number of participants (subjects) compared with survey research ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

4. Independent variables • Independent variable • Cause • Research “measure” (manipulate) independent variable by creating a condition or situation • Manipulation of independent variable create different treatments. • Event manipulation • Affecting the independent variable by altering the events that subjects experience • Presence versus absence • Instructional manipulation • Varying the independent variable by giving different sets of instructions to the subjects ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

5. Dependent variables • Effect (outcome) • Physical conditions, behaviors, attitudes, feelings, or beliefs of subjects that change in response to a treatment. • How to measure • Questionnaire, interviews • Observation • Test • Direct outcome ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

6. The importance of control • Internal validity -- The extent to which we can accurately state that the independent variable produced the observed effect ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

7. Strategies to achieve control • Keep some things constant • What are variables that need to be held constant in most experiments? • Include a control group • Treatment group (experimental group) • Between-subjects design • Randomly assign people to groups • Use matched pairs • Matched-subject design • Expose participants to both or all treatment conditions • Within group design ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

8. 7 3 7 3 10 3 3 5 1 8 2 7 10 7 1 6 4 10 9 2 6 8 4 9 5 2 2 2 8 8 5 9 5 9 4 1 6 Random assignment Randomly assign one member of each pair to each group 1 4 10 6 treatment control DV DV Between and matched-subjects design ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

9. Steps in conducting an experiment • State hypotheses • Decide on an experimental design • Decide the way to manipulate independent variables • Develop a valid and reliable measure for dependent variable • Pilot testing the treatment and dependent variable measures • Recruit subjects (or locate cases) • Assign subject to groups • Introduce treatment to treatment groups • Gather data for measure of the dependent variables • Hypotheses testing ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

10. Mini-workshop • Identify the IV and DV in the following hypotheses • Adults find it easier to remember a list of meaningful words than to remember a list of nonsense syllables • Nationality of salesperson will affect customers’ intention to buy a foreign made product • Perceptions of the characteristics of the “good” or effective teacher are in part determined by the perceiver’s attitudes toward education. • Determine how might you manipulate the IV and measure the DV ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

11. Design elements and notations • Observations or Measures • Symbolized by Oin design notation • Treatments or Programs • A manipulated independent variable • Symbolized by X in design notation • Time • Used to indicate the time you make the observation or take the measure • Time moves from left to right. Elements that are listed on the left occur before elements that are listed on the right • Assignment to group • R = random assignment • N = Nonequivalent groups • C = assignment by cutoff ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

12. Vertical alignment shows two Pretests are measured at same time Classic true experimental design • pretest-posttest • Treatment Versus control group • Randomized • Experimental design http://trochim.human.cornell.edu/kb/desintro.htm ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

13. Pre-experimental design • One-shot case study • Result usually not valid • Seldom used in serious research • One-group pretest-posttest design • Some effect after the treatment • Can not rule out alternative explanations for the effect • Static group comparison • Any posttest outcome difference between the groups could be due to group differences prior to the experiment instead of to the treatment ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

14. Quasi-experimental design • Non-equivalent control group design • What does the term “nonequivalent” mean? • Assignment to group was not random • The groups may be different prior to the study • Susceptible to internal validity threat of selection • Selection bias: groups were not comparable before the study ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

15. Time-series design • Interrupted time-series design • Why do we use time-series design • Difficult to find an equivalent group of subjects to serve as a control group • Single group pretest-posttest design is susceptible to lots of internal validity threat ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

16. An example • Research question: • The impact of distance learning on university students’ learning outcome and cognitive skills • Units of analysis – college students • Independent variables • Distance learning (2 level) • Dependent variables: • Learning outcome • Cognitive skills ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

17. An example (cont) • Hypotheses • H1. Students using distance learning system will achieve better learning outcome than students with traditional classroom learning do • H2. The cognitive skills of students using distance learning system will be lower than those of students with traditional classroom learning • Experimental design? • Subject recruitment? • Manipulation of independent variable • Measurement of dependent variables ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

18. Causal-comparative designs • What can we do if we are not able to manipulate the independent variable • Consider the following hypotheses • Girls who follow science courses in Year 12 are more aggressive than girls following non-science courses. • Working class children will learn nonsense syllables slower than middle-class children • Causal-comparative designs provide the means by which a research can examine how specific IVs (personal trait, history of family violence, etc) affect the dependent variable (s) of interest ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

19. Causal-comparative designs (cont) • Look at conditions that have already occurred and then collects data to investigate a possible relationship between these conditions and dependent variables of interest (ex post facto research) • Difference between causal-comparative design and correlational design • Less able to draw firm conclusion about cause and effect ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney

20. Field and controlled laboratory experiment • Field experiment • Experiments conducted in real-life or field settings • Researcher has less control over the experimental condition • Greater external validity but lower internal validity • Controlled laboratory experiment • Conducted under controlled conditions of a laboratory • Greater internal validity but lower external validity • Practical consideration • Planning and pilot testing • Instruction to subjects • Post experiment interview ISYS3015 Analytical Methods for IS Professionals School of IT, The University of Sydney