Experimental Design

1. Experimental Design Playing with variables

2. The nature of experiments • allow the investigator to control the research situation so that causal relationships among variables may be evaluated • One variable is manipulated and its effect upon another variable is measured, while other variables are held constant

3. So… you’ve decided to do an experiment • Decisions… decisions… decisions

4. Decision 1: Independent Variable? • value is changed or altered independently of other variables • hypothesized to be the causal influence • categorical or continuous (?) Experimental Treatments: • alternative manipulations of the Independent Variable

5. Experimental and Control Groups • Control Group • Experimental Groups • there can be more than one treatment level of the Independent Variable (basic or factorial) • there can be more than one IV Experimental Groups

6. Decision 2: Dependent Variable • The criterion or standard by which the results are judged • It is presumed that changes in the Dependent Variable are the result of changes in one or more Independent Variable • the choice of Dependent Variable determines the type of answer that is given to the research question

7. Decision 3: Test units/unit of analysis • The subjects or entities whose responses to the experimental treatment are being measured • People are the most common test unit in business research

8. Decision 4: Extraneous variables • A number of extraneous or “other” variables may affect the dependent variable and distort the results Conditions of constancy: • When extraneous variables cannot be eliminated we strive to hold Extraneous Variables constant for all subjects

9. But, what about ___________? • Problems… problems…

10. IMPACT OF THE RESEARCH SITUATION Demand Characteristics: experimental design procedures that unintentionally hint to subjects about the experimenter’s hypothesis • rumour • instructions • status and personality of researcher • unintentional cues from experimenter • experimental procedure itself • Setting: Field versus Laboratory

11. Field versus Laboratory • Field experiments: usually used to fine-tune strategy and determine sales volume • Laboratory: used when control over the experimental setting is more important

12. Experimental Design effects….

13. The Hawthorne effect • Subjects perform differently just because they know they are are experimental subjects • Western Electric’s Hawthorne Plant 1939 study of light intensity The Guinea Pig effect • exhibit the behaviour that they think is expected • Potential Solutions: • run experiment for a longer period • use a control group • Deception (?)

14. Experimental Treatment Diffusion • if treatment condition perceived as very desirable relative to the control condition, members of the control group may seek access to the treatment condition • Potential Solutions: -have control group in another site -of course, this introduces new variables!

15. John Henry Effect • legend of black railway worker • control group overcompensates • Potential Solutions: • don’t do threatening experiments • don’t set up obviously competitive situations • don’t tell control group that they are control group • conduct in another location somewhere else • unfortunately, produces new variable of different location, neighbourhood, etc.!

16. Resentful Demoralization of Control Group • Control group artificially demoralized if perceives experimental group receiving desirable treatment being withheld from it • Potential Solutions? • what about giving control group some perk to compensate? • don’t tell them they are control group! (but what about informed consent?)… Use of Placebo… use of blinding…

17. Getting control…. • Design decisions

18. Physical Control • Holding the value or level of extraneous variables constant throughout the course of an experiment. • Statistical Control • Adjusting for the effects of confounding variables by statistically adjusting the value of the dependent variable for each treatment conditions. • Design Control • Use of the experimental design to control extraneous causal factors.

19. Blinding • Blinding is utilized to control subjects knowledge of whether or not they have been given a particular experimental treatment • double-blind experiment • secrecy • but then violate principle of informed consent • screen out or balance number of placebo reactors in treatment & control groups

20. Sampling Who and How And How to Screw It up

21. Terms • Sample • Population (universe) • Population element • census

22. Why use a sample? • Cost • Speed • Sufficiently accurate (decreasing precision but maintaining accuracy) • More accurate than a census (?) • Destruction of test units

23. Stages in the Selection of a Sample Step 7: Conduct Fieldwork Step 2: Select The Sampling Frame Step 3: Probability OR Non-probability? Step 1: Define the the target population Step 6: Select Sampling units Step 5: Determine Sample Size Step 4: Plan Selection of sampling units

24. Step 1: Target Population • The specific, complete group relevant to the research project • Who really has the information/data you need • How do you define your target population

25. Bases for defining the population of interest include: • Geography • Demographics • Use • Awareness

26. Operational Definition • A definition that gives meaning to a concept by specifying the activities necessary to measure it. • “The population of interest is defined as all women in the City of Lethbridge who hold the most senior position in their organization.” • What variables need further definition?

27. Step 2: Sampling Frame • The list of elements from which a sample may be drawn. • Also known as: working population. • Examples?

28. Sampling Frame Error: • error that occurs when certain sample elements are not listed or available and are not represented in the sampling frame.

29. Sampling Units: • A single element or group of elements subject to selection in the sample. • Primary sampling unit • Secondary sampling unit

30. Error: Less than perfectly. representative samples. • Random sampling error. • Difference between the result of a sample and the result of a census conducted using identical procedures; a statistical fluctuation that occurs because of chance variation in the selection of the sample.

31. …Error • Systematic or non-sampling error. • Results from some imperfect aspect of the research design that causes response error or from a mistake in the execution of the research • Examples: Sample bias, mistakes in recording responses, non-responses, mortality etc,.

32. …Error • Non-response error. • The statistical difference between a survey that includes only those who responded and a survey that also includes those that failed to respond.

33. Step 3: Choice! • Probability Sample: • A sampling technique in which every member of the population will have a known, nonzero probability of being selected

34. Step 3: Choice! • Non-Probability Sample: • Units of the sample are chosen on the basis of personal judgment or convenience • There are no statistical techniques for measuring random sampling error in a non-probability sample. Therefore, generalizability is never statistically appropriate.

35. Classification of Sampling Methods Sampling Methods Probability Samples Non- probability Systematic Stratified Convenience Snowball Cluster Simple Random Judgment Quota

36. Probability Sampling Methods • Simple Random Sampling • the purest form of probability sampling. • Assures each element in the population has an equal chance of being included in the sample • Random number generators Sample Size Probability of Selection = Population Size

37. Advantages • minimal knowledge of population needed • External validity high; internal validity high; statistical estimation of error • Easy to analyze data • Disadvantages • High cost; low frequency of use • Requires sampling frame • Does not use researchers’ expertise • Larger risk of random error than stratified

38. Systematic Sampling • An initial starting point is selected by a random process, and then every nth number on the list is selected • n=sampling interval • The number of population elements between the units selected for the sample • Error: periodicity- the original list has a systematic pattern • ?? Is the list of elements randomized??

39. Advantages • Moderate cost; moderate usage • External validity high; internal validity high; statistical estimation of error • Simple to draw sample; easy to verify • Disadvantages • Periodic ordering • Requires sampling frame

40. Stratified Sampling • Sub-samples are randomly drawn from samples within different strata that are more or less equal on some characteristic • Why? • Can reduce random error • More accurately reflect the population by more proportional representation

41. How? 1.Identify variable(s) as an efficient basis for stratification. Must be known to be related to dependent variable. Usually a categorical variable 2.Complete list of population elements must be obtained 3.Use randomization to take a simple random sample from each stratum

42. Types of Stratified Samples • Proportional Stratified Sample: • The number of sampling units drawn from each stratum is in proportion to the relative population size of that stratum • Disproportional Stratified Sample: • The number of sampling units drawn from each stratum is allocated according to analytical considerations e.g. as variability increases sample size of stratum should increase

43. Types of Stratified Samples… • Optimal allocation stratified sample: • The number of sampling units drawn from each stratum is determined on the basis of both size and variation. • Calculated statistically

44. Advantages • Assures representation of all groups in sample population needed • Characteristics of each stratum can be estimated and comparisons made • Reduces variability from systematic • Disadvantages • Requires accurate information on proportions of each stratum • Stratified lists costly to prepare

45. Cluster Sampling • The primary sampling unit is not the individual element, but a large cluster of elements. Either the cluster is randomly selected or the elements within are randomly selected • Why? • Frequently used when no list of population available or because of cost • Ask: is the cluster as heterogeneous as the population? Can we assume it is representative?

46. Cluster Sampling example • You are asked to create a sample of all Management students who are working in Lethbridge during the summer term • There is no such list available • Using stratified sampling, compile a list of businesses in Lethbridge to identify clusters • Individual workers within these clusters are selected to take part in study

47. Types of Cluster Samples • Area sample: • Primary sampling unit is a geographical area • Multistage area sample: • Involves a combination of two or more types of probability sampling techniques. Typically, progressively smaller geographical areas are randomly selected in a series of steps

48. Advantages • Low cost/high frequency of use • Requires list of all clusters, but only of individuals within chosen clusters • Can estimate characteristics of both cluster and population • For multistage, has strengths of used methods • Disadvantages • Larger error for comparable size than other probability methods • Multistage very expensive and validity depends on other methods used

49. Classification of Sampling Methods Sampling Methods Probability Samples Non- probability Systematic Stratified Convenience Snowball Cluster Simple Random Judgment Quota

50. Non-Probability Sampling Methods • Convenience Sample • The sampling procedure used to obtain those units or people most conveniently available • Why: speed and cost • External validity? • Internal validity • Is it ever justified?