Research Design Experiments Observations & Surveys Prof. Craig Jackson Head of Psychology Division School of Social

1. Research Design Experiments Observations & Surveys Prof. Craig Jackson Head of Psychology Division School of Social Sciences BCU health.bcu.ac.uk/craigjackson craig.jackson@bcu.ac.uk

2. ! Objectives Experimental studies within-subjects studies between-subjects studies Observational studies case-controls cohorts RCTs Bias Placebo Control Groups

3. Experimental Longitudinal Prospective Randomised Controlled Trial Introduction Types of research Experimental vs. Observational Longitudinal vs. Cross-sectional Prospective vs. Retrospective

4. Qualititative VS Quantitative Research False opposition Observational methods equally valid Complementary roles Qualitative equally as hard to do Quantitative Qualitative

5. Experimental Studies • Investigator makes intervention • A “manipulation” • Then studies the effects of that intervention • Features: • Comparison e.g. before vs. after control vs. treatment • Always longitudinal • Always prospective • Experimental • Clinical • Trials • RCTs

6. Rationale of Experimental Studies Evaluate effectiveness of intervention / therapy Use similar samples – comparable groups Samples reflect population Differences in outcomes due to interventions (not differences between groups) Independent Variable (IV) alters Dependent Variable (DV) Best evidence of cause and effect Sometimes inconclusive

7. Types of Experimental Studies Between Subjects Studies Each group receives different treatment Groups compared Within Subjects Studies Each individual is measured before & after intervention Advantage that each participant is own control Between subject variability removed

8. Within Subjects Studies Cross-over-studies Each patient receives treatment in sequence “Washout” period between treatments Order of treatments randomised Matched-pairs study Parallel study Patient in arm 1 matched with patient in arm 2 Matched based on prognostic factors Data is linked Paired individuals Gp A Gp A Treatment 2 Treatment 1 Gp B Treatment 1 Treatment 2 Gp B

9. Avoiding Bias Validity of study depends on avoiding bias Bias = “Systematic distortion of results due to unforeseen factors” gp1 = pill gp2 = no pill How will the “no pill”group progress? Any effects of them knowing they have no treatment? Handling differences may influence + complicate trial results Known as confounding factors To minimize bias… control group randomisation blinding

10. Placebo effect – it really does work! • Most effectivemedication known • In approx. 30% of pop. • Subjected to more clinical trialsthan any other medicament • Nearly always does better thananticipated • The range of susceptible conditions seems limitless • Does not always occur • Present in subjective and objective outcomes • Negative outcomes can occur (Nocebo effect) • Placebo • Big pills better than smaller pills • Red pills better than blue • 4 pills better than 2 • 30% of pop. Patient’s “knowledge” of their treatment causes biase.g. Benedetti & the Turin study

11. Control Groups • Allow comparison in Between Group studies • Evaluations without comparison? • Patient knowledge of their treatment causes bias • e.g. Benedetti & the Turin study • Types of Control Groups • “no treatment” group likely to be confounded by condition • “placebo” group ethically dodgy? • “low dose” groupavoids ethical issues • “standard treatment” group avoids ethical issues • “gold standard” group avoids ethical issues • “historical controls” unreliable due to many confounders

12. Population (60 million) Sample (1000) Gp A (500) Gp B (500) Drug X 53 yrs 80% male 20% fem Drug Y 27 years 50% male 50% fem Control Groups: Random Allocation Doesn’t guarantee groups will be homogonous Ensures allocation independent of patient features Avoids (sub)conscious allocation bias e.g. severely sick people into treatment groups Guarantees allocationto be bias-free Non-homogenous groups may still occur due to chance – random errors

13. Comparison Groups: Random Sampling Ensures generalizability of findings to larger pop. e.g. in-custody sample limitations Treatment effects better detected if there is little between-group variability Exclusion Criteria & Inclusion Criteria keep groups comparable Paradox: greater uniformity of sample = less generalizable to gen. pop

14. Blinding: Importance of doing it Investigator or Subject know treatment = Bias Observations and Judgements become less reliable Patient responses change: Positive outcomes in active arm Negative outcomes in passive arm e.g. known cancer diagnoses and deterioration Use max. degree of blindness possible e.g. make subject and investigator both blind if possible e.g. A.A.Mason & Congenital Ichthyosis and Hypnosis 1951

15. Blinding: Methods of doing it Double-blind patient & investigator blind Treatment type Patient interaction Data manager

16. Un-blinding a problematic study Breaking code – anticipated in planning Criteria for breaking code – established and agreed Emergency access to randomisation code Treatment stopped and patient withdrawn Formal monitoring process – review and make recommendations

17. Blinding: Methods Double-blind patient & investigator blind Single-blind patient blind Triple-blind patient & investigator & data monitor blind Double-dummy 2 treatments patients get 2 pills (1 active, 1 dummy) Open trials patient & investigator aware of treatment Randomisation in a double-blind trial Envelope technique common Un-blinding – ethical necessity

18. Subject Variables that confound research STABLE FACTORSSITUATIONAL FACTORS Age Alcohol (recent use Education Caffeine (recent use) Sex Nicotine (recent use) Socioeconomics Medication (recent use) Language Paints, glues, pesticides Handedness Near visual acuity Computer experience Restricted movement (injury) Caffeine (habitual use) Cold / flu Alcohol (habitual use) Stress Nicotine (habitual use) Arousal / Fatigue Medication (habitual use) Sleep Paints, glues, pesticides Screen luminance Diabetes Time of day Epilepsy Time of year Other CNS / PNS disease Head injury (out >1 hr) Alcohol / drug addiction Physical activity

19. Randomized Controlled Trials in Practice 90% consultations take place in GP surgery RCT is really 50 years old Potential problems 2 Key areas: Recruitment Bias Randomisation Bias Over-focus on failings of RCTs

20. RCTs in Practice RCTs justified insituations of genuine clinical uncertainty Samples large enough to establish any worthwhile benefit (effectiveness or cost, or both) Need for larger numbers of participants More than are availableto single practices Requires “club together” approach Practitioners: no contractual obligation (i) unwilling to take part if no immediate benefit for clients (ii) while possibly disruptingthe delivery of service/care

21. RCTs in Practice Conflict of interest between: Role and Wish to benefit future offenders Academic merit Long term nature of practitionerand client relationship may engender loyalties unfairly coerce clientsto give consent Patients' fearsabout: confidentiality risks of the intervention apparent disadvantage of being allocated to a control group may further inhibit recruitment Fail to recruit consecutiveclients may introduce potential for selection bias

22. RCTs in Practice Provides rigorous,sound basis for evaluating treatments May disrupt care Too much disruption = no reflection of real practice Methodologicalproblems reduce scientific reliability of the results (Recruitment & Randomisation) Practice not a laboratory Peopleare not experimental animals Case-control studies, retrospective, prospective cohort studies, and descriptive studies areall acceptable methods. Observation is OK Should accept alternative methodswhen RCT difficult or flawed

23. RCT Deficiencies Trials too small Trials too short Poor quality Poorly presented Address wrong question Methodological inadequacies Inadequate measures of quality of life (changing) Cost-data poorly presented Ethical neglect Participants given limited understanding Poor trial management Politics Marketeering Why still the dominant model?

24. Observational Studies Investigator observes existing situation Describes Analyses Interprets No influence on events Longitudinal observation studies case-control studies: retrospective cohort-studies: prospective Cross-sectional observation studies surveys examining subjects at one point in time based on random sample of interest population

25. Observational Studies: • Look for associations • Cause -> Effect • Exposure – Illness • Epidemiological • Incidence • Cause • Prevention • No control • Cannot use classical experimentation • No randomisation • Bias

26. Case-Control Study Identify group with condition / offence (cases) Identify group without condition / offence (controls) Both groups compared for exposure to (hypothesized) risk factors Greater exposure to risk factor in cases than controls = “causal relation” Lead time bias Recruitment of cases at similar points in time Newly labelled cases

27. Selection of Controls Cases have Theft offences Controls could be other young people patients or “normals” Matched Cases & Controls for age & gender Option of 2 Controls per Case Smoking years of cases and controls (matched for age and sex) Cases Controls n=456 n=456 F P Smoking yrs 13.75 6.12 7.5 0.04 (± 1.5) (± 2.1)

28. Case-Control Study: Other Biases Recall Bias Cases > associations with exposures / risk factors Unreliable Memories Retrospective nature Over-reliance on recall Unreliable Records Poor hospital records Repetitive, incomplete, inaccurate, irretrievable, interpretation Interview Bias Different interviewers

29. Cohort Study • ID and examination of a group (cohort) • Followed over time (20 years common!) • Looking for condition development / other end-point • Aetiology of condition (based on data collected) • Data more reliable than case-control studies • Requires large N • Requires long follow up • Inefficient • Expensive (espec. rare outcomes)

30. Cohort Study: Methods Volunteers in 2 groups e.g. exposed vs non-exposed All complete attitude survey every 12 months End point at 5 years: groups compared for Health Status Comparison of general health between users and non-users of mobile phones ill healthy mobile phone user 292 108 400 non-phone user 89 313 402 381 421 802

31. Cohort Study: Other Biases Lost to follow up Bias if reason related to exposure Validity affected Group sizes change Membership changes e.g ex-smokers Differential mortality Change in circumstance e.g. job change Exposures need calculation or re-calculation Surveillance bias Investigator aware of group membership Investigating exposed members more

32. Cross Sectional Study Subjects contacted & surveyed just once Questionnaire (post, email, phone) Random sample of defined pop. Limited causality Not temporal relationships Little insight into aetiology Source of descriptive data Prevalence rates Volunteer bias Non responses Self-selection Unrepresentative sample

33. Further Reading Altman, D.G. “Designing Research”. In: Altman, D.G., (ed.) Practical Statistics For Medical Research. London, Chapman and Hall, 1991; 74-106. Bland, M. “The design of experiments”. In: Bland, M., (ed.) An introduction to medical statistics. Oxford, Oxford Medical Publications, 1995; 5-25. Daly, L.E., Bourke, G.J. “Epidemiological and clinical research methods”. In: Daly L.E., Bourke, G.J., (eds.) Interpretation and uses of medical statistics. Oxford, Blackwell Science Ltd, 2000; 143-201. Jackson, C.A. “Study Design” & “Sample Size and Power”. In: Gao Smith, F. and Smith, J. (eds.) Key Topics in Clinical Research. Oxford, BIOS scientific Publications, 2002.

34. Further Reading Jackson, C.A. “Planning Health & Safety Research Projects in the Workplace”. Croner Health and Safety at Work Special Report 2002; 62: 1-16. Kumar, R. Research Methodology: a step by step guide for beginners. Sage, London 1999. Abbott, P. and Sapsford. Research methods for nurses and the caring professions. Open University Press, Buckingham 1988. Bowling, A. Measuring Health. Open University Press, Milton Keynes 1994 Polit, D. & Hungler, B. Nursing research: Principles and methods (7th ed.). Philadelphia: Lippincott, Williams & Wilkins 2003.