Subjects and Measurements

1. Subjects and Measurements Emily von Scheven, MD, MAS Pediatric Rheumatology UCSF

2. Today’s Objectives • Selecting the Subjects - How to use a systematic approach - Understand the ramifications for subject selection • Choosing the Measurements - Understand the implication of the measuring method

3. Subjects and variables are a CRITICAL to answering your research question • WHO you want to study? • WHAT you want to measure?

4. Through the lens of a real study

6. Neuro-psychiatric lupus in children Steinlin MI, et al. Pediatric Neurology 13: 191, 1995

7. The Research Question(s) • Is SLE associated with neurocognitive impairment in children? • Is neurocognitive impairment in children with SLE associated with disease characteristics, psychosocial well- being and intelligence?

8. Schematic

9. Selecting Your Subjects… WHO DO YOU WANT? HOW WILL YOU FIND & SELECT THEM?

10. Optimizing Subject Selection: Balancing Act Feasibility Accessibility Cost Time/Efficiency Generalizability Accuracy Diversity Adequate Size In the end…will I believe the findings and will I care?

11. Subject Selection: the details • Define Inclusion Criteria - Demographic features (age, gender, race) - Clinical criteria - Geographic/administrative characteristics - Sampling time frame • Define Exclusion Criteria (parsimonious) • Generalizability • Feasibility • Safety

12. Subject Sampling Convenience Samples • True convenience (25 clinic patients I know well) • Consecutive (next 100 patients undergoing liposuction) Probability Samples • Simple random (using random number table) • Stratified or weighted random (by gender) • Cluster (by clinic or neighborhood)

13. Subject Recruitment • Goal is to have a high response rate - Generalizable sample - Adequate size • For database only studies- easy • For prospective studies where you recruit- - Expect that it will be harder than you think! - Use reasonable inclusion/exclusion criteria - Acceptable subject burden/potential benefits - Subject payment

14. Subject Selection:Childhood SLE study Inclusion criteria: • age 10–21years • English speaking or fluently bilingual • Attends school in English for at least 2 years • Able to complete traditional neuropsychological testing in English Exclusion criteria: • Comorbid condition affecting cognitive functioning (e.g., cerebral palsy, Down syndrome)

15. Sampling: Childhood SLE study • Consecutive patients • Attending the Childhood Lupus Clinic • Five eligible patients declined participation

16. ControlsChildhood SLE study • Inclusion criteria - age, socioeconomic, and ethnicity-matched • Exclusion criteria - SLE or other autoimmune disease • Sampling - Mostly friends of the SLE subjects - Some healthy siblings - Some healthy neighborhood kids

17. Childhood SLE study Children with SLE Neurocog Abn

18. Childhood SLE study Children with SLE at Columbia Control sibs, friends and bystanders Children with SLE Minus: Those that refused

19. The Measurements:Implications for the choice of variables and measurement method

20. “The most elegant design of a clinical study will not overcome the damage caused by unreliable or imprecise measurement.” Fleiss, JL. The design and analysis of clinical experiments. pp. 1-5. 1986. John Wiley and Sons, New York. J.L. Fleiss (1986)

21. “Accuracy must be balanced against practical considerations, and that method chosen which will provide the maximal accuracy within the bounds of the investigator’s resources and other practical limitations.” Abramson, JH. Survey methods in community medicine (3rd Ed.), p. 121. 1984. Churchill Livingstone, Edinburgh. J.H. Abramson (1984)

22. Outcome measures:Childhood SLE study • “Neurocognitive impairment” • Ok for the research question • Not an analyzable unit • Not ok for statement of hypothesis • Not ok for a sample size calculation • Not ok to describe the analytic plan • Correct concept / too complex

23. Neurocognitive domains 1) Memory 2) Language and verbal reasoning 3) Visual–spatial reasoning 4) Executive functioning 5) Psychomotor speed 6) Fine motor dexterity 7) Academics

24. Neuropsychiatric Tests (including 23 sub-tests) Wechsler Abbreviated Scale of Intelligence for problem solving and reasoning skills Wide Range Assessment of Memory and Learning (4 subtests) Delis-Kaplan Executive Function System (1 subtest- Letter fluency) Comprehensive Trail-Making test (2 subtests) Stroop Color and Word Test (4 subtests) Wechsler Adult Intellectual Scale III Wechsler child Intellectual Scale (4 subtests) Purdue Pegboard (3 subtests) Wide Range Achievement Test

25. What do you end up with? • MANY discrete measurements/scores • - Raw score • - Z-score

26. More Variables • SLE vs control • Demographics: age, gender ethnicity • Socioeconomics • SLE manifestations • Medications • Neurocognitive syndromes • Grades in school • Parental education

27. Confounding Variables* Effect Modifiers* Organizing the Measurements Predictor* Outcome (interaction) * Often generally categorized as “exposures”

28. Confounding Variables Effect Modifiers Example: Coronary artery disease AGE Predictor Outcome MI Cholesterol (interaction) GENDER

29. Confounding Variables Effect Modifiers CNS Lupus Stroke SES Predictor Outcome SLE Neurocognitive impairment (interaction) ??

30. Variable types Continuous • Age • SLE disease activity • IQ Categorical • Dichotomous (Y/N) - Sex - Fracture - Death • Nominal (no order) - Blood type (A,B,O) - Ethnicity • Ordinal (ranked) - stage Relevant to statistical analysis Relevant to power

31. Obtaining the measurements: Data Sources Goal: If you can not measure the “gold standard”, choose the source that gives data closest to the “gold standard” while being feasible to collect • Survey/questionnaire • Interviews • Diaries • Direct observation • Environmental measurements • Databases/registries • Medical records • Physiologic measures • Biomarkers (e.g., DNA, sera), pathology, imaging

32. How to quantify the measurement • Drug Exposure - Dose (cumulative mg) - Dose (# pills/day) - Timing (date of first exposure, duration) - Distribution (early or late)

33. Can you trust your measurement? Precision: • You get the same result when measured repeatedly • within the same subject, between subjects, and over time Accuracy: • It represents what it’s really supposed to be • Sensitivity & specificity

34. Improving Precision & Accuracy Reducing Bias • Standardize measurement method • Pretest, pretest, pretest • Refine/automate instrument • Train & evaluate staff (& do it yourself!) • Timely editing, coding & correcting of forms • Multiple measurements

35. Improving Precision & Accuracy Reducing Bias, cont’d • Validate against “gold standard” • Less obtrusive measures • Blinding, eg exposure status • Institute quality control measures during data collection, processing, and analysis

36. Results: Childhood SLE studyScore compared to normal reference population (SD) • SLE patients scored low in - Executive function - Psychomotor speed - Fine motor speed • Consistent with prior data

37. Results:Childhood SLE studyAbsolute score compared to controls No difference between SLE and Control

38. Combining scores and converting to a dichotomous variable • No significant difference between SLE and controls • No association with clinical manifestations

39. Do you believe the results?Was there a Problem with Subject selection? • Systematic error/bias towards healthier SLE pts • Are the patients presenting to this clinic • (actual subjects) healthier than the • target population? • Maybe those who agreed to participate are • healthier (generalizability) • Maybe the controls were not really “normal” • Is it better to use friends or a random sample?

40. Do you believe the results?Was there a Problem with the measurements? • Test battery did not capture SLE patients neurocognitive deficits (accuracy) • Error in administration of tests • Insufficient power (41 SLE, 21 controls)?

41. Good luck with your projects!