1 / 21

Optimal Experimental Design in Experiments With Samples of Stimuli

Optimal Experimental Design in Experiments With Samples of Stimuli. Jacob Westfall University of Colorado Boulder David A. Kenny Charles M. Judd University of Connecticut University of Colorado Boulder.

nyla
Télécharger la présentation

Optimal Experimental Design in Experiments With Samples of Stimuli

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Optimal Experimental Design in Experiments With Samples of Stimuli Jacob Westfall University of Colorado Boulder David A. Kenny Charles M. Judd University of Connecticut University of Colorado Boulder

  2. Studies involving participants responding to stimuli (hypothetical data matrix): Subject # 1 2 3

  3. Just in domain of implicit prejudice and stereotyping: • IAT (Greenwald et al.) • Affective Priming (Fazio et al.) • Shooter task (Correll et al.) • Affect Misattribution Procedure (Payne et al.) • Go/No-Go task (Nosek et al.) • Primed Lexical Decision task (Wittenbrink et al.) • Many non-paradigmatic studies

  4. Hard questions • “How many stimuli should I use?” • “How similar or variable should the stimuli be?” • “When should I counterbalance the assignment of stimuli to conditions?” • “Is it better to have all participants respond to the same set of stimuli, or should each participant receive different stimuli?” • “Should participants make multiple responses to each stimulus, or should every response by a participant be to a unique stimulus?”

  5. Stimuli as a source of random variation • Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54-69.

  6. Power analysis in crossed designs • Power determined by several parameters: • 1 effect size (Cohen’s d) • 2 sample sizes • p = # of participants • q = # of stimuli • Set of Variance Partitioning Coefficients (VPCs) • VPCs describe what proportion of the random variation in the data comes from which sources • Different designs depend on different VPCs

  7. Definitions of VPCs • VP : Participant variance • Variance in participant intercepts • VS : Stimulus variance • Variance in stimulus intercepts • VP×C : Participant-by-Condition variance • Variance in participant slopes • VS×C : Stimulus-by-Condition variance • Variance in stimulus slopes • VP×S : Participant-by-Stimulus variance • Variance in participant-by-stimulus intercepts

  8. Four common experimental designs

  9. Stimuli-within-Condition designvs.Participants-within-Condition design • S-w-C is more powerful than P-w-C when: • Where p = # of participants, q = # of stimuli. • If VP is relatively large and/or p is small:  Choose Stimuli-within-Condition • If VS is relatively large and/or q is small:  Choose Participants-within-Condition

  10. Fully Crossed designvs.Counterbalanced design • If q is held constant, then Fully Crossed design is more powerful. • If the total number of responses per participant is held constant, then Counterbalanced design is more powerful when: • This condition will almost always be true! • If the number of unique stimuli is the limiting factor:  Choose Fully Crosseddesign • If # of responses per participant is the limiting factor:  Choose Counterbalanceddesign

  11. For power = 0.80, • need q ≈ 50

  12. For power = 0.80, • need p≈ 20

  13. ?

  14. Maximum attainable power • In crossed designs, power asymptotes at a maximum theoretically attainable value that depends on: • Effect size • Number of stimuli • Stimulus variability • Under realistic assumptions, maximum attainable power can be quite low!

  15. When q = 16, max power = .84

  16. Minimum number of stimuli to use? • A reasonable rule of thumb:  Use at least 16 stimuli per condition! (preferably more)

  17. Implications of maximum attainable power • Think hard about your experimental stimuli before you begin collecting data! • Once data collection begins, maximum attainable power is pretty much determined.

  18. Conclusion • There is a growing awareness and appreciation in experimental psychology of the importance of running adequately powered studies. • (Asendorpf et al., 2013; Bakker, Dijk, & Wicherts, 2012; Button et al., 2013; Ioannidis, 2008; Schimmack, 2012) • Discussions of how to maximize power almost always focus simply on recruiting as many participants as possible. • We hope that the present research begins the discussion of how stimuli ought to be sampled in order to maximize statistical power.

  19. The end URL for power app: JakeWestfall.org/power/ Manuscript reference: Westfall, J., Kenny, D. A., & Judd, C. M. (under review). Statistical Power and Optimal Design in Experiments in Which Samples of Participants Respond to Samples of Stimuli. Journal of Experimental Psychology: General.

More Related