1 / 13

Criticisms of Meta-Analysis

Criticisms of Meta-Analysis. Algera et al. Definition of rho Predictor Criterion Population Model is for a single combination in a single population. Applied to multiple predictors, criteria, unspecified population(s). JAP, a meta-analysis. Criterion Measures.

ulysses-savage
Télécharger la présentation

Criticisms of Meta-Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Criticisms of Meta-Analysis

  2. Algera et al. • Definition of rho • Predictor • Criterion • Population • Model is for a single combination in a single population. Applied to multiple predictors, criteria, unspecified population(s). • JAP, a meta-analysis

  3. Criterion Measures • Homogeneity of predictors and criteria • Supervisory ratings mostly • Multidimensionality of criteria

  4. 75 percent rule Unknown type I and type II error rates. Depends heavily on N/study Assumption that 25 percent is due to junk Q (chi-square) test Power depends on k Not worked out for corrected effect sizes Test of Situational Specificity

  5. SS vs. VG • Situational Specificity rejected if V(rho)=0. • Validity Generalizes if V(rho) >0 and CRLow > some value. • What test (predictor)? • What criterion? • What population?

  6. Meanings of Situation • Outside the individual e.g., working conditions, pay for performance • Nature of job performance, dimensionality, criterion factor structure (considered SS by SnH) • Research design, e.g., time between measurements, reliability, range restriction, etc.

  7. REVC is unsatisfactory • REVC represents unexplained variability in effect sizes • Theory is all about explanation • A good theory of, e.g., Situation, will result (ultimately) in a single estimate of rho.

  8. Sharpe • Apples & Oranges • File Drawer • GIGO, study rigor

  9. Apples & Oranges • Inclusion criteria • Homogeneity test • Not really helpful • The problem of moderators • May be sig moderator even if overall Q is n.s. • Quickly exhaust studies with multiple moderators

  10. File Drawer • Explain search for studies • Include published & unpublished studies, depending on study purpose • Report correlation between sample size and effect size • Calculate fail-safe N • May not be very meaninful, tho, assume ES=0, but ES could be negative • Use sophisticated bias detection methods, e.g., trim & fill

  11. GIGO • Are published studies really better? • “Best-evidence” synthesis • Meta-analyze only the best studies • Major disagreements about what ‘best’ means • Code for features, e.g., random assignment, blind to condition

  12. Borenstein et al. issues • One number cannot summarize a field • File drawer problem • Mixing apples and organes • Garbage in, garbage out • Important studies are ignored • Meta-analysis can disagree with randomized trials • Meta-analyses are performed poorly

  13. Other issues • Conclusions of meta-analyses disagree • Premature closure of research areas

More Related