Bayes Factors and inference

1. Bayes Factors and inference Greg Francis PSY 626: Bayesian Statistics for Psychological Science Fall 2018 Purdue University

2. Bayes Factor • The ratio of likelihood for the data under the null compared to the alternative • Nothing special about the null, it compares any two models • Likelihoods are averages across different possible parameter values specified by the model by a prior distribution

3. What does it mean? • Guidelines BFEvidence 1 – 3 Anecdotal 3 – 10 Substantial 10 – 30 Strong 30 – 100 Very strong >100 Decisive

4. Evidence for the null • BF01>1 implies (some) support the null hypothesis • Evidence for “invariances” • This is more or less impossible for NHST • It is a useful measure • Consider a recent study in Psychological Sciences • Liu, Wang, Wang & Jiang (2016). Conscious Access to Suppressed Threatening Information Is Modulated by Working Memory

5. Working memory face emotion • Explored whether keeping a face in working memory influenced its visibility under continuous flash suppression • To insure subjects kept face in memory, tested for identity

6. Working memory face emotion • Different types of face emotions: fearful face, neutral face • No significant differences of correct responses (same/different) for emotions: • Experiment 1: t(11)= -1.74, p=0.110 • If we compute the JZS Bayes Factor we get • > ttest.tstat(t=-1.74, n1=12, simple=TRUE) • B10 • 0.9240776 • Which is anecdotal support for the null hypothesis • You would want B10< 1/3 for substantial support for the null

7. Replications • Experiment 3 • t(11)=-1.62, p=.133 • Experiment 4 • t(13)=-1.37, p=.195 • Converting to JZS Bayes Factors suggests these are modest support for the null • Experiment 3 • ttest.tstat(t= -1.62, n1=12, simple=TRUE) • B10 • 0.8033315 • Experiment 4 • ttest.tstat(t= -1.37, n1=14, simple=TRUE) • B10 • 0.5857839

8. The null result matters • The authors wanted to demonstrate that faces with different emotions were equivalently represented in working memory • But differently affected visibility during the flash suppression part of a trial • Experiment 1: • Reaction times for seeing a face during continuous flash suppression were shorter for fearful faces than for neutral faces • Main effect of emotion: F(1, 11)=5.06, p=0.046 • Reaction times were shorter when the emotion of the face during continuous flash suppression matched the emotion of the face in working memory • Main effect of congruency: F(1, 11)=11.86, p=0.005

9. Main effects • We will talk about a Bayesian ANOVA later, but we can consider the t-test equivalent of these tests: • Effect of emotion • > ttest.tstat(t= sqrt(5.06), n1=12, simple=TRUE) • B10 • 1.769459 • Suggests anecdotal support for the alternative hypothesis • Effect of congruency • ttest.tstat(t= sqrt(11.86), n1=12, simple=TRUE) • B10 • 9.664241 • Suggests substantial support for the alternative hypothesis

10. Evidence • It is generally harder to get convincing evidence (BF>3 or BF>10) than to get p<.05 • Interaction: F(1, 11)=4.36, p=.061 • Contrasts: • RT for fearful faces shorter if congruent withworking memory: t(11)=-3.59, p=.004 • RT for neutral faces unaffected by congruencyt(11)=-0.45 • Bayesian interpretations of t-tests: • > ttest.tstat(t=-3.59, n1=12, simple=TRUE) • B10 • 11.94693 • > ttest.tstat(t=-0.45, n1=12, simple=TRUE) • B10 • 0.3136903

11. Substantial Evidence • For a two-sample t-test (n1=n2=10), a BF>3 corresponds to p<0.022 • For a two-sample t-test (n1=n2=100), a BF>3 corresponds to p<0.012 • For a two-sample t-test (n1=n2=1000), a BF>3 corresponds to p<0.004

12. Strong Evidence • For a two-sample t-test (n1=n2=10), a BF>10 corresponds to p<0.004 • For a two-sample t-test (n1=n2=100), a BF>10 corresponds to p<0.003 • For a two-sample t-test (n1=n2=1000), a BF>10 corresponds to p<0.001 • Of course, if you change your prior you change these values • (but not much) • Setting the scale parameter r=sqrt(2) (ultra wide) gives • For a two-sample t-test (n1=n2=10), a BF>10 corresponds to p<0.005 • For a two-sample t-test (n1=n2=100), a BF>10 corresponds to p<0.0017 • For a two-sample t-test (n1=n2=1000), a BF>10 corresponds to p<0.00054

13. Bayesian meta-analysis • Rouder & Morey (2011) identified how to combine replication studies to produce a JZS Bayes Factor that accumulates the information across experiments • The formula for a one-sample, one-tailed t-test for BF10 is • f( ) is the Cauchy (or half-Cauchy) distribution • g( ) is the non-central t distribution • It looks complicated, but it is easy enough to calculate

14. Bayesian meta-analysis • Consider the null results on face emotion and memorability • Experiment 1 • t(11)= -1.74, p=0.110 • Experiment 3 • t(11)=-1.62, p=.133 • Experiment 4 • t(13)=-1.37, p=.195 • Strong support for the alternative! > tvalues<-c(-1.74, -1.62, -1.37) > nvalues<-c(12, 12, 14) > meta.ttestBF(t=tvalues, n1=nvalues) Bayes factor analysis -------------- [1] Alt., r=0.707 : 4.414733 ±0% Against denominator: Null, d = 0 --- Bayes factor type: BFmetat, JZS

15. Equivalent statistics • Bayes Factors are not magic, and they use the very same information as other approaches to statistical inference • Consider a variety of statistics for different inferential methods • Standardized effect size (Cohen’s d, Hedge’s g) • Confidence interval for d or g • JZS Bayes Factor • Akaiki Information Criterion (AIC) • Bayesian Information Criterion (BIC)

16. Equivalent statistics • For a 2-sample t-test with known sample sizes n1 and n2, all of these statistics are mathematicallyequivalent to each other • Given one statistic, you can compute all the others • You should use the statistic that is appropriate for the inference you want to make

17. Equivalent statistics • Each of these statistics is a “sufficient statistic” for the population effect size • A data set provides an estimate of the population effect size • It is “sufficient” because knowing the whole data set provides no more information aboutδthan just knowing d

18. Equivalent statistics: d, t, p • Any invertible transformation of a sufficient statistic is also sufficient • For example, • Similarly, a t value corresponds to a unique p value

19. Equivalent statistics: CIs • The variance of Cohen’s d is a function of only sample size and d • This means that if you know d and the sample sizes, you can compute either limit of a confidence interval of d • If you know either limit of a confidence interval of d you can also compute d • You get no more information about the data set by reporting a confidence interval of d than by reporting a p value

20. Equivalent statistics: Likelihood • Many statistics are based on likelihood • Essentially the “probability” of the observed data, given a specific model (not quite probability because a specific value of a continuous variable has probability zero- so it is a product of the probability density function values) • For a two-sample t-test, the alternative hypothesis (full model) is that a score from group s (1 or 2) is defined as • With different means for each group s • Likelihood for the full model is then:

21. Equivalent statistics: Likelihood • For a two-sample t-test, the null hypothesis (reduced model) is that a score from group s (1 or 2) is defined as • With the same mean for each group s • These calculations always use estimates of the mean and standard deviation that maximize the likelihood value for that model

22. Equivalent statistics: Likelihood • Compare the full (alternative) model against the reduced (null) model • Log likelihood ratio • Because the reduced model is a special case of the full model, LF > LR • If Λ is sufficiently big, you can argue that the full model is better than the reduced model • Likelihood test

23. Equivalent statistics: t, Likelihood • No new information here • Let n = n1 + n2 • Then

24. Equivalent statistics: AIC • As we saw earlier, just adding complexity to a model will make its claims unreplicable • The model ends up “explaining” random noise • The model will poorly predict future random samples • A better approach is to adjust the likelihood to consider the complexity of the model • Models are penalized for complexity • Akaiki Information Criterion (AIC) • Smaller (more negative) values are better

25. Equivalent statistics: AIC • For a two-sample t-test, we can compare the full (alternative, 3 parameters) model and the reduced (null, 2 parameters) model • When ΔAIC>0, choose the full model • When ΔAIC<0, choose the null model

26. Equivalent statistics: AIC • For small sample sizes, you will do better with a “corrected” formula • So, for a two-sample t-test • When ΔAICc>0, choose the full model • When ΔAICc<0, choose the null model • The chosen model is expected to do the better job of predicting future data • This does not mean it will do a “good” job, maybe both models are bad

27. Equivalent statistics: AIC • Model selection based on AIC is appropriate when you want to predict future data, but you do not have a lot of confidence that you have an appropriate model • You expect the model to change with future data • Perhaps guided by the current model • To me, this feels like a lot of research in experimental psychology • The calculations are based on the very same information in a data set as the t-value, d-value, and p-value

28. Equivalent statistics: AIC • Inference based on AIC is actually more lenient than the traditional criterion for p-values

29. Equivalent statistics: BIC • Decisions based on AIC are not guaranteed to pick the “correct” model • An alternative complexity correction does better in this regard • Bayesian Information Criterion • For a two-sample t-test

30. Equivalent statistics: BIC • Inference based on BIC is much more stringent than the traditional criterion for p-values

31. Equivalent statistics: JZS BF • AIC and BIC use the “best” (maximum likelihood) model parameters • A fully Bayesian approach is to average likelihood across plausible parameters • Requires a prior probability density function • Compute the ratio of average likelihood for the full (alternative) and reduced (null) model • Bayes Factor • The JZS prior is for the standardized effect size • It’s Bayes Factor is simply a function of t and the sample sizes • It contains no more information about the data set than a p-value

32. Equivalent statistics: JZS BF • Inference based on the JZS Bayes Factor is much more stringent than the traditional criterion for p-values

33. Equivalent statistics: JZS BF • Model selection based on BIC and the JZS Bayes Factor is guaranteed to select the “true” model, if it is being tested • So, if you think you understand a situation well enough that you can identify plausible “true” models, then the BIC or Bayes Factor process is a good choice for identifying the true model

34. Equivalent statistics • I created a web site to do the conversions between statistics • http://psych.purdue.edu/~gfrancis/EquivalentStatistics/ • Also computes other relevant statistics (e.g., post hoc power)

35. Equivalent statistics • The various statistics are equivalent, but that does not mean you should report whatever you want • It means you should think very carefully about your analysis • Do you want to predict future data? • Do you think you can identify the “true” model? • Do you want to control the Type I error rate? • Do you want to estimate the effect size? • You also need to think carefully about whether you can satisfy the requirements of the inference • Can you avoid optional stopping in data collection? • Is your prior informative?

36. What should we do? • The first step is to identify what you want to do • Not as easy as it seems • “Produce a significant result”is not an appropriate answer • Your options are basically: • 1) Control Type I error: [identify an appropriate sample size and fix it; identify the appropriate analyses and adjust the significance criterion appropriately; do not include data from any other studies (past or future)] • 2) Estimate an effect size: [sample until you have a precise enough measurement; have to figure what “precise enough” means; explore/describe the data without drawing conclusions] • 3) Find the “true” model: [sample until the Bayes Factor provides overwhelming evidence for one model versus other models; have to identify prior distributions of “belief” in those models; have to believe that the true model is among the set being considered] • 4) Find the model that best predicts future data: [machine learning techniques such as cross-validation; information criterion; be willing to accept that your current model is probably wrong]

37. Equivalent statistics • Common statistics are equivalent with regard to the information in the data set • But no method of statistical inference is appropriate for every situation • The choice of what to do can give radically different answers to seeming similar questions • n1=n2=250, d=0.183 • p=0.04 • ΔBIC= - 2.03 (evidence for null) • ΔAICc = 2.16 (full model better predicts future data than the null model) • JZS Bayes Factor = 0.755 (weak evidence that slightly favors the null model)

38. What should we do? • Do you even need to make a decision? (choose a model, reject a null) • Oftentimes the decision of a hypothesis test is really just a description of the data • When you make a decision you need to consider the context (weigh probabilities and utilities) • For example, suppose a teacher needs to improve mean reading scores by 7 points for a class of 30 students • Approach A (compared to current method): , s=5, d=1.2 • Approach B (compared to current method): , s=50, d=0.1 B: P(Mean>7)=0.41 A: P(Mean>7)=0.14

39. Conclusions • These differences make sense because science involves many different activities at different stages of investigation • Discovery • Theorizing • Verification • Prediction • Testing • Bayes Factors fit into part (but not all) of these activities