New techniques in testing the significance of mediated effects

1. New techniques in testing the significance of mediated effects Rob Dvorak & Ryan Engdahl Department of Psychology The University of South Dakota

2. Outline Introduction What is a mediator? How do you test for mediation? Problems with traditional tests of mediated effects? New techniques in testing mediated effects. Examples

3. Introduction There are svereal reasons for wanting to examine mediation. In social science research it has become increasingly clear that there is seldom one effect in a causal chain, and often the effects of one variable are transferred to another variable. Although it is most commonly seen in correlational analyses in the lit, as you will see it is also an important part of experimental research. In the example we will be looking at today we used an experimental manipulation in which we depleted subjects state self-control, we found partial mediation via decrements in blood glucose on task persistence. Without the new techniques, we would not have been able to find these effects. Why is this important? Because as researchers, we have to stay on top of the most advanced statistical procedures. To glean the best information from your data you need to use the best available tools. This fact was evident last year when a member of the USD faculty who was applying for an NIH grant was told that they could no longer use the traditional tests of mediation and should be using the new 2007 version by Mackinnon. They were told this in February of 2007.There are svereal reasons for wanting to examine mediation. In social science research it has become increasingly clear that there is seldom one effect in a causal chain, and often the effects of one variable are transferred to another variable. Although it is most commonly seen in correlational analyses in the lit, as you will see it is also an important part of experimental research. In the example we will be looking at today we used an experimental manipulation in which we depleted subjects state self-control, we found partial mediation via decrements in blood glucose on task persistence. Without the new techniques, we would not have been able to find these effects. Why is this important? Because as researchers, we have to stay on top of the most advanced statistical procedures. To glean the best information from your data you need to use the best available tools. This fact was evident last year when a member of the USD faculty who was applying for an NIH grant was told that they could no longer use the traditional tests of mediation and should be using the new 2007 version by Mackinnon. They were told this in February of 2007.

4. What is a mediator?

5. What is a mediator? The mechanism by which one variable affects another variable 

6. What is a mediator? The mechanism by which one variable affects another variable 

7. Testing Mediation Baron & Kenny (1986) Step 1: IV DV Step 2: IV Mediator Step 3: Mediator DV Step 4: Effect of IV on DV is significantly reduced by controlling for the mediator.

8. Testing Mediated Effects Goodman (1960) test z-value = a*b/SQRT(b2*sa2 + a2*sb2 - sa2*sb2) Sobel (1982) test z-value = a*b/SQRT(b2*sa2 + a2*sb2) Aroian (1944/1947) test z-value = a*b/SQRT(b2*sa2 + a2*sb2 + sa2*sb2) Saying that the relationship between the IV & the DV is reduced after inclusion of the MV doesn�t really tell you if adding the MV changes anything. For instance, what if the reduction were only a mariginal or nonsignificant amount. Then the addition of the MV is not the most parsimonious model. To correct for this problem, B&K suggested using tests of mediated effects. One of the first tests was the Aroian test (which is what B&K suggested be used). The Goodman test cam later is the least restrictive. It assumes that the product of the variance of a & b are so small that they should be excluded from the model. The Sobel test also assumes that it is so small that it needn�t be accounted for, and simply removes it from the analysis completely. The problem is that with a small n, one could reasonably expect the product of these two to be increasingly large. The Aorian makes no assumptions about the size of the product. However, as n increases the difference in estimates of the Sobel & Aroian decrease. Saying that the relationship between the IV & the DV is reduced after inclusion of the MV doesn�t really tell you if adding the MV changes anything. For instance, what if the reduction were only a mariginal or nonsignificant amount. Then the addition of the MV is not the most parsimonious model. To correct for this problem, B&K suggested using tests of mediated effects. One of the first tests was the Aroian test (which is what B&K suggested be used). The Goodman test cam later is the least restrictive. It assumes that the product of the variance of a & b are so small that they should be excluded from the model. The Sobel test also assumes that it is so small that it needn�t be accounted for, and simply removes it from the analysis completely. The problem is that with a small n, one could reasonably expect the product of these two to be increasingly large. The Aorian makes no assumptions about the size of the product. However, as n increases the difference in estimates of the Sobel & Aroian decrease.

9. Problems with Traditional Tests The traditional tests are simply z-tests ME aren�t normally distributed +/- 1.96 may, or may not, actually be significant There are some problems with the traditional tests of mediated effects. Springer in 1979 showed that the product of two normally distributed variables is only normal in special cases. For instance, the product of two standard normal variables (mean of zero) has a kurtosis equal to 6. A normal distribution has kurtosis equal to zero. Rather than looking at Z scores, Mackinnon proposed that we examine the CI of the mediated effects based on distributions of a lrage scale monte carlo study that was conducted in his lab. Rather than Z, this gives of 95% CIs which will tell you if your ME is significant. There are some problems with the traditional tests of mediated effects. Springer in 1979 showed that the product of two normally distributed variables is only normal in special cases. For instance, the product of two standard normal variables (mean of zero) has a kurtosis equal to 6. A normal distribution has kurtosis equal to zero. Rather than looking at Z scores, Mackinnon proposed that we examine the CI of the mediated effects based on distributions of a lrage scale monte carlo study that was conducted in his lab. Rather than Z, this gives of 95% CIs which will tell you if your ME is significant.

10. Here�s Our Model Here�s the model we hypothesized. Explain Baumeister�s findings.Here�s the model we hypothesized. Explain Baumeister�s findings.

11. Here�s Our Model Here is what we actually found. All these paths were significant. So the question is: Is our mediated effect significant in this model (does it matter).Here is what we actually found. All these paths were significant. So the question is: Is our mediated effect significant in this model (does it matter).

12. For our data The traditional way is to look at measures of mediated effects via one of the three tests we talked about earlier. If we do this then what we find is that in our data, the mediated effect is not significant.The traditional way is to look at measures of mediated effects via one of the three tests we talked about earlier. If we do this then what we find is that in our data, the mediated effect is not significant.

13. Three ways to go from here You can use re-sampling techniques to bootstrap the standard error of your path coefficients. You can refer to MacKinnon�s technique to correct for the distribution of Mediated effects (PRODCLIN). You can utilize computationally intensive techniques that will boot strap the standard error of the Mediated effect based on your particular distribution. There�s 3 things you can do from here. Remember, your science is only as good as the tools you�re using.There�s 3 things you can do from here. Remember, your science is only as good as the tools you�re using.

14. Bootstrapped standard error of path Coefficients You can bootstrap the SE of your coefficients. Bootstrapping is becoming more and more popular. There has been this myth perpetrated upon us poor researchers by our statistics professors. They tell us about the assumptions of linearity, and normality and then say in special cases� The truth is, linearity and normality are really only special cases that seldom appear. Anyone who has taken a test from Dr. Schweinle knows that all data is skewed kurtotic, missing, and polynomic. So the use of the bootstrap is becoming more and more popluar. Bootstrapping is the practice of estimating properties of an estimator (such as its variance) by measuring those properties when sampling from an approximating distribution. One standard choice for an approximating distribution is the empirical distribution of the observed data. Bootstrapping comes with some assumptions of its own however. The bootstrap procedure assumes that your sample distribution is representative of your population The disadvantage of bootstrapping is that while (under some conditions) it is asymptotically consistent, it does not provide general finite sample guarantees, and has a tendency to be overly optimistic Of course there is a further problem when using it for mediation if your only applying it to the coefficients, and that is that the product of the coefficients is still going to be non-normal. It just gives you a slightly better estimate of the standard error. You can bootstrap the SE of your coefficients. Bootstrapping is becoming more and more popular. There has been this myth perpetrated upon us poor researchers by our statistics professors. They tell us about the assumptions of linearity, and normality and then say in special cases� The truth is, linearity and normality are really only special cases that seldom appear. Anyone who has taken a test from Dr. Schweinle knows that all data is skewed kurtotic, missing, and polynomic. So the use of the bootstrap is becoming more and more popluar. Bootstrapping is the practice of estimating properties of an estimator (such as its variance) by measuring those properties when sampling from an approximating distribution. One standard choice for an approximating distribution is the empirical distribution of the observed data. Bootstrapping comes with some assumptions of its own however. The bootstrap procedure assumes that your sample distribution is representative of your population The disadvantage of bootstrapping is that while (under some conditions) it is asymptotically consistent, it does not provide general finite sample guarantees, and has a tendency to be overly optimistic Of course there is a further problem when using it for mediation if your only applying it to the coefficients, and that is that the product of the coefficients is still going to be non-normal. It just gives you a slightly better estimate of the standard error.

15. Using PRODCLIN Distribution of the PROduct Confidence Limits for INdirect effects (PRODCLIN) http://www.public.asu.edu/~davidpm/ripl/Prodclin/ PRODCLIN is a set of SAS, SPSS, or R code developed specifically to utilize the distribution of Mediated Effects obtained from a large scale Monte Carlo study. The Mediated effects distributions are bias corrected estimates based on mean, variance, skew, and kurtosis associated with the typical distribution at your given N and with your z values. Has two components, the fortran executable and the stats package specific code. Lets see how it works. PRODCLIN is a set of SAS, SPSS, or R code developed specifically to utilize the distribution of Mediated Effects obtained from a large scale Monte Carlo study. The Mediated effects distributions are bias corrected estimates based on mean, variance, skew, and kurtosis associated with the typical distribution at your given N and with your z values. Has two components, the fortran executable and the stats package specific code. Lets see how it works.

16. PRODCLIN Results OUTPUT: a=-2.455927 sea=1.168241 b=12.322910 seb= 3.005995 ab= -30.264167 sobelse= 16.178686 rho= 0.000000 Type1 error= .050000 Normlow= -61.973810 Normup= 1.445475 Prodclin lower critical value -2.821298 Prodclin upper critical value 1.534485 Prodlow= -75.909062 Produp= -5.438211 Here�s the output. Explain.Here�s the output. Explain.

17. Bootstrapping the Standard Error of Mediated Effects This is the most computationally intensive method for generating an accurate mediated effect. Utilizes the bootstrap for the SE of the mediated effect. The method was initially developed by Lockwood & Mackinnon (1998) and expanded upon by Preacher & Hayes (2004). It isn�t subject to the same assumptions as using the Mackinnon tables from the Monte Carlo study (i.e. your sample may not have the same distribution that Mackinnon�s Monte Carlo Sample had).The method was initially developed by Lockwood & Mackinnon (1998) and expanded upon by Preacher & Hayes (2004). It isn�t subject to the same assumptions as using the Mackinnon tables from the Monte Carlo study (i.e. your sample may not have the same distribution that Mackinnon�s Monte Carlo Sample had).

18. Using the Bootstrap method AMOS, EQS, LISREL, and Mplus are all capable of conducting bootstrap resampling A recent program was written for STATA that can do it regardless of variable type for the IV MV or DV. So, lets take a look at our current dataset to see how it works in STATA. sgmediationCENDV time ,mv(bg2) iv(group) cv(sex video bg1 gbsc pbsc groupXgbsc) bootstrap�reps(100)So, lets take a look at our current dataset to see how it works in STATA. sgmediationCENDV time ,mv(bg2) iv(group) cv(sex video bg1 gbsc pbsc groupXgbsc) bootstrap�reps(100)

19. STATA Bootstrap ME Search sgmediation

20. Conclusions Mediation is becoming increasingly more important There are several new ways to test for mediated effects that help to reduce the error associated with product distribution problems The use of these �new� will likely become standard procedures in the near future; so you might as well start using them now.

21. References Aroian, L. A. (1944/1947). The probability function of the product of two normally distributed variables. Annals of Mathematical Statistics, 18, 265-271. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. Goodman, L. A. (1960). On the exact variance of products. Journal of the American Statistical Association, 55, 708-713. Hoyle, R. H., & Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. Hoyle (Ed.) Statistical Strategies for Small Sample Research. Thousand Oaks, CA: Sage Publications. MacKinnon, D. P., Fritz, M. S., Williams, J., & Lockwood, C. M. (2007). A comparison of methods to test mediation and other intervening variable effects. Behavior Research Methods, 39, 384-389. MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83-104. MacKinnon, D. P., Warsi, G., & Dwyer, J. H. (1995). A simulation study of mediated effect measures. Multivariate Behavioral Research, 30(1), 41-62. Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36(4), 717-731. Sobel, M. E. (1982). Asymptotic intervals for indirect effects in structural equations models. In S. Leinhart (Ed.), Sociological methodology 1982 (pp.290-312). San Francisco: Jossey-Bass.