MT2004

MT2004 Olivier GIMENEZ Telephone: 01334 461827 E-mail: olivier@mcs.st-and.ac.uk Website: http://www.creem.st-and.ac.uk/olivier/OGimenez.html

13. Analysis of variance • So far, we’ve investigated the relationship between a response variable and one or several continuous explanatory variables • The objective here is to study the relationship between a response variable Y and one or two discrete explanatory variables

13. Analysis of variance 13.1 One-way ANOVA • Example: a standard measurement of the flammability of fabric is given by the length of the burnt portion of a piece of the fabric which has been held over a flame for a given time. An investigation to see whether or not there was a difference between the measurement obtained by 5 laboratories produced the following data.

13. Analysis of variance 13.1 One-way ANOVA laboratory 1 2 3 4 5 2.9 2.7 3.3 3.3 4.1 3.1 3.4 3.3 3.2 4.1 3.1 3.6 3.5 3.4 3.7 3.7 3.2 3.5 2.7 4.2 3.1 4.0 2.8 2.7 3.1 4.2 4.1 2.8 3.3 3.5 3.7 3.8 3.2 2.9 2.8 3.9 3.8 2.8 3.2 3.5 3.1 4.3 3.8 2.9 3.7 3.0 3.4 3.5 2.6 3.5 2.9 3.3 3.8 2.8 3.9 Measurements of length obtained by 5 laboratories

13.1 One-way ANOVA • The problem here is to compare several populations • The technique we will use is the one-way analysis of variance • This is a special case of the ANOVA introduced in the Regression Section • Consider k distributions (or populations) with means 1,…,k, and suppose we wish to test: • H0: 1=…=k • against • H0: 1,…,k are not all equal • WARNING: the alternative hypothesis does not imply that all the i are different, but at least one pair. E.g. with k = 3, 1=23 would be OK.

13.1 One-way ANOVA • In the example, we wish to test the null hypothesis that the means of lengths obtained by the k = 5 laboratories are the same. • Suppose that we have random sample of sizes n1,…,nk from the k distributions. Note that the random samples do not need to have same sample size. • yij denotes the jth observation on the ith distribution, i = 1,…, k and j = 1,…, ni

13. Analysis of variance 13.1 One-way ANOVA laboratory 1 2 3 4 5 2.9 2.7 3.3 3.3 4.1 3.1 3.4 3.3 3.2 4.1 3.1 3.6 3.5 3.4 3.7 3.7 3.2 3.5 2.7 4.2 3.1 4.0 2.8 2.7 3.1 4.2 4.1 2.8 3.3 3.5 3.7 3.8 3.2 2.9 2.8 3.9 3.8 2.8 3.2 3.5 3.1 4.3 3.8 2.9 3.7 3.0 3.4 3.5 2.6 3.5 2.9 3.3 3.8 2.8 3.9 y23 Measurements of length obtained by 5 laboratories y57

13.1 One-way ANOVA • In the example, we wish to test the null hypothesis that the means of lengths obtained by the k = 5 laboratories are the same. • Suppose that we have random sample of sizes n1,…,nk from the k distributions. Note that the random samples do not need to have same sample size. • yij denotes the jth observation on the ith distribution, i = 1,…, k and j = 1,…, ni • We will assume that yij is an observation from a random variable Yij where: • Yij N(i,2), i = 1,…, k and j = 1,…, ni, Yij independent • We thus have that E(Yij) = i

13.1 One-way ANOVA • Actually, this model is a particular case of a multiple regression • Define indicator variables x1,…, xk by: • Then the equation E(Yij) = i can be rewritten as: • E(Yij) = 1x1 + … + kxk • This equation defines a multiple regression without intercept  • Now, to test the null hypothesis, we can apply the results of the end of the Regression Section (we place equality restrictions on the full model)

13.1 One-way ANOVA • The full model has k parameters (1,…,k) thus p1 = k. • The submodel under H0 is E(Yij) = , thus p0 = 1. • We have n = n1 + … + nk observations. • So an appropriate statistic to test the null hypothesis is: • If H0 is false (i.e. 1,…,k are not all equal), then this statistic will tend to take values too large to be consistent with the quantile of a F distribution with k-1 and n-k degrees of freedom.

13.1 One-way ANOVA • We provide other expressions for rss0 and rss1, much easier to manipulate • Let denote the overall sample mean • Let denote the sample mean of the ith random sample (pop.)

13.1 One-way ANOVA • We provide other expressions for rss_0 and rss_1, much easier to manipulate • Let denote the overall sample mean • Let denote the sample mean of the ith random sample (pop.)

13.1 One-way ANOVA • We provide other expressions for rss_0 and rss_1, much easier to manipulate • Let denote the overall sample mean • Let denote the sample mean of the ith random sample (pop.) • It can be shown that the total variability is the sum of the between and within variability:

13.1 One-way ANOVA • It can also be shown that the maximum likelihood are given: • For the full model by: • For the submodel by: • And that:

13.1 One-way ANOVA • If we define: • Then • Becomes:

13.1 One-way ANOVA • Most often, the sums of squares, mean squares, F values, p-values are displayed in an ANOVA table • With • Note that the within mean square MSW is an unbiased estimator of the variance 2, called the residual s.e.

13. Analysis of variance 13.1.1 One-way ANOVA in R • Example: a standard measurement of the flammability of fabric is given by the length of the burnt portion of a piece of the fabric which has been held over a flame for a given time. An investigation to see whether or not there was a difference between the measurement obtained by 5 laboratories produced the following data.

13. Analysis of variance 13.1.1 One-way ANOVA in R laboratory 1 2 3 4 5 2.9 2.7 3.3 3.3 4.1 3.1 3.4 3.3 3.2 4.1 3.1 3.6 3.5 3.4 3.7 3.7 3.2 3.5 2.7 4.2 3.1 4.0 2.8 2.7 3.1 4.2 4.1 2.8 3.3 3.5 3.7 3.8 3.2 2.9 2.8 3.9 3.8 2.8 3.2 3.5 3.1 4.3 3.8 2.9 3.7 3.0 3.4 3.5 2.6 3.5 2.9 3.3 3.8 2.8 3.9 Measurements of length obtained by 5 laboratories

13. Analysis of variance 13.1.1 One-way ANOVA in R • We wish to test the null hypothesis: • H0: 1 = … = 5 • Against the alternative hypothesis • H1: at least one pair of i’s are not equal • Where i is the mean length of burnt fabric in measurements from laboratory i (i = 1,…, 5)

13.1.1 One-way ANOVA in R > lengthlab1<-c(2.9,3.1,3.1,3.7,3.1,4.2,3.7,3.9,3.1,3.0,2.9) > lengthlab2<-c(2.7,3.4,3.6,3.2,4.0,4.1,3.8,3.8,4.3,3.4,3.3) > lengthlab3<-c(3.3,3.3,3.5,3.5,2.8,2.8,3.2,2.8,3.8,3.5,3.8) > lengthlab4<-c(3.3,3.2,3.4,2.7,2.7,3.3,2.9,3.2,2.9,2.6,2.8) > lengthlab5<-c(4.1,4.1,3.7,4.2,3.1,3.5,2.8,3.5,3.7,3.5,3.9) > lab1 <- rep(1,11) > lab2 <- rep(2,11) > lab3 <- rep(3,11) > lab4 <- rep(4,11) > lab5 <- rep(5,11) > fabric<-data.frame(lab=c(lab1,lab2,lab3,lab4,lab5),length=c(lengthlab1,lengthlab2,lengthlab3,lengthlab4,lengthlab5)) > plot(fabric$lab,fabric$length)

13.1.1 One-way ANOVA in R

13.1.1 One-way ANOVA in R H0: 1=2=3=4=5 =  ?

13.1.1 One-way ANOVA in R > reglab <- lm(length~as.factor(lab), data=fabric) > reglab Call: lm(formula = length ~ as.factor(lab), data = fabric) Coefficients: (Intercept) as.factor(lab)2 as.factor(lab)3 as.factor(lab)4 3.33636 0.26364 -0.03636 -0.33636 as.factor(lab)5 0.30909 The lm command produces in that case the parameters estimates of model E(Yij) = 1x1 + … + 5x5

13.1.1 One-way ANOVA in R • > anova(reglab) • Analysis of Variance Table • Response: length • Df Sum Sq Mean Sq F value Pr(>F) • as.factor(lab) 4 2.9865 0.7466 4.5346 0.003337 ** • Residuals 50 8.2327 0.1647 • --- • Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 • The R command anova applied to the regression object reglab produces the ANOVA table • The pvalue is small, we reject H0 that 1 = … = 5

13.1.1 One-way ANOVA in R Checking the assumptions • It is crucial to test the assumptions of the ANOVA model, in particular: • The observations in each group come from a normal distribution • The variances are equal (= 2)

13.1.1 One-way ANOVA in R Checking the assumptions • It is crucial to test the assumptions of the ANOVA model, in particular: • Normality: use QQplot on residuals • Homogeneity of variance: inspect the variances

13.1.1 One-way ANOVA in R Checking the assumptions 1. The observations in each group come from a normal distribution. We check normality of the residuals: > resfab1<-lab1-mean(lab1) > resfab2<-lab2-mean(lab2) > resfab3<-lab3-mean(lab3) > resfab4<-lab4-mean(lab4) > resfab5<-lab5-mean(lab5) > resfab<-c(resfab1,resfab2,resfab3,resfab4,resfab5) > qqnorm(resfab) > qqline(resfab)

13.1.1 One-way ANOVA in R Normality is OK…

13.1.1 One-way ANOVA in R Checking the assumptions 2. The variances are equal (2). We inspect the variances: > var(lab1) [1] 0.2045455 > var(lab2) [1] 0.212 > var(lab3) [1] 0.138 > var(lab4) [1] 0.082 > var(lab5) [1] 0.1867273 Variances are roughly equal

13. Analysis of variance 13.1.2 Least Significant Differences • When performing an ANOVA, we wish to test the null hypothesis: • H0: 1 = … = 5 • Against the alternative hypothesis • H1: at least one pair of i’s are not equal • So if the null hypothesis is rejected, the question is which differences between groups are most important • In other words, we wish to test H0: i = j,  i  j

13. Analysis of variance 13.1.2 Least Significant Differences • An appropriate test to compare the groups in pairs is the 2-sample t-test • Under H0: i = j, we have that • But 2 is unknown • We will replace 2 by s2 = rss1 / (n-k) = MSW (given in the ANOVA table)

13.1.2 Least Significant Differences • On one hand, we have that s2 is independent of • On the other hand: • So, an appropriate test statistic is: • To be compared with the quantile t0.025;n-k for a 2-sided test

13. Analysis of variance 13.1.2 Least Significant Differences • WARNING: This is not exactly the same formula as for the 2-sample t-test since: • s2 is calculated using all the data, and not just group i and j • the degree of freedom is n - k rather than ni + nj - 2

13. Analysis of variance 13.1.2 Least Significant Differences • If the samples are of equal size, i.e. n1 = … = nk • It’s easier to calculate the smallest difference in sample means leading to rejection of the null hypothesis that the 2 groups have equal means • This is called the Least Significant Differences (LSD) • If k groups, each with m observations (n = mk), the LSD for significance level  is: • Once the LSD is calculated, then look for the pairs of groups with sample means differing by more than the LSD

13.1.2 Least Significant Differences Example (Fabric data): The Least Significant Differences for significance level  is: > LSD <- qt(0.975,55-5)*sqrt(2*0.1647/11) > LSD [1] 0.3475762

13.1.2 Least Significant Differences > mean(lab5) [1] 3.645455 > mean(lab2) [1] 3.6 > mean(lab1) [1] 3.336364 > mean(lab3) [1] 3.3 > mean(lab4) [1] 3 We calculate the sample mean for each group

13.1.2 Least Significant Differences > mean(lab5) [1] 3.645455 > mean(lab2) [1] 3.6 > mean(lab1) [1] 3.336364 > mean(lab3) [1] 3.3 > mean(lab4) [1] 3 And then look for the pairs of groups with sample means differing by more than the LSD = 0.3475762

13.1.2 Least Significant Differences > mean(lab5) [1] 3.645455 > mean(lab2) [1] 3.6 > mean(lab1) [1] 3.336364 > mean(lab3) [1] 3.3 > mean(lab4) [1] 3 Suggests that 4 < 2

13.1.2 Least Significant Differences > mean(lab5) [1] 3.645455 > mean(lab2) [1] 3.6 > mean(lab1) [1] 3.336364 > mean(lab3) [1] 3.3 > mean(lab4) [1] 3 Suggests that 4 < 5

13.1.2 Least Significant Differences > mean(lab5) [1] 3.645455 > mean(lab2) [1] 3.6 > mean(lab1) [1] 3.336364 > mean(lab3) [1] 3.3 > mean(lab4) [1] 3 Suggests that 4 < 2 and 4 < 5, but does not suggest any other differences between the i

13.2 Two-way ANOVA • So far, we've considered only one explanatory discrete variable (lab in the fabric data example) • Let's assume now that each observation belongs to 2 groups • This is a 2-way ANOVA • Example: consider a reading comprehension test given to pupils of age 9, 10 and 11 from 4 schools (A, B, C and D), giving the scores:

13.2 Two-way ANOVA • Example: consider a reading comprehension test given to pupils of age 9, 10 and 11 from 4 schools (A, B, C and D), giving the scores: • So observation/score yi belongs to school j (j = 1,..., J) and age k (k = 1,..., K) • yi is an observation of an independent r.v. Yi N(i,2)

13.2 Two-way ANOVA • There are four models of potential interest. • Model 3: the expected comprehension score E(Yi) = i is the sum of a school effect and an age effect:

13.2 Two-way ANOVA • There are four models of potential interest. • Model 1: the expected comprehension score E(Yi) = i is the result of a school effect only (k = 0, k, k = 1,..., K):

13.2 Two-way ANOVA • There are four models of potential interest. • Model 2: the expected comprehension score E(Yi) = i is the result of an age effect only (j = 0, j, j = 1,..., J):

13.2 Two-way ANOVA • There are four models of potential interest. • Model 0: the expected comprehension score E(Yi) = i is not the result of a school effect nor an age effect:

13.2 Two-way ANOVA • The ANOVA table for comparing models 0, 1, 2 to model 3 is:

13.2 Two-way ANOVA • The ANOVA table for comparing model 0, 1, 2 and 3 is: compare model 2 vs model 3

13.2 Two-way ANOVA • The ANOVA table for comparing model 0, 1, 2 and 3 is: compare model 1 vs model 3

MT2004

MT2004

Presentation Transcript

MT2004

MT2004

MT2004

MT2004

MT2004