1 / 42

Topic4 Ordinary Least Squares

Topic4 Ordinary Least Squares. Suppose that X is a non-random variable Y is a random variable that is affected by X in a linear fashion and by the random variable e with E( e ) = 0 That is, E(Y) = b 1 + b 2 X Or, Y = b 1 + b 2 X + e. Y. Observed points.

noam
Télécharger la présentation

Topic4 Ordinary Least Squares

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Topic4 Ordinary Least Squares

  2. Suppose that X is a non-random variable • Y is a random variable that is affected by X in a linear fashion and by the random variable e with E(e) = 0 That is, E(Y) = b1 + b2X Or, Y = b1 + b2X + e

  3. Y . . . Observed points . . O X

  4. Y Actual Line . . . . . Y= b1 + b2x O X

  5. Y . Actual Line . . . . Y= b1 + b2x O X

  6. Y . Actual Line . . . . Y= b1 + b2x O X

  7. Y . Actual Line . . . . Y= b1 + b2x O X

  8. Y . Actual Line . . . Y= b1 + b2x . O X

  9. Y . . Actual Line . . Y= b1 + b2x . O X

  10. Y= b1 + b2x Fitted Line Y . . . Actual Line C . B . A . Y= b1 + b2x . BC is an error of Estimation AC is an effect of the random factor O X

  11. The Ordinary Least Squares (OLS) estimates are obtained by minimising the sum of the squares of each of these errors. • The OLS estimates are obtained from the values of X and the actual Y values (YA) as follows:

  12. Error of estimation (e)  |YA–YE | where YE is the estimated value of Y. Se2S [YA–YE ]2 Se2S [YA–(b1 + b2 X)]2 dSe2/db12S[YA–(b1 + b2X)] (-1) =0 dSe2 /db22S [YA–(b1 + b2X)] (-X) = 0

  13. S [Y–(b1 + b2X)] (-1) = 0 -NYMEAN + N b1 + b2NXMEAN = 0 b1= YMEAN – b2XMEAN ….. (1)

  14. dSe2/db2 2S [Y–(b1+ b2X)] (-X) = 0 S [Y–(b1 + b2X)] (-X) = 0 b1SX –b2SX2 = SXY ………..(2) b1= YMEAN - b2XMEAN ….. (1)

  15. These estimates are given below (with the superscripts for Y dropped). b^1 = (∑Y)(ΣX2) – (∑X)(∑XY) N∑ X2 - (∑X)2 b^2 = N∑YX – (∑X)(∑Y) N∑ X2 - (∑X)2

  16. Alternatively, b^1= YMEAN - b^2XMEAN b^2 = Covariance(X,Y) Variance(X)

  17. Two Important Results (a)     ei(Yi– YiE) = 0 and (b) X2ieiX2i(Yi– YiE) = 0 where YiEis the estimated value of Yi. X2i is the same as Xi from before Proof: (Yi– YiE) =S(Yi– b^1 - b^2 X2i) =SYi–Sb^1 - Sb^2 X2i = nYMEAN – nb^1 - nb^2 XMEAN = n(YMEAN – b^1 - b^2 XMEAN) = 0 [ since b^1 = YMEAN - b^2XMEAN ]

  18. See the lecture notes for a proof of part (b) Total sum of squares (TSS) (Yi– YMEAN )2 Residual sum of squares (RSS)  (Yi– YiE)2 Explained sum of squares (ESS)  (YiE– YMEAN )2

  19. To prove that TSS = RSS + ESS TSS ≡ S(Yi– YMEAN)2 = S{(Yi– YiE + YiE– YMEAN)}2 = S(Yi– YiE)2 + S(YiE– YMEAN)}2 +2S(Yi– YiE)(YiE– YMEAN) = RSS + ESS +2S(Yi– YiE)(YiE– YMEAN)

  20. S(Yi– YiE)(YiE– YMEAN) • S(Yi– YiE)(YiE ) -YMEANS(Yi– YiE) • S(Yi– YiE)(YiE ) [by (a) above] S(Yi– YiE)(YiE ) =S(Yi– YiE)( b^1 + b^2 Xi) =b^1 S(Yi– YiE) + b^2 Xi(Yi– YiE) = 0 [by (a) and (b) above]

  21. R2 ≡ ESS/TSS Since TSS = RSS + ESS, it follows that 0 R2  1

  22. Topic 5 Properties of Estimators

  23. In the discussion that follows, q^ is an estimator of the parameter of interest, q Bias of q^ ≡ E(q^) - q q^ is unbiased if Bias of q^ = 0. q^ is negatively biased if Bias of q^ < 0. q^ is positively biased if Bias of q^ > 0.

  24. Mean Squared Errors (MSE) of estimation for q^  is given as MSEq^ ≡ E[(q^-q)]2 MSEq^ ≡ E[(q^-q)2] ≡ E[{q^-E(q^) +E(q^)- q}2] ≡ E[{q^-E(q^)}2] + E[{E(q^)- q}2] + 2E[{q^-E(q^)}*{E(q^)- q}] ≡ Var(q^) + {E(q^)- q}2+ 2E[{q^-E(q^)}*{E(q^)- q}]

  25. Now, E[{q^-E(q^)}*{E(q^)- q}] ≡ {E(q^)-E(q^)}*{E(q^)- q}] ≡ 0*{E(q^)- q}] = 0 MSEq^≡ Var(q^) + {E(q^)- q}2 MSEq^ ≡ Var(q^) + (bias)2 .

  26. If q^ is unbiased, that is, if E(q ^)- q = 0. then we have, MSEq^ ≡ Var(q^) An unbiased estimator q^ of a parameter qis efficientif and only if it has the smallest variance of all unbiased estimators. That is, for any other unbiased estimator p of q, Var(q^)≤ Var(p)

  27. An estimator q^is said to be consistentif it converges inprobability toq. That is, Limn  Prob(|q^-q | > e) = 0 for every e> 0.

  28. When the above condition holds, q^ is said to be the probability limit of q, that is, plim q^ = q Sufficient conditions for consistency: If the mean of q^converges to q and var(q^) converges to zero (as n approaches ) then q^is consistent.

  29. That is, q^n is consistent if it can be shown that Lim n  E(q^n) = q And Lim n  Var(q^n) = 0

  30. The Regression Model with TWO Variables The Model :: Y = b1 + b2X + e Y is the DEPENDENT variable X is the INDEPENDENT variable Yi= b1X1i+ b2X2i+ ei

  31. Yi= b1X1i+ b2X2i+ ei Here X1i ≡ 1 for all i and X2 is nothing but X . The OLS estimates b^1 and b^2 are sample statistics used to estimate b1andb2respectively

  32. (1a)X2 is non-random (chosen by the investigator) Assumptions about X2: (1b) Random sampling is performed from a population of fixed values of X2 . (1c) : Lim (1/n) S(x22i) = Q > 0 n [ where x2i X2i – X2MEAN.] (1c) : Lim (1/n)S(X2i) = P > 0 n 

  33. 2a. E() = 0 Assumptions about the disturbance term e 2b. Var(ei) = 2 for all i. Homoskedasticity 2c. Cov(ei, ej ) = 0 for i  j. (The  values are uncorrelated across observations). 2d. The ei all have a normal distribution

  34. Proof: b^2 = Covariance(X,Y) Variance(X) Result :b^2 is linear in the dependent variable Yi b^2 = S(Yi–YMEAN )(Xi–XMEAN ) S(Xi–XMEAN )2

  35. b^2 = SYi(Xi–XMEAN ) + K S(Xi–XMEAN )2 =S CiYi + K where the Ci andK are constants

  36. Therefore, b^2 is a linear function of Yi Since, Yi= b1X1i+ b2X2i+ ei b^2 is a linear function of ei and hence is normally distributed

  37. Similarly, b^1 is a linear function of Yi (and hence ei ) and is normally distributed Both b^1 and b^2 are unbiased estimates of b1 and b2 respectively. That is, E( b^1 ) = b1 and E( b^2 ) = b2

  38. Each of b^1 and b^2 is an efficientestimators of b1 and b2 respectively. Thus, each of b^1 and b^2 is a Best (efficient) Linear (in the dependent variable Yi ) Unbiased Estimatorof b1 and b2 respectively. Also, Each of b^1 and b^2 is a consistent estimator of b1 and b2 respectively.

  39. Var(b^1 ) = s2 (1/n +X 2mean2/Sx2i2) Var(b^2 ) = s2 /Sx2i2) . Cov(b^1, b^2 ) = -s2 X 2mean/Sx2i2

  40. LimVar(b^2 ) n  = Lim s2/Sx2i2 n  = Lim s2/n/Sx2i2/n n  = 0/Q [using assumption (1c)] = 0

  41. Because b^2 is an unbiased estimator of b2 and LimVar(b^2 ) = 0 n  b^2 is a consistent estimator of b2

  42. The variance of the random term, s2, is not known To perform statistical analysis, we estimate s2by s^2  RSS/(n-2) This is because s^2 is an unbiased estimator of s2

More Related