Ec 120BC – ECONOMETRICS B and C LECTURE NOTES
Foster, UCSD April 10, 2014
TOPIC 12. MULTIPLE REGRESSION
A. Multiple OLS Linear Regression Model
1. Assumptions of the OLS Model:
OLS Model Assumptions
A yi= β 1+ β x2 iβ w 3 …i+ β v + u k i i Linear specification
A Cov(X, u) = Cov(W, u) = Cov(V, u) = Regressors (X,W,V) uncorrelated with error
A E(u) = 0, i = 1…n Zeromean error
A Var(u)i= σ , i = 1…n Homoscedastic error
A Cov(u, i) j 0, i ≠ j Statisticallyindependent errors
A u ~ N(0, σ ) Implies NIID errors
a) There are k+1 parameters to estimate.
1) There are k regression coefficients β, j = 1jk, and σ , the variance of error term u.
2) The error terms u can ie thought of as the “vertical distance” (parallel to the Yaxis) from a data
point (y,ix,iw, i) ti the hyperplane β + β 1 + β 2 + … +3β V. k
3) β = intercept or constant term. It can be thought of as the coefficient of a “variable” which = 1,
i = 1…n.
β1+β 2 i
b) Fitted values of Y on the hyperplane:
• ̂yi=β +1 x +2 wi…+β 3 i ̂k i , i = 1…n
c) Equation of sample points relative to fitted hyperplane: [Compare to fitted line]
̂ ̂ ̂ ̂
• yi=β +1 x +2 wi…+β 3 +ui k i ̂ i , i = 1…n Ec 120BC MULTIPLE REGRESSION Page 2 of 19
d) Fitted residuals.
1) û =iy – i, ii= 1…n
2) û is an estimate of the true but unobserved u.
e) These are fundamentally the same model and assumptions presented in Topic 11. The only real
change is that we have more than one RHS regressor (independent variable).
1) We assume that the regressors (X, W, etc.) are each uncorrelated with (i.e., statistically
independent of) the error term (u).
2) We assume homoscedastic errors: Var(u) = σ ,ia constant, for all i = 1…n.
3) Given these assumptions, OLS estimators are unbiased, consistent, asymptotically normal, and
4) If u ~ N(0, σ ), then estimators β j are normally distributed, even for small n.
5) All ttests now have n–k degrees of freedom.
6) The estimator of σ isunow s = ∑ ui= SSR ;notethatSER=s = s 2 .
u n−k n−k u √ u
3. Multiple Regression and Ceteris paribus:
a) Suppose we want to measure the effect of water on growth rates of a species of plant.
1) A biologist would do a controlled scientific experiment. In a greenhouse with many plant
specimens, she would hold other factors affecting growth rate (temperature, hours of sunlight,
soil fertility) at constant levels, then vary the amount of water given to the plants and measure
changes in the growth rates. Such measurements are called “experimental data.”
2) Any growth rate differences can almost unambiguously be attributed to differences in the
amount of water, since all other factors were held constant (cet. paribus).
b) Economists can rarely do controlled laboratory experiments. [but: Vernon Smith]
1) If we want to measure the effect of price changes on shoe sales, we gather observations on shoe
prices and shoe sales from i = 1…n stores.
2) But ceteris is not paribus. Other factors that affect shoe sales are not constant between stores.
There are also variations in store location, atmosphere, weather, prices of other goods sold in the
3) Differences in sales volume among stores cannot be attributed solely to differences in prices
charged for shoes.
c) Multiple regression gives economics some semblance of scientific rigor by permitting the next best
thing to controlled experiments. Consider the water/growth rate example.
1) An economist would randomly sample n specimens of the plant species in the wild and measure
growth rate, water, temperature, sunlight, soil fertility, etc., for each one.
2) Then he would run the following regression:
GROW = βi + β1WATE2 + β TEMPi+ β3SUN + βiFERT4+ u, i 5 i i
3) β is an estimate of partial derivative ∂GROW/∂WATER, the marginal effect of water on
growth rate, ceteris paribus. (Recall that all other variables are held constant in partial
derivatives.) This is as close as economics can get to controlled experimental results.
• I call the constant term β 1 Many textbooks call it β ,0and S TATA calls it “const.”
• We have k regression coefficients in Y = β + 1 X +2β W + 3 + β V + u k
• There are k–1 explanatory variables (regressors X, W, … V)
• Have n–k degrees of freedom for many of our hypothesis tests Ec 120BC MULTIPLE REGRESSION Page 3 of 19 Ec 120BC MULTIPLE REGRESSION Page 4 of 19
B. Goodness of Fit
1. The Simple R Statistic:
a) Everything you learned about R in Topic 11 is still true.
R =1− SSR =r(y,̂ y) 0≤R ≤12
R is the proportion of the total variation in Y (TSS or m ) eyylained by the dependence of Y on
explanatory variables X, W, etc. The higher the proportion, the better the fit.
b) R has the characteristic that, if another variable is added to the RHS of the regression model, R 2
never falls, and usually rises (at least a little bit). An intuitive proof:
• Recall, first, that OLS minimizes SSR.
• Regress Y on X and W, and get j , j = 1…3, and SSR 1
• Add V to the model, regressing Y on X, W, and V, and get ̂ j , j = 1…4, and SSR . 2
• If SSR 2 SSR , 1hen setting ̂4 = 0 would make SSR = S2R . But1that means that the
OLS estimators ̂ did not minimize SSR , a contradiction.
• The contradiction implies that SSR ≤ 2SR , so 1 ≥ R .2 21
c) Because R can almost always be raised by the addition of more variables to the regression model, it
is flawed as a measure of goodness of fit.
1) Variables should be included or excluded from a model based on good theoretical reasoning.
But researchers usually want to demonstrate that their theory fits the facts, and have a vested
interest in obtaining a high R . Many succumb to the temptation to add variables simply to
improve the appearance of their research results. This is “shotgun regression” or “fishing,” and
is very bad form!
2) The adjusted R measure and Akaike Information Criterion are goodness of fit statistics that
overcome the potential for abuse of R .
2. ‘R or “Adjusted R ”:
a) R , also called “R adjusted for degrees of freedom,” is a goodness of fit measure that is harder to
2 n−1 2 s u
R =1− ( ) (1−R =1−) 2
n−k s y
b) When variables are added to the model, R rises, but so does k, and‘R may rise or fall.
1) If the extra variable greatly increases R , it is probably relevant to the model and should be
added.‘R will rise, indicating a better fit and better model.
2) If the extra variable only slightly increases R , it is probably irrelevant and should be
omitted.‘R will fall, indicating a worse fit and poorer model.
3) Add2ng a 2ariable will raise‘R if its c2efficient has a |tratio| > 1.
4) ‘R 0 Reject H 0f t ≥ α
• Leftsided alternative H 0 β ≥ β0; H1: β 2. Ec 120BC MULTIPLE REGRESSION Page 7 of 19
-tα/2 0 tα/2 t(n–k)
Rej H o
Rej H o
5. Wald FTests on Subsets of Coefficients:
a) For this section, consider the model Y = β + β X 1 β W2 β Q +3β M + 4 H + u 5 6
b) We often test if 2 or more coefficients equal certain values (a “joint hypothesis”).
1) Because of the problem of multicollinearity (discussed later), we cannot simply do 2 or more
separate singlecoefficient ttests.
2) The preferred method of testing sets of coefficients is called a Wald Test.
c) “Zero hypotheses” – suppose we want to test that m F(m, n–k) . α
Fα F(m, n–k)
Rej H 0
at α level Ec 120BC MULTIPLE REGRESSION Page 8 of 19
d) Use of the reported (“overall”) Fstatistic.
1) Every regression program prints an Fstatistic for the regression, which is the F for testing
that all of the regression coefficients except the constant = 0, a special case of a “zero
• H 0 β 2= β 3= … = β =k0 H : 1ot so (at least one ≠ 0)
• F= , where m = k–1 restrictions in numerator
2) Choose α and find F forαv = 1–1, v = n2k df in numerator and denominator.
3) If you cannot reject H , 0hen the model is totally misspecified none of the regressors are
e) The testing method is similar for nonzero hypotheses.
1) For example H : 0 = 2, β = 3 H : N1t so
2) If H 0s true, then the regression model becomes
• Y = β 1 2X + W+ β Q + 4 + u, or Y – 2X – W = β + β Q + …1+ u 4
3) Generate new variable Y* = Y–2X–W, then run two regressions and record each R . 2
• Unrestricted y i β 1+ β x2 iβ w+3βiq + 4 i + β5hi+ u,6 i= 1…i
• Restricted y = β + β q + β m + β h + u, i = 1…n
i 1 4 i 5 i 6 i i
The second regression restricts β =22 and β = 3.
(R uR r)
4) Compute test statistic 2 .
• m = number of restricted coefficients.
• If H0 is true, F ̃~ F(m, n–k).
• For our example, v =12, v =2n–6.
f) Linear restrictions – occasionally we want to test whether sums, differences, or other linear
combinations of coefficients equal certain values.
1) For example H : 0 +2β =31 H : N1t so
2) If H 0s true, then β 3= 1–β ,2and the regression model becomes Ec 120BC MULTIPLE REGRESSION Page 9 of 19
• Y = β 1+ β X2+ (1–β )W+2β Q + … 4 u = β + β (X–W)1+ W+2β Q + … + u 4
• So Y – W = β + 1 (X 2 W) + β Q + β 4 + β H 5 u 6