Lecture 8: Regression Analysis (cont.)

Quick Overview

The lecture details the mathematical framework for Ordinary Least Squares (OLS) regression, deriving the best linear unbiased estimators (BLUE) via the normal equations and establishing the multivariate normality of the least squares estimate (beta hat) and residuals under the assumption of i.i.d. normal errors, culminating in the derivation of the t-distribution for hypothesis testing and the concept of generalized least squares (GLS) for non-constant error variance.

Key Points: The OLS criterion minimizes the sum-of-squares function Q(beta) = (y - x beta) transpose (y - x beta), solved by setting the first derivative to zero, yielding the normal equations: X transpose X beta hat = X transpose y. If the X transpose X matrix is invertible (X must have full column rank), the least squares estimate is beta hat = (X transpose X) inverse X transpose y, and the fitted values utilize the hat matrix H, which is a projection matrix onto the column space of X. Under the assumption of i.i.d. normal errors (Normal Linear Regression Model), the least squares estimate beta hat follows a multivariate normal distribution with mean beta and covariance matrix sigma squared (X transpose X) inverse. The sum of squared residuals' expected value equals (n - p) sigma squared, providing an unbiased estimate for the error variance: sigma hat squared = (Sum of Squared Residuals) / (n - p). The error vector (epsilon hat) and the least squares estimate (beta hat) are independent, allowing the construction of a t-statistic, tj, which follows a t-distribution with n - p degrees of freedom under the null hypothesis that beta j equals 0. Standardizing covariates (X variables) by subtracting the mean and dividing by the standard deviation yields regression coefficients whose magnitude represents the impact of a one standard deviation move of the independent variable. The Gauss-Markov theorem asserts that under specific assumptions (zero mean errors, covariance = sigma squared I), OLS estimates are the Best Linear Unbiased Estimates (BLUE); this extends to Generalized Least Squares (GLS) when the error covariance matrix is non-diagonal (known structure sigma).