# Lecture 8: Regression Analysis (cont.) Source: https://www.youtube.com/watch?v=cMF_c2WNPyU Recap page: https://rapidrecap.app/video/cMF_c2WNPyU Generated: 2025-12-03T16:12:36.489+00:00 --- ## Quick Overview The lecture details the mathematical framework for Ordinary Least Squares (OLS) regression, deriving the best linear unbiased estimators (BLUE) via the normal equations and establishing the multivariate normality of the least squares estimate (beta hat) and residuals under the assumption of i.i.d. normal errors, culminating in the derivation of the t-distribution for hypothesis testing and the concept of generalized least squares (GLS) for non-constant error variance. **Key Points:** - The OLS criterion minimizes the sum-of-squares function Q(beta) = (y - x beta) transpose (y - x beta), solved by setting the first derivative to zero, yielding the normal equations: X transpose X beta hat = X transpose y. - If the X transpose X matrix is invertible (X must have full column rank), the least squares estimate is beta hat = (X transpose X) inverse X transpose y, and the fitted values utilize the hat matrix H, which is a projection matrix onto the column space of X. - Under the assumption of i.i.d. normal errors (Normal Linear Regression Model), the least squares estimate beta hat follows a multivariate normal distribution with mean beta and covariance matrix sigma squared (X transpose X) inverse. - The sum of squared residuals' expected value equals (n - p) sigma squared, providing an unbiased estimate for the error variance: sigma hat squared = (Sum of Squared Residuals) / (n - p). - The error vector (epsilon hat) and the least squares estimate (beta hat) are independent, allowing the construction of a t-statistic, tj, which follows a t-distribution with n - p degrees of freedom under the null hypothesis that beta j equals 0. - Standardizing covariates (X variables) by subtracting the mean and dividing by the standard deviation yields regression coefficients whose magnitude represents the impact of a one standard deviation move of the independent variable. - The Gauss-Markov theorem asserts that under specific assumptions (zero mean errors, covariance = sigma squared I), OLS estimates are the Best Linear Unbiased Estimates (BLUE); this extends to Generalized Least Squares (GLS) when the error covariance matrix is non-diagonal (known structure sigma). **Context:** Peter Kempthorne continues a lecture on linear regression modeling, formalizing the process of fitting a model by specifying criteria, finding estimators, checking assumptions, and potentially modifying the model. The core focus shifts from defining the OLS criterion via calculus minimization to establishing the probabilistic distribution of the resulting estimates, beta hat, using moment-generating functions, which requires assuming normally distributed, independent, and identically distributed (i.i.d.) errors. ## Detailed Analysis The lecture systematically covers the mathematical derivation and probabilistic properties of the Ordinary Least Squares (OLS) regression estimates. Minimizing the sum of squared errors leads to the normal equations, which, when solved (requiring full column rank for X), yield the estimate beta hat = (X transpose X) inverse X transpose y, involving the projection matrix H (the hat matrix). Introducing the Normal Linear Regression Model assumption (i.i.d. normal errors) allows the use of moment-generating functions to prove that beta hat is multivariate normal. This probabilistic framework enables the calculation of an unbiased estimator for the error variance, sigma hat squared, based on the sum of squared residuals divided by n-p degrees of freedom. A critical finding is the independence between the error vector (epsilon hat) and beta hat, which underpins the t-distribution used for testing individual regression coefficients (H0: beta j = 0). The discussion also covers standardizing covariates to interpret coefficient magnitudes in standard deviation units, calculating model fit via the multiple R-squared, and employing regression diagnostics like studentized residuals and Cook's distance. Finally, the lecture generalizes OLS using the Gauss-Markov theorem, showing that Generalized Least Squares (GLS) provides the BLUE when error variances are known but unequal (non-diagonal covariance matrix, often seen in time series), effectively performing a weighted regression. ### OLS Derivation and Normal Equations - Minimizing Q(beta) = (y - x beta) transpose (y - x beta) using first-order conditions leads to the normal equations; the solution is beta hat = (X transpose X) inverse X transpose y, provided X has full column rank. ### The Hat Matrix and Projections - The fitted values are given by y hat = H y, where H is the hat matrix, which functions as a projection matrix onto the column space of X; the residual projection matrix (I - H) is also verified as a projection. ### Normal Linear Model Distributions - Assuming i.i.d. normal errors yields a multivariate normal distribution for y, beta hat, and epsilon hat; the independence of beta hat and epsilon hat is proven via joint moment-generating functions. ### Hypothesis Testing and t-Distribution - The t-statistic, tj = (beta hat j - true beta j) / (sigma hat * sqrt(Cjj)), follows a t-distribution with n-p degrees of freedom, allowing testing of null hypotheses like beta j = 0; the history of the t-distribution (Student at Guinness) is briefly mentioned. ### Coefficient Interpretation and Standardization - Standardizing covariates allows interpretation of the resulting regression coefficients as the impact of a one standard deviation move of the independent variable, maintaining identical t-values and p-values compared to the original units. ### Model Diagnostics and Fit - Model quality is assessed by the multiple R-squared (the square of the correlation between observed and fitted values) and diagnostics like studentized residuals and Cook's distance, which measure case influence. ### Gauss-Markov and Generalized Least Squares - The Gauss-Markov theorem states OLS provides BLUE under Gauss-Markov assumptions (covariance = sigma squared I); GLS extends this by transforming data when error covariance is known but non-diagonal (sigma matrix), resulting in weighted regression estimates.