Least Squares Estimator

Definition

S (b) = E [(Y - X^{T} b^{2})]

The moment estimator of $S (b)$ is

\hat{S} (b) = \frac{1}{n} \sum_{i = 1}^{n} (Y_{i} - X_{i}^{T} b)^{2}

define $\hat{β}$ as the minimizer of $\hat{S} (b)$

\hat{β} \equiv \underset{b \in R^{k}}{\arg min} \hat{S} (b)

and call $\hat{β}$ as the least squares estimator of $β$

注意， $β$ 是总体参数，是一个数； $\hat{β}$ 是一个样本估计量，是一个随机变量，其随机性来源于抽样。

定义 Sum of Squared Risiduals 为 $S S R (b) \equiv n \hat{S} (b) = \sum_{i = 1}^{n} (Y_{i} - X_{i}^{T} β)^{2}$ ，易知

\hat{β} \equiv \underset{b \in R^{k}}{\arg min} \hat{S} (b) = \underset{b \in R^{k}}{\arg min} S S R (b)

写作矩阵形式

\hat{β} \equiv \underset{b \in R^{k}}{\arg min} S S R (b) = \underset{b \in R^{k}}{\arg min} e^{T} e

先展开再求导：

\begin{aligned} e^{T} e & = (Y - X b)^{T} (Y - X b) \\ = Y^{T} Y - Y^{T} X b - b^{T} X^{T} Y + b^{T} X^{T} X b \\ = Y^{T} Y - 2 b^{T} X^{T} Y + b^{T} X^{T} X b \end{aligned}

F.O.C.（称之为正规方程组）

\frac{\partial S S R (b)}{\partial b} = - 2 X^{T} Y + 2 X^{T} X b = 0

therefore

\hat{β} \equiv {(X^{T} X)}^{- 1} (X^{T} Y)

S.O.C.

\frac{\partial^{2} S S R (b)}{\partial b \partial b^{T}} = 2 X^{T} X

这要求 $X^{T} X$ 是正定矩阵。

也可以先求导再展开：
F.O.C. （前导不变，后导转置）

\begin{aligned} \frac{\partial S S R (b)}{\partial b} & = \frac{\partial e^{T} (b) e (b)}{\partial b} \\ = \frac{\partial e^{T} (b)}{\partial b} e (b) + {[e^{T} (b) \frac{\partial e (b)}{\partial b}]}^{T} \\ (※) & = 2 \frac{\partial e^{T} (b)}{\partial b} e (b) = 0 \\ = - 2 X^{T} (Y - X b) = 0 \end{aligned}

其中

\frac{\partial e^{T} (b)}{\partial b} = \frac{\partial (Y - X b)^{T}}{\partial b} = - X^{T}

根据前面的（※）可知

X^{T} e = 0_{k \times 1}

若自变量包含常数项，则有

[\begin{matrix} 1_{n}^{T} \\ X_{1}^{T} \\ ⋮ \\ X_{k - 1}^{T} \end{matrix}] e = 0 ⟹ 1_{n}^{T} e = 0 ⟹ \bar{e} = 0

若自变量包含虚拟变量，则残差的分组均值为零。

S S R (b) = \sum_{i = 1}^{n} (Y_{i} - X_{i} b)^{2} = (\sum_{i = 1}^{n} Y_{i}^{2}) + 2 b (\sum_{i = 1}^{n} X_{i} Y_{i}) + b^{2} (\sum_{i = 1}^{n} X_{i}^{2})

F.O.C

\frac{d S S R (b)}{d b} = 2 (\sum_{i = 1}^{n} X_{i} Y_{i}) + 2 b (\sum_{i = 1}^{n} X_{i}^{2}) = 0

therefore

\hat{β} \equiv \frac{\sum_{i = 1}^{n} X_{i} Y_{i}}{\sum_{i = 1}^{n} X_{i}^{2}}

\begin{aligned} S S R (b) = & \sum_{i = 1}^{n} (Y_{i} - X_{i}^{T} b) (Y_{i} - X_{i}^{T} b) \\ = & \sum_{i = 1}^{n} Y_{i}^{2} - 2 \sum_{i = 1}^{n} Y_{i} X_{i}^{T} b + \sum_{i = 1}^{n} X_{i}^{T} b X_{i}^{T} b \\ = & \sum_{i = 1}^{n} Y_{i}^{2} - 2 b^{T} \sum_{i = 1}^{n} X_{i} Y_{i} + b^{T} \sum_{i = 1}^{n} X_{i} X_{i}^{T} b \end{aligned}

注意： $X_{i}^{T} b$ 是一个数因此转置后不变，即 $X_{i}^{T} b = b^{T} X_{i}$ ，这样变换将有助于后续应用矩阵求导公式。

F.O.C.

\begin{aligned} \frac{d S S R (b)}{d b} & = - 2 \sum_{i = 1}^{n} X_{i} Y_{i} + [(\sum_{i = 1}^{n} X_{i} X_{i}^{T}) + {(\sum_{i = 1}^{n} X_{i} X_{i}^{T})}^{T}] b \\ = - 2 \sum_{i = 1}^{n} X_{i} Y_{i} + 2 \sum_{i = 1}^{n} X_{i} X_{i}^{T} b \\ = 0 \end{aligned}

注意： $\sum_{i = 1}^{n} X_{i} X_{i}^{T}$ 是一个对称矩阵所以转置后不变。

therefore

\hat{β} \equiv {(\sum_{i = 1}^{n} X_{i} X_{i}^{T})}^{- 1} (\sum_{i = 1}^{n} X_{i} Y_{i})

S.O.C

\frac{\partial S S R (b)}{\partial b \partial b^{T}} = 2 \sum_{i = 1}^{n} X_{i} X_{i}^{T}

which is a positive semi-definite matrix

二阶条件对应总体模型的假设 $E (X X^{T}) ≻ O$