最小平方估计量

Abstract

遵循最小平方被估量（estimand）的构造思路，推导相应的最小平方估计量（estimator）。

已知总体最小平方误差定义为

S (b) = E [(Y - X^{T} b^{2})]

其矩估计量（moment estimator）为

\hat{S} (b) = \frac{1}{n} \sum_{i = 1}^{n} (Y_{i} - X_{i}^{T} b)^{2}

定义最小平方估计量 $\hat{β}$ 为

\hat{β} \equiv \underset{b \in R^{k}}{\arg min} \hat{S} (b)

注意， $β$ 是总体参数，是一个数； $\hat{β}$ 是一个样本估计量，是一个随机变量，其随机性来源于抽样。

求解估计量

定义残差为

{\hat{e}}_{i} \equiv Y_{i} - X_{i}^{T} \hat{β}

定义残差平方和（sum of squared risiduals）为

S S R (\hat{β}) = \sum_{i = 1}^{n} ({\hat{e}}_{i})^{2} = {\hat{e}}^{T} \hat{e}

显然残差平方和是目标函数的等价形式

S S R (b) = n \hat{S} (b) ⟹ \hat{β} = \underset{b \in R^{k}}{\arg min} S S R (b) = \underset{b \in R^{k}}{\arg min} {\hat{e}}^{T} \hat{e}

先展开再求导：

\begin{aligned} {\hat{e}}^{T} \hat{e} & = (Y - X b)^{T} (Y - X b) \\ = Y^{T} Y - Y^{T} X b - b^{T} X^{T} Y + b^{T} X^{T} X b \\ = Y^{T} Y - 2 b^{T} X^{T} Y + b^{T} X^{T} X b \end{aligned}

F.O.C.（称之为正规方程组）

\frac{\partial S S R (b)}{\partial b} = - 2 X^{T} Y + 2 X^{T} X b = 0

therefore

\hat{β} \equiv {(X^{T} X)}^{- 1} (X^{T} Y)

S.O.C.

\frac{\partial^{2} S S R (b)}{\partial b \partial b^{T}} = 2 X^{T} X

这要求 $X^{T} X$ 是正定矩阵。

先求导再展开：

F.O.C. （前导不变，后导转置）

\begin{aligned} \frac{\partial S S R (b)}{\partial b} & = \frac{\partial {\hat{e}}^{T} (b) \hat{e} (b)}{\partial b} \\ = \frac{\partial {\hat{e}}^{T} (b)}{\partial b} \hat{e} (b) + {[{\hat{e}}^{T} (b) \frac{\partial \hat{e} (b)}{\partial b}]}^{T} \\ = 2 \frac{\partial {\hat{e}}^{T} (b)}{\partial b} \hat{e} (b) \\ = - 2 X^{T} (Y - X b) \\ = 0 \end{aligned}

其中

\frac{\partial {\hat{e}}^{T} (b)}{\partial b} = \frac{\partial (Y - X b)^{T}}{\partial b} = - X^{T}

注意到最后一个等式相当于

X^{T} \hat{e} = 0_{k \times 1}

若自变量包含常数项，则有

[\begin{matrix} 1_{n}^{T} \\ X_{1}^{T} \\ ⋮ \\ X_{k - 1}^{T} \end{matrix}] \hat{e} = 0 ⟹ 1_{n}^{T} \hat{e} = 0 ⟹ \bar{e} = 0

推广：若自变量包含虚拟变量，则残差的分组均值为零。（详见 Frisch-Waugh-Lovell Theorem）

S S R (b) = \sum_{i = 1}^{n} (Y_{i} - X_{i} b)^{2} = (\sum_{i = 1}^{n} Y_{i}^{2}) + 2 b (\sum_{i = 1}^{n} X_{i} Y_{i}) + b^{2} (\sum_{i = 1}^{n} X_{i}^{2})

F.O.C

\frac{d S S R (b)}{d b} = 2 (\sum_{i = 1}^{n} X_{i} Y_{i}) + 2 b (\sum_{i = 1}^{n} X_{i}^{2}) = 0

therefore

\hat{β} \equiv \frac{\sum_{i = 1}^{n} X_{i} Y_{i}}{\sum_{i = 1}^{n} X_{i}^{2}}

\begin{aligned} S S R (b) = & \sum_{i = 1}^{n} (Y_{i} - X_{i}^{T} b) (Y_{i} - X_{i}^{T} b) \\ = & \sum_{i = 1}^{n} Y_{i}^{2} - 2 \sum_{i = 1}^{n} Y_{i} X_{i}^{T} b + \sum_{i = 1}^{n} X_{i}^{T} b X_{i}^{T} b \\ = & \sum_{i = 1}^{n} Y_{i}^{2} - 2 b^{T} \sum_{i = 1}^{n} X_{i} Y_{i} + b^{T} \sum_{i = 1}^{n} X_{i} X_{i}^{T} b \end{aligned}

注意： $X_{i}^{T} b$ 是一个数因此转置后不变，即 $X_{i}^{T} b = b^{T} X_{i}$ ，这样变换将有助于后续应用矩阵求导公式。

F.O.C.

\begin{aligned} \frac{d S S R (b)}{d b} & = - 2 \sum_{i = 1}^{n} X_{i} Y_{i} + [(\sum_{i = 1}^{n} X_{i} X_{i}^{T}) + {(\sum_{i = 1}^{n} X_{i} X_{i}^{T})}^{T}] b \\ = - 2 \sum_{i = 1}^{n} X_{i} Y_{i} + 2 \sum_{i = 1}^{n} X_{i} X_{i}^{T} b \\ = 0 \end{aligned}

注意： $\sum_{i = 1}^{n} X_{i} X_{i}^{T}$ 是一个对称矩阵所以转置后不变。

therefore

\hat{β} \equiv {(\sum_{i = 1}^{n} X_{i} X_{i}^{T})}^{- 1} (\sum_{i = 1}^{n} X_{i} Y_{i})

S.O.C

\frac{\partial S S R (b)}{\partial b \partial b^{T}} = 2 \sum_{i = 1}^{n} X_{i} X_{i}^{T}

which is a positive semi-definite matrix

二阶条件对应总体模型的假设 $E (X X^{T}) ≻ O$