Goodness-of-Fit

Coefficient of determination

根据 Projection Matrix 和 Annihilator Matrix 的性质

\begin{aligned} {\hat{Y}}^{T} \hat{Y} & = (P Y)^{T} (P Y) = Y^{T} P Y \\ = Y^{T} (I - M) Y = Y^{T} Y - (M Y)^{T} (M Y) \\ = Y^{T} Y - e^{T} e \end{aligned}

由此得到平方和分解法则

Y^{T} Y = {\hat{Y}}^{T} \hat{Y} + e^{T} e

回归方程中心化后应用平方和分解法则可得

\begin{aligned} (M_{0} Y)^{T} (M_{0} Y) & = (M_{0} \hat{Y})^{T} (M_{0} \hat{Y}) + e^{T} e \\ Y^{T} M_{0} Y & = \hat{Y} M_{0} \hat{Y} + e^{T} e \\ \underset{SST}{\underset{⏟}{(Y - \bar{Y})^{T} (Y - \bar{Y})}} & = \underset{SSE}{\underset{⏟}{(\hat{Y} - \bar{Y})^{T} (\hat{Y} - \bar{Y})}} + \underset{SSR}{\underset{⏟}{e^{T} e}} \end{aligned}

定义 uncenterd R-squared 为

R_{u}^{2} = \frac{{\hat{Y}}^{T} \hat{Y}}{Y^{T} Y} = 1 - \frac{e^{T} e}{Y^{T} Y}

定义 centered R-squared 为

R^{2} = \frac{SSE}{SST} = 1 - \frac{SSR}{SST}

uncenterd R-quared 可以看作将模型未解释的信息和总信息（即没有模型）比较，而 centered R-squared 是将模型未解释的信息和最简模型（常数项回归）未解释的信息比较。后者更切实际也更常用。

定义 adjust R-squared 为

{\bar{R}}^{2} = 1 - \frac{SSR / n - k}{SST / n - 1} = 1 - (1 - R^{2}) \frac{n - 1}{n - k}

其中， $SST$ 的自由度相当于 $Y$ 的维度减去 $\bar{Y}$ 占用的一个维度； $SSR$ 的自由度相当于残差的维度，而残差的维度即在投影过程中失去的维度，恰好等于消去矩阵的迹 $Tr (M) = n - k$ ，；类似地， $SSE$ 的自由度相当于投影到 $s p a n (X)$ 的维度，恰好等于投影矩阵的迹 $Tr (P) = k$

Theorem $R^{2}$ 相当于因变量真值 $Y$ 和拟合值 $\hat{Y}$ 的样本相关系数的平方，即

R^{2} = r (Y, \hat{Y})^{2}

Proof.
样本相关系数为

r (Y, \hat{Y}) = \frac{C o v (Y, \hat{Y})}{\sqrt{V a r (Y) V a r (\hat{Y})}} = \frac{(M_{0} Y)^{T} (M_{0} \hat{Y})}{\sqrt{(Y^{T} M_{0} Y) ({\hat{Y}}^{T} M_{0} \hat{Y})}}

其中 $M_{0} Y = M_{0} (\hat{Y} + e) = M_{0} \hat{Y}$ ，因此

r (Y, \hat{Y})^{2} = \frac{({\hat{Y}}^{T} M_{0} \hat{Y})^{2}}{(Y^{T} M_{0} Y) ({\hat{Y}}^{T} M_{0} \hat{Y})} = \frac{{\hat{Y}}^{T} M_{0} \hat{Y}}{Y^{T} M_{0} Y} = \frac{SSE}{SST} = R^{2}

附录：no matrix

总离差为

Y_{i} - \bar{Y} = (Y_{i} - {\hat{Y}}_{i}) + ({\hat{Y}}_{i} - \bar{Y})

总离差平方和为

\sum (Y_{i} - \bar{Y})^{2} = \sum [{\hat{e}}_{i} + ({\hat{Y}}_{i} - \bar{Y})]^{2} = \sum {\hat{e}}_{i}^{2} + \sum ({\hat{Y}}_{i} - \bar{Y})^{2} + 2 \sum {\hat{e}}_{i} ({\hat{Y}}_{i} - \bar{Y})

其中

\begin{aligned} \sum {\hat{e}}_{i} ({\hat{Y}}_{i} - \bar{Y}) & = \sum {\hat{e}}_{i} {\hat{Y}}_{i} + \bar{Y} \sum {\hat{e}}_{i} \\ = \sum {\hat{e}}_{i} (X_{i}^{T} β) \\ = β \sum [{\hat{e}}_{i} X_{i 1}, \dots, {\hat{e}}_{i} X_{i k}] \\ = 0 \end{aligned}

定义

\begin{aligned} SST & \equiv \sum (Y_{i} - \bar{Y})^{2} \\ SSE & \equiv \sum ({\hat{Y}}_{i} - \bar{{\hat{Y}}_{i}})^{2} = \sum ({\hat{Y}}_{i} - \bar{Y})^{2} \\ SSR & \equiv \sum ({\hat{e}}_{i} - \bar{{\hat{e}}_{i}})^{2} = \sum {\hat{e}}_{i}^{2} \end{aligned}

从而有

SST = SSE + SSR

定义

R^{2} \equiv \frac{SSE}{SST} = 1 - \frac{SSR}{SST} = \frac{\sum ({\hat{Y}}_{i} - \bar{Y})^{2}}{\sum (Y_{i} - \bar{Y})^{2}}

可以证明， $R^{2}$ 等价于 $Y$ 与 $\hat{Y}$ 相关系数的平方