Deriving Regression coefficient from Residual Sum of Squares
When Linear Regression equation, \(y=\hat{\alpha} + \hat{\beta} x\), is found from the data, the formula for their coefficients, \(\hat{\alpha}\) and \(\hat{\beta}\), are the followings.
$$ \hat{\beta} = r_{xy}\frac{s_{y}}{s_{x}} $$
$$ \hat{\alpha} = \bar{y} – \hat{\beta}\bar{x_i} $$
\(r_{xy} \): correlation coefficient
\(s_{x} \): standard deviation of \(x\), \(s_{y} \): standard deviation of \(y\)
Here we will explain how to derive these equation from the residual sum of squares.
Derivation
We will use the following residual sum of regression below.
$$ S(\hat{\alpha}, \hat{\beta}) = \sum^{n}_{i=1}(y_i – \hat{y})^2 = \sum^{n}_{i=1}(y_i – (\hat{\alpha} + \hat{\beta} x_i ))^2 $$
To find \(\hat{\alpha}\) and \(\hat{\beta}\) to minimize \(S(\hat{\alpha}, \hat{\beta})\), we will differentiate partially with \(\hat{\alpha}, \hat{\beta}\).
$$ \frac{\partial S}{\partial \hat{\alpha}} = 2\times(-1)\times\sum^{n}_{i=1}(y_i – \hat{\alpha} – \hat{\beta} x_i ) =0 \\ \sum^{n}_{i=1}y_i = \sum^{n}_{i=1}\hat{\alpha} + \sum^{n}_{i=1}\hat{\beta}x_i \\ = n\hat{\alpha} + \hat{\beta}\sum^{n}_{i=1}x_i$$
Dividing both side by \(n\)
$$ \frac{\sum^{n}_{i=1}y_i}{n} = \hat{\alpha} + \frac{\sum^{n}_{i=1}x_i}{n} \ \bar{y} = \hat{\alpha} + \hat{\beta}\bar{x} \\ \hat{\alpha} = \bar{y} – \hat{\beta}\bar{x}$$
we can get \(\hat{\alpha}\).
Now to get \(\hat{\beta}\), we will differentiate \(S\) partially by \(\hat{\beta}\).
$$ \frac{\partial S}{\partial \hat{\beta}} = 2\times(-1)\times\sum^{n}_{i=1}(y_i – \hat{\alpha} – \hat{\beta} x_i ) x_i =0 \\ \sum^{n}_{i=1}x_i y_i = \sum^{n}_{i=1}\hat{\alpha} x_i + \sum^{n}_{i=1}\hat{\beta} x_i^2 \ \sum^{n}_{i=1}x_i y_i= \hat{\alpha}\sum^{n}_{i=1}x_i + \hat{\beta}\sum^{n}_{i=1}x_i^2$$
Dividing both sides by \(n\)
$$ \frac{\sum^{n}_{i=1}x_iy_i}{n} = \hat{\alpha}\frac{\sum^{n}_{i=1}x_i}{n} + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}} \\ \frac{\sum^{n}_{i=1}x_iy_i}{n} = \hat{\alpha}\bar{x} + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}} $$
Here we can use \(\hat{\alpha} = \bar{y} – \hat{\beta}\bar{x}\) that we already got.
$$ \frac{\sum^{n}_{i=1}x_iy_i}{n} = \bar{x}\bar{y} – \hat{\beta}\bar{x}^2 + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}} \\ = \bar{x}\bar{y} + \hat{\beta}(\frac{\sum^{n}_{i=1}x_i^2}{n}-\bar{x}^2)$$
\( \frac{\sum^{n}_{i=1}x_i^2}{n}-\bar{x}^2\) is variance of \(x\) then we define it as \(s_{xx}\).
$$\frac{\sum^{n}_{i=1}x_iy_i}{n} = \bar{x}\bar{y} + \hat{\beta}s_{xx} \\ \frac{\sum^{n}_{i=1}x_iy_i }{n}- \bar{x}\bar{y} = \hat{\beta}s_{xx}$$
Here we can change \(\frac{\sum^{n}_{i=1}x_iy_i}{n} – \bar{x}\bar{y}\) to the following.
$$ \frac{\sum^{n}_{i=1}x_iy_i}{n} + \bar{x}\bar{y} – 2\bar{x}\bar{y} \\ = \frac{\sum^{n}_{i=1}x_iy_i}{n} + \frac{\sum^{n}_{i=1}\bar{x}\bar{y}}{n} – \frac{\sum^{n}_{i=1}x_i}{n}\bar{y} – \frac{\sum^{n}_{i=1}y_i}{n}\bar{x} = \frac{\sum^{n}_{i=1}(x_iy_i – x_i\bar{y} – \bar{x}y_i + \bar{x}\bar{y})}{n}$$
\(\frac{\sum^{n}_{i=1}x_iy_i}{n}- \bar{x}\bar{y}\) is covariance of \(x, y\) then we define it as \(s_{xy}\).
$$s_{xy} = \hat{\beta}s_{xx} \\ \hat{\beta} = \frac{s_{xy}}{s_{xx}} = \frac{s_{xy}}{s_xs_y}\cdot \frac{s_y}{s_x} $$
We express correlation coefficient as \(r_{xy} = \frac{s_{xy}}{s_{x}s_{y}}\)
$$ \hat{\beta} = r_{xy}\frac{s_y}{s_x}$$
Finally we can get \(\hat{\alpha}\) and \(\hat{\beta}\).
$$ \hat{\beta} = r_{xy}\frac{s_{y}}{s_{x}} $$
$$ \hat{\alpha} = \bar{y} – \hat{\beta}\bar{x_i} $$
