Deriving Regression coefficient from Residual Sum of Squares

When Linear Regression equation, $y=\hat{\alpha} + \hat{\beta} x$, is found from the data, the formula for their coefficients, $\hat{\alpha}$ and $\hat{\beta}$, are the followings.

$$\hat{\beta} = r_{xy}\frac{s_{y}}{s_{x}}$$

$$\hat{\alpha} = \bar{y} – \hat{\beta}\bar{x_i}$$

$r_{xy}$: correlation coefficient

$s_{x}$: standard deviation of $x$, $s_{y}$: standard deviation of $y$

Here we will explain how to derive these equation from the residual sum of squares.

Derivation

We will use the following residual sum of regression below.

$$S(\hat{\alpha}, \hat{\beta}) = \sum^{n}_{i=1}(y_i – \hat{y})^2 = \sum^{n}_{i=1}(y_i – (\hat{\alpha} + \hat{\beta} x_i ))^2$$

To find $\hat{\alpha}$ and $\hat{\beta}$ to minimize $S(\hat{\alpha}, \hat{\beta})$, we will differentiate partially with $\hat{\alpha}, \hat{\beta}$.

$$\frac{\partial S}{\partial \hat{\alpha}} = 2\times(-1)\times\sum^{n}_{i=1}(y_i – \hat{\alpha} – \hat{\beta} x_i ) =0 \\ \sum^{n}_{i=1}y_i = \sum^{n}_{i=1}\hat{\alpha} + \sum^{n}_{i=1}\hat{\beta}x_i \\ = n\hat{\alpha} + \hat{\beta}\sum^{n}_{i=1}x_i$$

Dividing both side by $n$

$$\frac{\sum^{n}_{i=1}y_i}{n} = \hat{\alpha} + \frac{\sum^{n}_{i=1}x_i}{n} \ \bar{y} = \hat{\alpha} + \hat{\beta}\bar{x} \\ \hat{\alpha} = \bar{y} – \hat{\beta}\bar{x}$$

we can get $\hat{\alpha}$.

Now to get $\hat{\beta}$, we will differentiate $S$ partially by $\hat{\beta}$.

$$\frac{\partial S}{\partial \hat{\beta}} = 2\times(-1)\times\sum^{n}_{i=1}(y_i – \hat{\alpha} – \hat{\beta} x_i ) x_i =0 \\ \sum^{n}_{i=1}x_i y_i = \sum^{n}_{i=1}\hat{\alpha} x_i + \sum^{n}_{i=1}\hat{\beta} x_i^2 \ \sum^{n}_{i=1}x_i y_i= \hat{\alpha}\sum^{n}_{i=1}x_i + \hat{\beta}\sum^{n}_{i=1}x_i^2$$

Dividing both sides by $n$

$$\frac{\sum^{n}_{i=1}x_iy_i}{n} = \hat{\alpha}\frac{\sum^{n}_{i=1}x_i}{n} + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}} \\ \frac{\sum^{n}_{i=1}x_iy_i}{n} = \hat{\alpha}\bar{x} + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}}$$

Here we can use $\hat{\alpha} = \bar{y} – \hat{\beta}\bar{x}$ that we already got.

$$\frac{\sum^{n}_{i=1}x_iy_i}{n} = \bar{x}\bar{y} – \hat{\beta}\bar{x}^2 + \hat{\beta}{\frac{\sum^{n}_{i=1}x_i^2}{n}} \\ = \bar{x}\bar{y} + \hat{\beta}(\frac{\sum^{n}_{i=1}x_i^2}{n}-\bar{x}^2)$$

$\frac{\sum^{n}_{i=1}x_i^2}{n}-\bar{x}^2$ is variance of $x$ then we define it as $s_{xx}$.

$$\frac{\sum^{n}_{i=1}x_iy_i}{n} = \bar{x}\bar{y} + \hat{\beta}s_{xx} \\ \frac{\sum^{n}_{i=1}x_iy_i }{n}- \bar{x}\bar{y} = \hat{\beta}s_{xx}$$

Here we can change $\frac{\sum^{n}_{i=1}x_iy_i}{n} – \bar{x}\bar{y}$ to the following.

$$\frac{\sum^{n}_{i=1}x_iy_i}{n} + \bar{x}\bar{y} – 2\bar{x}\bar{y} \\ = \frac{\sum^{n}_{i=1}x_iy_i}{n} + \frac{\sum^{n}_{i=1}\bar{x}\bar{y}}{n} – \frac{\sum^{n}_{i=1}x_i}{n}\bar{y} – \frac{\sum^{n}_{i=1}y_i}{n}\bar{x} = \frac{\sum^{n}_{i=1}(x_iy_i – x_i\bar{y} – \bar{x}y_i + \bar{x}\bar{y})}{n}$$

$\frac{\sum^{n}_{i=1}x_iy_i}{n}- \bar{x}\bar{y}$ is covariance of $x, y$ then we define it as $s_{xy}$.

$$s_{xy} = \hat{\beta}s_{xx} \\ \hat{\beta} = \frac{s_{xy}}{s_{xx}} = \frac{s_{xy}}{s_xs_y}\cdot \frac{s_y}{s_x}$$

We express correlation coefficient as $r_{xy} = \frac{s_{xy}}{s_{x}s_{y}}$

$$\hat{\beta} = r_{xy}\frac{s_y}{s_x}$$

Finally we can get $\hat{\alpha}$ and $\hat{\beta}$.

$$\hat{\beta} = r_{xy}\frac{s_{y}}{s_{x}}$$

$$\hat{\alpha} = \bar{y} – \hat{\beta}\bar{x_i}$$