introduction to linear regression

This post briefly tells the general idea of linear regression. Examples are also included.

This post will cover:

• Fundamental Knowledge before Get Started
• Simple Linear Regression

Fundamental Knowledge

Before get started, recall some basic things in statistic:
$newcommand{uE}{mathop{}negthinspacemathrm{E}}
newcommand{ucov}{mathop{}negthinspacemathrm{Cov}}
newcommand{uvar}{mathop{}negthinspacemathrm{Var}}$

Expected Value

Properties:

$$uE(a+bX) = a+buE(X)$$

Variance and Covariance

Definition of variance:

$$uvar(Y) = uEBig[(Y-uE(Y))^2Big] = uE(Y^2) - Big[uE(Y)Big]^2$$

Definition of covariance:

$$ucov(Y,Z) = uEBig[(Y-uE(Y))cdot(Z-uE(Z))Big]$$

Properties:

$$uvar(a+cY)=c^2(uvar(Y))$$
$$ucov(a+bY, c+dZ)=bducov(Y,Z)$$

And there’s a significant property when calculate the variance of summation of random variables:

$$uvar(sum_{i=1}^n a_iY_i) = sum_{i=1}^nsum_{i=j}^n a_ia_jucov(Y_i,Y_j),$$

which also has a special form when $Y_i$ are mutually independent:

$$uvar(sum_{i=1}^n a_iY_i) = sum_{i=1}^n a_i^2uvar(Y_i)$$

Coefficient of Correlation

In case that we might use this later, the definition of coefficient of correlation is introduced here:

$$rho(Y,Z) = frac{ucov(Y,Z)}{sqrt{uvar(Y)uvar(Z)}} in [-1,1]$$

When $Y$ and $Z$ are independent, we know $ucov(Y,Z)=0, rho(Y,Z)=0$.