fundamental statistics learning note(23)

Linear Regression

This chapter will talk about using MLE to estimate the parameters in linear regression. Though the most common way nowdays is to use least square. I think it can bridge the gap between probability and regression theoretically.

Suppose we want to construct a regression model to simulate the relationship between the weight of moose and the size of its anteler.
Model: $y_isim N(beta_0+beta_1x_i,sigma^2)$, where $y_i$ is the weight of moose and $x_i$ is the size of the anteler. Because $y_i$ is dependent on $x_i$, so $y_i$ is indepedent.
First we get likelihood function:
$$begin{equation}begin{split}
L(beta_0,beta_1,sigma^2|y_1,dots,y_n) & = prodlimits_{i=1}^n frac{1}{sigmasqrt{2pi}}e^{-frac{y_i-(beta_0+beta_1x_i)^2}{2sigma^2}} \
& = frac{1}{sigma^n (2pi)^{frac{n}{2}}}e^{-frac{sum(y_i-beta_0-beta_1x_i)^2}{2sigma^2}}
end{split}end{equation}$$
Then get log-likelihood function
$$begin{equation}begin{split}
l(beta_0,beta_1,sigma^2) & = -frac{n}{2}log(sigma^2)-frac{n}{2}log(2pi)-frac{sum(y_i-beta_0-beta_1x_i)^2}{2sigma^2}
end{split}end{equation}$$
Take partial derivative and set partitals to 0
$$begin{equation}begin{split}
frac{partial l}{partial beta_0} & = sum 2(y_i-beta_0-beta_1x_i)(-1)overset{text{set}}=0 \
& Rightarrow sum(y_i-beta_0-beta_1x_i)=0 \
& Rightarrow sum y_i-nbeta_0-beta_1sum x_i = 0 \
& Rightarrow nbeta_0 = sum y_i - beta_1sum x_i \
& Rightarrow beta_0 = bar y - beta_1 bar x
end{split}end{equation}$$
which is intercept.
$$begin{equation}begin{split}
frac{partial l}{partial beta_1} & = sum 2(y_i-bar y + beta_1bar x+beta_1x_i)(-1) \
& = -frac{1}{2sigma^2}[sum2lbrace (y_1-bar y)-beta_1(x_i-bar x)rbrace(x_i-bar x)(-1)] overset{text{set}}=0 \
& Rightarrow sum[(y_i-bar y)(x_i-bar x)-beta_1sum(x_i-bar x)^2=0 \
& Rightarrow hat beta_1 = frac{sum(y_i-bar y)(x_i-bar x)}{sum(x_i-bar x)^2}
end{split}end{equation}$$
which is slope.