fundamental statistics learning note (12)

Working…

Recall:
$$begin{equation}begin{split}
& hattheta_{MLE}overset{text{approx}}sim N(theta, frac{1}{I_n(theta)}) \
& i.e. Var(hat theta_{MLE})approx frac{1}{I_n(theta)} \
& hat Var(hat theta_{MLE}) = frac{-1}{l’’(hat theta_{MLE})}
end{split}end{equation}$$

Suppose $g(theta)$ is a function of the parameter, by MLE invaiance property, $g(theta)$ has MLE, $g(hat theta_{MLE})$.
$$begin{equation}begin{split}
& g(hattheta_{MLE})overset{text{approx}}sim N(g(theta), frac{[g’(theta)]^2}{I_n(theta)}) \
& Var(g(hattheta_{MLE}))approx frac{[g’(theta)]^2}{I_n(theta)} \
& hat Var(g(hattheta_{MLE}))= -frac{[g’(hat theta_{MLE})]^2}{l’’(hat theta_{MLE})}
end{split}end{equation}$$

Ex. $x_1,x_2,dots,x_n overset{text{iid}}sim Bern(p)$, recall MLE is $hat p = bar x$, what is MVUE of $p$?
Ans: It is $bar x$ since $E(bar x)=p$ and minimum sufficient statistics $sum_{i=1}^n x_i$.
In this case, we know exactly $$Var(hat p)= Var(bar x)=frac{p(1-p)}{n}$$
estimate by $$hat Var(hat p)= frac{hat p(1-hat p)}{n}$$
Odds is $frac{p}{1-p}$, the MLE of the odds is $frac{hat p}{1-hat p} (invariance, g(p)=frac{p}{1-p})$, however, $$ Var(frac{hat p}{1-hat p}) = Var(frac{bar x}{1-bar x})$$
is not known.

When n is large, $$hat {Var}(frac{hat p}{1-hat p})=-frac{[g’(hat p)]^2}{l’’(hat p)} $$
$$g’(p)=frac{(1-p)+p}{(1-p)^2}=frac{1}{(1-p)^2}$$
$$begin{equation}begin{split}
L(p|x_1,x_2,dots,x_n)&=prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} \
&=p^{sum x_i}(1-p)^{n-sum x_i}
end{split}end{equation}$$
$$l(p) = sum x_i logp+(n-sum x_i)log(1-p) $$
$$l’(p) = frac{sum x_i}{p}-frac{n-sum x_i}{1-p} $$
$$l’’(p)=-frac{sum x_i}{p^2} - frac{n-sum x_i}{(1-p)^2} = -frac{nhat p}{p^2}-frac{n-nhat p}{(1-p)^2} $$

$$begin{equation}begin{split}
l’’(hat p) &= -frac{nhat p}{hat p^2}-frac{n(1-hat p)}{(1- hat p)^2} \
&= -frac{n}{hat p}-frac{n}{1-hat p} \
&= -frac{-n}{hat p(1-hat p)}
end{split}end{equation}$$

$$Rightarrow hat {Var}(frac{hat p}{1-hat p})=frac{[frac{1}{(1-hat p)^2}]^2}{frac{n}{hat p(1-hat p)}}=frac{hat p}{n(1-hat p)^3}$$
This is approximate Var of the odds MLE when n is large.