[notes on mathematics for esl] chapter 5: basis expansions and regularization

5.4 Smoothing Splines

Derivation of Equation (5.12)

Equal the derivative of Equation (5.11) as zero, we get

$$frac{partialtext{RSS}(theta,lambda)}{partialtheta} = -2N^T(y-Ntheta)+2lambdaOmega_Ntheta = 0$$

Put the terms related to $theta$ on one side and the others on the other side, we get

$$(N^TN+lambdaOmega_N)theta = N^Ty$$

Multiply the inverse of $N^TN+lambdaOmega_N$ on both sides completes the derivation of Equation (5.12)

$$hattheta = (N^TN+lambdaOmega_N)^{-1}N^Ty$$

Explanations on Equation (5.17) and (5.18)

It’s a little confusing to get Equation (5.18) directly from Equation (5.17) and its original form Equation (5.11). In order to give a clear explanation, here we give the proof of the equation,

$$theta^TOmega_Ntheta=mathbf{f}^TKmathbf{f}.$$

which are the different terms between Equation (5.11) and Equation (5.18).

We know that

$$% <![CDATA[ begin{split} mathbf{f}&=N(N^TN+lambdaOmega_N)^{-1}N^Ty \ &=S_{lambda}y = Ntheta end{split} %]]>$$

following Equation (5.14) in the book.

From Equation (5.17), we can get

$$% <![CDATA[ begin{split} K&=frac1lambda(S_lambda^{-1}-I)\ &=frac1lambdaleft(N^{-T}(N^TN+lambdaOmega_N)N^{-1}-Iright)\ &=N^{-T}Omega_NN^{-1} end{split} %]]>$$

Plug the above two equation into the right side of the equation remains to be proved, we get

$$% <![CDATA[ begin{split} mathbf{f}^TKmathbf{f} &= theta^TN^TN^{-T}Omega_NN^{-1}Ntheta \ &=theta^TOmega_Ntheta end{split} %]]>$$

which completes the proof.