Ytukyg

In a book The Elements of
Statistical Learning published by Springer we can find following statement:

We can write

$RSS(beta) = (mathbf{y}-mathbf{X}beta)^T(mathbf{y}-mathbf{X}beta)$

where $mathbf{X}$ is an $Ntimes p$ matrix with each row an input vector, and $mathbf{y}$ is an $N$-vector of the outputs in the training set. Differentiating w.r.t. $beta$ we get the normal equations

$mathbf{X}^T(mathbf{y}-mathbf{X}beta) = 0$

Questions

How do I formally derive the normal equations operating on matrix level calculations without diving into operating on scalar elements?

Is my Second attemp valid?

First attemp

Note: $beta$ is an $p$-vector. Let us assume that vectors are vertical matrixes.

As in The Matrix Cookbook (http://www.math.uwaterloo.ca/~hwolkowi//matrixcookbook.pdf) let us assume that
$partialmathbf{X}^T = (partialmathbf{X})^T$
and
$partial(mathbf{XY})=partial(mathbf{X})mathbf{Y}+mathbf{X}partial(mathbf{Y})$.

Let us differentiate with respect to $beta$ and observe that $partial (mathbf{y}-mathbf{X}beta)=-mathbf{X}$.

Now $partial RSS(beta)=(partial (mathbf{y}-mathbf{X}beta))^T(mathbf{y}-mathbf{X}beta)+(mathbf{y}-mathbf{X}beta)^T partial (mathbf{y}-mathbf{X}beta)$

Which gives us $partial RSS(beta)=-mathbf{X}^T (mathbf{y}-mathbf{X}beta)-(mathbf{X}^T (mathbf{y}-mathbf{X}beta))^T$

At this point we find contradiction because dimensions are incompatible to perform summation. $mathbf{X}^T (mathbf{y}-mathbf{X}beta)$ is vertical $p$-vector, while $(mathbf{X}^T (mathbf{y}-mathbf{X}beta))^T$ is horizontal $p$-vector.

Second attemp

If I assumed $partial(mathbf{X}^Tmathbf{Y})=partial(mathbf{X})^Tmathbf{Y}+(mathbf{X}^Tpartial(mathbf{Y}))^T$

I would get that
$partial RSS(beta)=-2mathbf{X}^T (mathbf{y}-mathbf{X}beta)$, which matches with the normal equations from the book.

If you have not found an example, then here it goes.

Before we start deriving the gradient, some facts and notations for brevity:

• Trace and Frobenius product relation $$leftlangle A, B Crightrangle={rm tr}(A^TBC) := A : B C$$


• Cyclic properties of Trace/Frobenius product
  begin{align}
  A : B C
  &= BC : A \
  &= B^T A : C \
  &= {text{etc.}} cr
  end{align}
  

Let $f := left|y- Xbeta right|^2 = left(y- Xbeta right)^T left(y- Xbeta right) = y- Xbeta:y- Xbeta$.

Now, we can obtain the differential first, and then the gradient.
begin{align}
df
&= dleft( y- Xbeta:y- Xbeta right) \
&= 2left(y- Xbeta right) : -X dbeta \
&= -2X^Tleft(y- Xbetaright) : dbeta\
end{align}

Thus, the gradient is
begin{align}
frac{partial}{partial beta} left( left|y - X beta right|^2 right)= -2X^Tleft(y- Xbetaright).
end{align}