Vtyjkyuk

I have the following equation which I wish to solve:

$$fracpartialpartial mathbf w(mathbf y - mathbf Xmathbf w)^T(mathbf y - mathbf X mathbf w) = 0$$

Here $mathbf y_n*1, mathbf X_n*2,mathbf w_2*1,$

My solution (done on paper because MathJax is a bit difficult for me to use):

Also, is my reasoning for step 4 correct?

Line $3$ to line $4$, note that
$$
fracpartialpartial w (y^TXw) = X^Ty,
$$
then you'll get the right answer
$$
hatw = (X^TX)^-1X^Ty.
$$

Explicit derivation:
Note that
$$
y^TXw = w_1sum_i=1^ny_i + w_2sum_i=1^ny_ix_1i+cdots+w_psum_i=1^ny_ix_pi,
$$
taking derivative w.r.t vector $w$, $w in mathbbR^p$, will result in a gradient, i.e., vector with $p$ rows and $1$ column, namely
$$
beginpmatrix
sum y_i \
sum y_i x_1i\
vdots \
sum y_i x_pi
endpmatrix,
$$
where the $j$th row is the derivative of $y^TXw$ w.r.t. $w_j$.
Now, as $X^T$ is $ptimes n$ and $y$ is $n times 1$, hence $X^Ty$ is $p times 1$ as required.

No your reasoning in step 4 is wrong. For example if $X$ is a square matrix, $mathbfX^T mathbfX$ will not be a scalar. Therefore your result is wrong. Do note that $$fracpartialpartial mathbfw left(mathbfw^T mathbfX^T mathbfX mathbfw right) = 2 mathbfX^T mathbfX mathbfw$$

I am sure that you can get to the right answer from here.

Let $M = mathbfy - mathbfX mathbfw$

and $f = M^T M = M : M$.

We will utilize the following the identities

• Trace and Frobenius product relation $$A:B=rm tr(A^TB)$$ or $$A^T:B=rm tr(AB)$$


• Cyclic property of Trace/Frobenius product $$eqalign
  A:BC
  &= AC^T:B cr
  &= B^TA:C cr
  &= text etc. cr
  $$

Now, we obtain the differential first and thereafter we obtain the gradient.

So,
beginalign
df &= left( d M: M right) + left( M : dM right)\
&= 2M : dM \
&= 2M : left( - mathbfX d mathbfw right) \
&= - 2mathbfX^T M : d mathbfw hspace8mm textnote: utilized cyclic property of Frobenius product \
&= - 2mathbfX^T left( mathbfy - mathbfX mathbfw right) : d mathbfw .
endalign

Thus, the gradient reads
beginalign
fracpartialpartial mathbfw f
= - 2mathbfX^T left( mathbfy - mathbfX mathbfw right) .
endalign

Then you can set the gradient to $0$ and obtain your $$mathbfw = left( mathbfX^T mathbfX right) ^-1 mathbfX^T mathbfy$$