Vtyjkyuk

Looking for the analytic solution of $x^2 - x ln x -k = 0$ for $x$. Have tried symbolic solver in MATLAB but couldn't find a closed-form expression. Here $0 < k < 1$.

You cannot find an analytical solution and you will need numerical methods.

Consider that you look for the zero(s) of function
$$f(x)=x^2 - x log(x) -k $$
$$f'(x)=2 x-log (x)-1$$
$$f''(x)=2-frac1x$$

The first derivative cancels at a point
$$x_*=-frac12 Wleft(-frac2eright)$$ where appears Lambert function. But this is not a real value since, in the real domain, this function is "defined" if $x geq -frac 1e$ and then the first derivative is always positive.

Since the function is bounded by $g(x)=x^2-k$, you could start Newton method with $x_0=sqrt k$ (except for $k=1$ for which $x=1$ is the solution) and generate iterates according to
$$x_n+1=x_n-fracx_n^2-x_n log (x_n)-k2 x_n-log (x_n)-1$$ and this would converge quite fast as shown below for $k=frac12$.
$$left(
beginarraycc
n & x_n \
0 & 0.7071067812 \
1 & 0.3849870727 \
2 & 0.3633137719 \
3 & 0.3635195741 \
4 & 0.3635195956
endarray
right)$$ You could generate a better estimate assuming that $x^2-xlog(x)approx x^a$ and get $a$ minimizing
$$Phi(a)=int_0^1 left(x^2-xlog(x)- x^a right)^2,dx=frac12 a+1-frac2(a+2)^2-frac2a+3+frac4311080$$
$$fracdPhi(a)da=-frac2(2 a+1)^2+frac4(a+2)^3+frac2(a+3)^2=0 implies aapprox 0.7implies x_0=k^10/7$$ Surprisingly, the optimal value of $a=0.692477$ is very close to $log(2)$.

For the worked example, this would give $x_0=0.3715$ and Newton method will converge quite fast.