Vtyjkyuk

Consider the two variables $x_1,x_2$ which are functions of time $t$, we refer $dotx_1$ as the time derivative of $x_1$.

Let $alpha$ be a constant parameter, and consider the following system of equations:

beginalign*
dotx_1 &= alpha x_1 - x_2 - x_1(x_1^2 + x_2^2)\
dotx_2 &= x_1 + alpha x_2 - x_2(x_1^2 + x_2^2).
endalign*

Now we transform the above system to polar $(rho,theta)$ coordinates. I think after the transformation the above equations lead to the following:

beginalign*
dotrho &= rho(alpha - rho^2)\
dottheta &= 1.
endalign*

Where $x_1 = rho cos(theta)$ and $x_2 = rho sin(theta)$. If I substitute then I am thinking about the partial derivatives involved and I think which variable I should differentiate with respect to the time?
I can observe that the $x_1^2 +x_2^2 = 1$ which could simplify a lot but how should I deal with $dotx_1, dotx_2$ which can help me in getting the polar form of the above system?

I don't understand your last paragraph (notice for instance $x_1^2+x_2^2 = rho^2$, not $1$) or how you got your second set of equations. To transform your differential equations, just apply the product and chain rules: for instance the first equation becomes
$$dotrho cos(theta) - rho sin(theta)dottheta = alpha rho cos(theta) - rho sin(theta) - rho^3 cos(theta),$$
and you will get something similar for the second equation. You can then apply standard variable elimination techniques to isolate equations for $dotrho$ and $dottheta$ and try to simplify them. Give it a try and let me know if you need more help.