Vtyjkyuk

My idea was to compute the derivative of the integrand, in order to find its maximum in $[0,1]$. It turns out it is attained at $t=sqrtfracalphaalpha+2k$. Then the claim follows from $$0<int_0^1 t^alpha(1-t^2)^kdtleleft(fracalphaalpha+2kright)^alpha/2left(1-fracalphaalpha+2kright)^kto0.$$

Is this correct? How else could one do it?

Your method seems fine to me.

Another way is as follow :

$$forall t in [0,1], 0 le t^alpha(1-t^2)^k le 1$$

But $1 in L^1([0,1])$. So by dominated convergence theorem :

$$lim_k to infty int_0^1 t^alpha(1-t^2)^kdt=int_0^1 lim_k to infty t^alpha(1-t^2)^kdt=int_0^10~dt=0$$

I suppose your $alpha $ is positive. Your proof is correct. If you know measure theory you can get it immediately from DCT: the integrand tends to $0$ because $1-t^2<1$ and the integrand is dominated by $t^alpha$ which is integrable.

Alternative method.

Solution. $blacktriangleleft$ Consider the upper limit. Compute as follows.

beginalign*
varlimsup_k left|int_0^1 t^alpha (1 - t^2)^kmathrm d tright| &leqslant varlimsup_k left(int_0^c + int_c^1right)t^alpha (1 - t^2)^k mathrm dt quad [text the function geqslant 0, cin (0,1)] \
&leqslant varlimsup_k int_0^c (1-t^2)^kmathrm dt + varlimsup_k int_c^1 (1- t^2)^k mathrm d t quad [t^alpha leqslant1]\
&leqslant int_0^c mathrm dt + varlimsup_k int_c^1 (1-c^2)^k mathrm dt quad [1 geqslant 1 -t^2 searrow 0]\
&=c + varlimsup_k(1-c)(1 -c^2)^k\
&= c + 0 quad[1-c^2<1, (1-c^2)^k to 0 text as k to infty]\
&= c to 0 quad [c to 0^+]. blacktriangleright
endalign*

Another way via Beta and Gamma functions $$int_0^1 t^alpha(1-t^2)^kdt= int_0^1 t^alpha/2+1/2-1(1-t)^k+1-1dt \=B(alpha/2+1/2,k+1)=fracGamma(alpha/2+1/2)Gamma(k+1)Gamma(alpha/2+k+3/2)to 0$$