Vtyjkyuk

Problem definition:

Show that image of $S$ under $A$, i.e., $A(S) = Ax : x in S $ is (a) convex and (b) but not closed in general if $S subseteq mathbbR^n$ is convex. Also, can we show when $A(S)$ is closed and when it is not?

However, if we consider the preimage $S$ under $A$, i.e., $A^-1(S) = x : Ax in S $, then it is both closed and convex.

EDIT: $S subseteq mathbbR^n$ is convex and $A in mathbbR^m times n$.

EDIT EDIT: Additionally, assume that $S$ is closed, i.e., $S$ is a closed and convex set.

(1) I suspect to your proof of convexity. You need to prove that$$textif quad v_1,v_2in A(S)quadtext then quad alpha v_1+(1-alpha )v_2in A(S)quad,quad0le alpha le1$$if $v_1,v_2in A(S)$ then there exist $x_1,x_2in S$ such that$$v_1=Ax_1\v_2=Ax_2$$also $S$ is convex so $alpha x_1+(1-alpha )x_2in S$ which means that $$A(alpha x_1+(1-alpha )x_2)in A(S)$$and $A$ is linear therefore$$A(alpha x_1+(1-alpha )x_2)=alpha Ax_1+(1-alpha)Ax_2=alpha v_1+(1-alpha)v_2in A(S)$$which completes the proof on convexity.
We prove the following theorem $$textA(S)text is closed if and only if Stext is closed.$$proof 1: let $S$ be closed. Then every convergent sequence $v_1,v_2,v_3,cdots in A(S)$ converges to some $v$ whether in $A(S)$ or not. By definition we have $$v_1=Ax_1\v_2=Ax_2\v_3=Ax_3\.\.\.$$since $v_i$s converge and $A$ is linear so the sequence $x_1,x_2,x_3,cdots$ is convergent to some $x$ and because S is closed hence $xin S$ which is equivalent to $Axin A(S)$. Therefore the sequence $v_1,v_2,cdots in A(S)$ converges to $v=Axin A(S)$ and $A(S)$ is closed.
proof 2: let $S$ is not closed and assume by contradiction that $A(S)$ is closed. Therefore there exists some convergent sequence $x_1,x_2,x_3cdots in S$ converging to some point outside of $S$. This sequence can be translated to $v_1=Ax_1,v_2=Ax_2,v_3=Ax_3,cdots in A(S)$ which is convergent to $vin A(S)$. Hence the definition $$exists xin Squad,quad v=Ax$$therefore the sequence $x_1,x_2,x_3cdots in S$ is convergent to $xin S$ which is contradiction. Therefore $A(S)$ is not closed and the proof is complete.
(2) The proof of convexity is simple, but I'm not sure if $A^-1(S)$ is always closed. Here is a counterexample. Take $A=I$ and $S$ any convex open set (such as inside the unit ball). Then $A^-1(S)=xin S=S$ which is convex but open.

(b) I assume that $S$ is a closed convex set; otherwise, an example is too trivial. I claim that, for a noninvertible nonzero linear transformation $A:mathbbR^ntomathbbR^m$, there exists a closed convex set $S$ such that $A(S)$ is not closed. Indeed, we have the following proposition.

Proposition. Let $A:mathbbR^ntomathbbR^m$ be a linear transformation. Then, for every closed set $S$, the set $A(S)$ is closed if and only if $A=0$ or $A$ is invertible. (The convexity of $S$ is unnecessary.)

If $A$ is noninvertible and nonzero, then there exists a nonzero vector $vinmathbbR^n$ such that $Av=0$. Since $A$ is nonzero, there exists a nonzero vector $uinmathbbR^n$ perpendicular to $v$ with respect to the usual Euclidean norm on $mathbbR^n$. Define $$S:=bigxu+yv,big,.$$ It can be easily seen that $S$ is a convex set. However, $A(S)=bigt,Au,$ obviously is not a closed set (since $0$ is a limit point of $A(S)$ that does not lie in $A(S)$).

If $A$ is zero, then $A(S)=emptyset$ or $A(S)=0$, which are closed sets. If $A$ is invertible, then $m=n$ and $A$ is a $mathcalC^infty$-diffeomorphism from $mathbbR^n$ to $mathbbR^m$, so $A$ sends closed sets to closed sets (and $A$ sends convex sets to convex sets due to Part (a)).

However, if you have a specific closed set $S$ and you want to determine whether $A(S)$ is also closed, then it is also possible to answer. First, let $N$ denote the nullspace of $A$. Then, $A:mathbbR^ntomathbbR^m$ factors through the canonical projection $pi:mathbbR^ntomathbbR^n/N$ via $alpha:mathbbR^n/NtomathbbR^m$ sending $v+Nmapsto Av$ for any $vinmathbbR^n$ (that is, $A=alphacircpi$). Note that $alpha$ is a $mathcalC^infty$-diffeomorpism from $mathbbR^n/N$ to its image. Therefore, it suffices to determine whether $pi(S)$ is a closed set.