So, we can represent the Cartesian vector $r= x\hat{x}+ y\hat{y}$ as the column vector where the entries are:
$$\begin{bmatrix}
x\\
y\\
\end{bmatrix} $$
How would the polar coordinates position vector :$r= r\hat{r}$ be represented as the entries of a column vector? Like this?
$$\begin{bmatrix}
r\\
0\\
\end{bmatrix} $$
I want to know this because I wish to discuss linear maps in different coordinates. For instance a rotation in Cartesian coordinates around the z axis is
$$r' = R\begin{bmatrix}
x\\
y\\
\end{bmatrix} $$
But the same operation in polar coordinates doesn't make much sense in this representation, since only the angle changes.
Best Answer
So polar coordinate $(r,\theta)$ is a parametrization of $\mathbb R^2$ i.e. $$g:\mathbb R_+\times[0,2\pi)\to\mathbb R^2,(r,\theta)\mapsto(r\cos\theta,r\sin\theta)$$ Definitely, you can represent the polar coordinate as a column vector. However, this is not a linear basis so you cannot represent a point by linear summation of the basis vectors like this $$ \hat r =x \hat x+y \hat y $$
As you said, a linear transform $T$ viewed in a nonlinear paramatrization is not linear, so it's not surprising that it cannot be represented as a matrix. $$ [x',y']^T=T[x,y]^T\\ [r',\theta']^T=g(Tg^{-1}([r,\theta]^T)) $$
Nevertheless, if you study differential geometry, even if the whole map is nonlinear, you could represent local infinitesimal changes (tangent vectors) by a linear transform (aka Jacobian, pushforward, differential map) $$ d\mathbb r'=dg\circ T\circ dg^{-1}d\mathbb r $$