Injectivity
$F(x)=F(y)$, with $x\ne0$ and $y\ne0$ means
$$
\tan\biggl(\frac{\pi \|x\|}{2}\biggr)\frac{x}{\|x\|}=
\tan\biggl(\frac{\pi \|y\|}{2}\biggr)\frac{y}{\|y\|}
$$
So we can assume $x\ne0$ and $y\ne0$. Then, taking norms,
$$
\tan\biggl(\frac{\pi \|x\|}{2}\biggr)=\tan\biggl(\frac{\pi \|y\|}{2}\biggr)
$$
because $x/\|x\|$ has norm $1$ and the same for $y/\|y\|$; moreover $\tan t>0$ if $t>0$. By the injectivity of the tangent you get
$$
\frac{\pi \|x\|}{2}=\frac{\pi \|y\|}{2}
$$
so $\|x\|=\|y\|$ and, finally $x=y$.
It's clear that $F(x)=0$ if and only if $x=0$, so the proof is complete.
Surjectivity
$0=F(0)$, so we only need to show that, for $z\ne0$, we can find $x$ with $F(x)=z$. We should have
$$
z=\tan\biggl(\frac{\pi \|x\|}{2}\biggr)\frac{x}{\|x\|}
$$
so
$$
\|z\|=\tan\biggl(\frac{\pi \|x\|}{2}\biggr)
$$
hence
$$
\frac{\pi\|x\|}{2}=\arctan\|z\|
$$
that is,
$$
\|x\|=\frac{2\arctan\|z\|}{\pi}.
$$
Thus the candidate is
$$
x=\frac{2\arctan\|z\|}{\pi\|z\|}z
$$
Verify it's the right one.
Continuity
The continuity of $F$ is obvious in the points different from $0$, because it's obtained by continuous functions. Since your domain consists of vectors with norm $<1$, there's no problem with the tangent function, because you consider only arguments in $(0,\pi/2)$. The continuity of the inverse is obvious as well
outside $0$
Are $F$ and its inverse continuous at $0$?
A well known fact is that $\lim_{x\to0}F(x)=0$ if and only if $\lim_{x\to0}\|F(x)\|=0$; now
$$
\|F(x)\|=\biggl\|\tan\biggl(\frac{\pi\|x\|}{2}\biggr)\frac{x}{\|x\|}\biggr\|
=\tan\biggl(\frac{\pi\|x\|}{2}\biggr)
$$
that obviously satisfies the requested property. The same for the continuity at $0$ of $F^{-1}$.
I don't know why I didn't say the following at the outset. I guess I was just going along with your method.
Rid your proof of trigonometric functions and instead do the following:
$$
t\mapsto (x,y) = \begin{cases} \phantom{\lim\limits_{t\to\infty}} \left( \dfrac{1-t^2}{1+t^2}, \dfrac{2t}{1+t^2} \right) & \text{if }t\ne\infty, \\[10pt]
\lim\limits_{t\to\infty} \left( \dfrac{1-t^2}{1+t^2}, \dfrac{2t}{1+t^2} \right) & \text{if }t=\infty.
\end{cases}
$$
That's a homeomorphism from $\mathbb R\cup\{\infty\}$ to $\{(x,y)\in\mathbb R^2:x^2+y^2=1\}$.
To show that it's surjective, show that $t=\dfrac{y}{x+1}$ (and notice that $\dfrac{y}{x+1}\to\infty$ as $(x,y)\to(-1,0)$ along the curve $x^2+y^2=1$).
Best Answer
You do not mention whether you consider the closed or open unit disk resp. unit square. Here, let us consider the closed unit disk $D$ and the closed unit square $Q$.
Both receive their topologies as a subspaces of $\mathbb{R}^2$ which carries its standard topology. There are various equivalent descriptions of this topology. It is
1) the product topology on $\mathbb{R} \times \mathbb{R}$, where $\mathbb{R}$ has its standard topology.
2) the topology generated by the Euclidean norm $\lVert (x,y) \rVert_2 = \sqrt{x^2+y^2}$.
3) the topology generated by the maximum norm $\lVert (x,y) \rVert_\infty = \max(\lvert x \rvert, \lvert y \rvert)$.
Let us modify your definition of $f : \mathbb{R}^2 \to \mathbb{R}^2$ as follows: $$f(z) = \begin{cases} \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}z &\quad\text{if } z \ne 0 \\ \text{} 0 &\quad\text{if } z = 0 \\ \end{cases}$$ Your definition was $\frac{\lVert z \rVert_2^2}{\lVert z \rVert_\infty}z$ in the first line, but the present one is easier. To see that $f$ is continuous, let us use the maximum norm.
Continuity in $0$: $\lVert f(z) - f(0) \rVert_\infty = \lVert f(z)\rVert_\infty = \lVert z \rVert_2 \to 0$ as $z \to 0$.
Continuity in $w \ne 0$: $\lVert f(z) - f(w) \rVert_\infty = \left\lVert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}z - \frac{\lVert w \rVert_2}{\lVert w \rVert_\infty}w \right\rVert_\infty = \left\lVert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}(z-w) + (\frac{\lVert z \rVert_2}{\lVert z \rVert_\infty} - \frac{\lVert w \rVert_2}{\lVert w \rVert_\infty})w \right\rVert_\infty \le $ $\frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}\lVert z - w \rVert_\infty + \left\lvert \frac{\lVert z \rVert_2}{\lVert z \rVert_\infty}\lVert w \rVert_\infty - \lVert w \rVert_2 \right\rvert \to 0$ as $z \to w$.
Moreover, $f(D) \subset Q$: We have $z \in D$ iff $\lVert z \rVert_2 \le 1$, thus $\lVert f(z )\rVert_\infty = \lVert z \rVert_2 \le 1$ which means $f(z) \in Q$.
Now define $g : \mathbb{R}^2 \to \mathbb{R}^2$ as follows: $$g(z) = \begin{cases} \frac{\lVert z \rVert_\infty}{\lVert z \rVert_2}z &\quad\text{if } z \ne 0 \\ \text{} 0 &\quad\text{if } z = 0 \\ \end{cases}$$ We can easily verify that $g$ is continuous and $g(Q) \subset D$. A final computation shows that $g \circ f = id$ and $f \circ g = id$.