So what you have already shown is that
- $f \colon {\mathbb R}/{\sim} \to S^1$ is continuous;
- $f$ is bijective.
What is left is to show that $f^{-1}$ is continuous as well, or, equivalently in this case, that $f$ is open or closed.
You could start monkeying around with open subsets of ${\mathbb R}/{\sim}$ and trying to prove that their image is open. This is bound to involve lot of handwaving, not because it is particularly difficult, but because it is extremely tricky to write down properly. (Note, by the way, that typical $\epsilon$-$\delta$-computations involving the topology on ${\mathbb R}$ and ${\mathbb R}^2$ for the continuity of $f$ are hidden in the continuity of $\sin$ and $\cos$).
So, I'll resort to trickery. The one-shot solution to this particular problem is the following theorem.
Theorem. Every continuous map from a compact space to a Hausdorff space is closed.
This settles it: ${\mathbb R}/{\sim}$ is compact, because it can also be seen as the quotient $[0,1]/{\sim}$ and a quotient of a compact space is compact; and $S^1$ is Hausdorff. So $f$ is continuous, bijective, and closed, and therefore a homeomorphism.
Now maybe you don't like this, for instance because you haven't seen this theorem yet. So, here's another approach. I'll still try to avoid having to do $\epsilon$-$\delta$-computations; this time I'll hide them in the continuity of $\arccos \colon (-1,1) \to (0, \pi)$.
The map $(x,y) \mapsto \arccos(x)/(2\pi)$ is a continuous map from $\{(x,y) \in {\mathbb R}^2 \mid y > 0 \}$ to $(0, \frac{1}{2})$. Restricting to $\{(x,y) \in S^1 \mid y > 0\}$ and composing with the projection ${\mathbb R} \to {\mathbb R}/{\sim}$ gives a continuous map that is exactly the restriction of $f^{-1}$ to $\{(x,y) \in S^1 \mid y > 0\}$.
Similary, the map $(x,y) \mapsto \arccos(y)/(2\pi) + \frac{1}{4}$ is a continuous map from $\{(x,y) \in {\mathbb R}^2 \mid x < 0\}$ to $(\frac{1}{4},\frac{3}{4})$. Restricting this one to $\{(x,y) \in S^1 \mid x < 0\}$ and composing with the projection ${\mathbb R} \to {\mathbb R}/{\sim}$ gives a continuous map that is exactly the restriction of $f^{-1}$ to $\{(x,y) \in S^1 \mid x < 0\}$.
Now do this twice more to find that the restriction of $f^{-1}$ to each of those four open subsets of $S^1$ (with $y > 0$, $x < 0$, $y < 0$, $x > 0$) is continuous. Because those four open subsets cover $S^1$, $f^{-1}$ is continuous.
Best Answer
Hint You should be able to use the fact that if you remove any point from $S^1$, the remaining space is connected. The same is not true for $[0,1)$.
If you want to use the inverse, note that $$\lim_{t \to 0^+} f(t)=\lim_{t \to 1^-}f(t)=f(0)=(1,0) \,. $$
Can you deduce from here that $f^{-1}$ cannot be continuous at $(1,0)$?