As you said "a circle, with the distance between two points measured along the shortest arc connecting them, is a (complete) convex metric space." But its not a convex metric space with the norm induced by Euclidean norm. So the statement "if a closed subset of a Euclidean space together with the induced distance is a convex metric space, then it is a convex set" is not satisfying.
Maybe the following can at least partly answer your question. First we look at
topological vector spaces in increasing generality:
\begin{align*}
&\text{finite dimensional vector spaces - Hilbert Spaces - Banach Spaces}\\
&\text{Frechet Spaces - locally convex vector spaces}\\
\end{align*}
as Jänich does in his Topology. There he mentions an example from Dieudonnes Treatise on Analysis Volume II of a locally convex, but not metrizable and so not pre-Frechet topologial Vector Space.
He states, that these non-metrizable spaces occur naturally in functional analysis for example when we want to find on a given topological Vector Space $E$ the weak topology, i.e. the coarsest topology, so that all continous, linear mappings are continous. He further states, that if $E$ is an infinite dimensional Hilbert Space, then $E$ equipped with the weak topology is already a locally convex Hausdorff Space, but not metrizable.
And now some deeper information about this natural wish to develop a theory around locally convex vector spaces.
Here's an extract from Bourbakis Elements of the History of Mathematics. He writes in the end of chapter 21: Topological Vector Spaces:
Excerpt from Bourbakis Elements of the History of Mathematics ch. 21:
It had on the other hand been observed, before 1930, that notions such as simple convergence, convergence in measure for measurable functions, or compact convergence for entire functions, are not capable of being defined by means of a norm; and in 1926, Fréchet had noted, that vector spaces of this nature can be metrizable and complete.
But the theory of these more general spaces was only to develop in a fruitful way in combination with the idea of convexity. This latter (that we saw appearing with Helly) was an object of study for Banach and his pupils, who recognised the possibility of interpreting thus in a more geometric way numerous statements of the theory of normed spaces, preparing the way for the general definition of locally convex spaces, given by J. von Neumann in 1935. $\ldots$
Finally and especially, it is certain that the main impulsion that motivated this research came from new possibilities for applications to Analysis, in domains where Banach Theory was inoperative: the theory of sequence spaces must be mentioned in this context, developed by Köthe, Toeplitz and their pupils since 1934 in series of memoirs, the recent setting up of the theory of analytical functionals of Fantappié, and above all the theory of distributions of L. Schwartz, where the modern theory of locally convex spaces found a field of applications that is without doubt a long way from being exhausted.
Best Answer
I think that your approach should work.
Let $C\subseteq \mathbb{R}^n$ be a closed, convex set. Let $\mathcal{H}$ be the collection of closed, half-spaces that contain $C$. You would like to show that $$C = \bigcap_{H\in \mathcal{H}}H.$$
First we can show $C \subseteq \bigcap_{H\in \mathcal{H}}H.$ Let $x\in C$. By the definition of $\mathcal{H}$, any $H\in \mathcal{H}$ satisfies $C\subseteq H$. Hence $x\in H$ for any $H\in \mathcal{H}$ and therefore $x\in \bigcap_{H\in \mathcal{H}}H.$ This gives us the desired inclusion.
It is left to show that $C \supseteq \bigcap_{H\in \mathcal{H}}H.$ We prove this using the contrapositive, that is we will show that if $x\not\in C$ then $x\not\in \bigcap_{H\in \mathcal{H}}H.$ So choose $x$ such that $x\not\in C$. Since $C$ is closed and convex, there is a hyperplane that strictly separates $x$ from $C$. This hyperplane defines a half space $H$ containing $C$. Hence $x\not\in H$ implying that $x\not\in \bigcap_{H\in \mathcal{H}}H$. This proves the desired inclusion.