To prove the above statement we need an additional statement.
Lemma. $h_{+}(A,B) = \sup\limits_{a \in A}\; \mathop{\mathrm{dist}}{(a,B)}$.
Proof.
$$
h_{+}(A,B) \leq \varepsilon \Leftrightarrow A \subseteq B+\mathbb{B}(\varepsilon,0) \Leftrightarrow \forall a \in A \; (a \in B+\mathbb{B}(\varepsilon,0)) \\
\Leftrightarrow \forall a \in A \; \exists b\in B \colon|b-a|\leq\varepsilon \Leftrightarrow \forall a \in A \; \mathop{\mathrm{dist}}{(a,B)} \leq \varepsilon \\
\Leftrightarrow \sup\limits_{a \in A} \; \mathop{\mathrm{dist}}(a,B) \leq \varepsilon.
$$
Since $h_{+}(A,B) \leq \varepsilon$ iff $\sup_{a \in A} \; \mathop{\mathrm{dist}}(a,B) \leq \varepsilon$ they are equal. $\blacksquare$
Hence we have an equality
$$
h(A,B) = \max \left\{ \sup\limits_{a \in A} \; \mathop{\mathrm{dist}}(a,B), \; \sup\limits_{b \in B} \; \mathop{\mathrm{dist}}(b,A) \right\}.
$$
Recall that convex conjugate function of $x \mapsto \mathop{\mathrm{dist}}(x,B)$ is a support function of compact convex set $B$, i.e.
$$
d(a,B) = \sup\limits_{\|l\| \leq 1} \left( \langle l, a \rangle - c(l \mid B) \right). \tag{1}
$$
Now we are ready to proove our main formula. We have
$$
\sup\limits_{a \in A} \;\mathop{\mathrm{dist}}(a,B) = \sup\limits_{a \in A} \sup\limits_{\|l\| \leq 1} ( \langle l, a \rangle - c(l \mid B) ) = \sup\limits_{\|l\| \leq 1} ( c(l \mid A) - c (l \mid B) ).
$$
We have changed the order of supremums in the latter equality. Now since
$$
\sup\limits_{\|l\| \leq 1} | c(l \mid A) -c (l \mid B) | = \max \{ \sup\limits_{\|l\| \leq 1} ( c(l \mid A) - c (l \mid B) ), \sup\limits_{\|l\| \leq 1} ( c(l \mid B) - c (l \mid A) ) \}
$$
we obtain the needed formula:
$$
h(A,B) = \sup\limits_{\|l\| \leq 1} | c(l \mid A) -c (l \mid B) |.
$$
Added. As concerns the proof of (1). Put $f(x) = \mathop{\mathrm{dist}} (x,B)$. Then
$$
f^*(l) = \sup_x \bigl( \langle l, x\rangle - f(x) \bigr) \\
= \sup_{b \in B} \sup_x \bigl( \langle l, x\rangle - \|x-b\| \bigr) \\
= \sup_{b \in B} \sup_{\alpha > 0} \sup_{\|x-b\|=\alpha} \bigl( \bigl( \langle l,x-b\rangle - \alpha \bigr) + \langle l, b \rangle \bigr) \\
= \sup_{b \in B} \sup_{\alpha > 0} \bigl( \alpha(\|l\|-1) + \langle l, b\rangle \bigr) \\
= \sup_{b \in B} \bigl( \langle l, b\rangle + \delta_1(\|l\|) \bigr) \\
= c(l|B) + \delta_1(\|l\|),
$$
where $\delta_1(t) = 0$ if $t\leq 1$ and $\delta_1(t) = +\infty$ otherwise. Hence,
$$
\mathop{\mathrm{dist}} d(x,B) = \sup_l \bigl( \langle l,x\rangle - c(l|B) - \delta_1(\|l\|) \bigr) \\
= \sup_{\|l\|\leq 1} \bigl( \langle l, x \rangle - c(l|B) \bigr).
$$
is there some kind of characterization of $h$ such that the domain $Q$ is convex
No, because the support function of $Q$ is equal to the support function of the convex hull of $Q$.
is there some kind of characterization of $h$ such that the domain $Q$ is strictly convex
Yes: if and only if $h$ is differentiable. In other words, if and only if the polar set of $Q$ has smooth boundary. It's worth noticing that the polar of polar set is the original set.
A corner in a convex set creates a line segment in the boundary of its polar, because the value of support function comes from the same point (the corner) for some interval of $\theta$. Conversely, suppose the boundary of a convex set contains a vertical segment to the right of origin. Then $h$ is not differentiable at $\theta=0$, because a small change of $\theta$ in either direction increases $h$ at a linear rate. (So, the graph of $h$ has a nonsmooth minimum like $|\theta|$.)
The general form of the above is known as duality of smoothness and rotundity, and it holds in all finite dimensional spaces.
Is there any characterization of $h$ such that the boundary $\partial Q$ admits nonvanishing curvature?
Yes: if and only if $h\in C^2$. I recommend Convex Bodies: The Brunn-Minkowski Theory by Rolf Schneider, specifically section 2.5 Higher regularity and curvature. For the two-dimensional case, see also page 2 of Lectures on Mean Curvature Flows by Xi-Ping Zhu.
Sketch. Introduce the Gauss map $\nu:\partial Q\to S^1$ which gives exterior unit normal vector at every boundary point. This map is defined when $Q$ has $C^1$ boundary. If $\partial Q$ is also strictly convex, then $\nu$ is injective. And if $\partial Q$ has nonvanishing curvature, then $\nu$ is a diffeomorphism -- indeed, this is if and only if, because the derivative of $\nu$ is the curvature of $\partial Q$.
It is convenient to define $h_Q$ as a function on $\mathbb R^2$, via $h_Q(x)=\sup_{q\in Q} x\cdot q$. Then one can check that $ h_Q(x)= x\cdot \nu^{-1}(x/|x|) $ and
$ \nabla h_Q(x)= \nu^{-1}(x/|x|)$.
Thus, the equivalence is: nonvanishing curvature $\iff$ $\nu $ is a diffeomorphism $\iff$ $h_Q\in C^2(\mathbb R^2\setminus \{0\})$.
Best Answer
The main problem you have is giving the same name to different things: $\theta$ means two different things, and so does $a$.
The parametric equation $x=a\cos t$, $y=b\sin t$, leads to $$ h_E(\theta)=\sup_t (a\cos t,b\sin t)\cdot (\cos\theta, \sin\theta) $$ Observe that the dot product can just as well be written as $$ (\cos t,\sin t)\cdot (a\cos\theta, b\sin\theta) $$ which is simply the projection of $(a\cos\theta, b\sin\theta)$ onto the direction determined by $t$. The maximal possible value of scalar projection is the length of the vector, hence $$ h_E(\theta)=|(a\cos\theta, b\sin\theta)| = \sqrt{a^2\cos^2\theta+b^2\sin^2\theta} $$