You only need to consider the case $\mathfrak{h}_{s}^\ast(A) \lt \infty$, but you need to be a bit careful in choosing the outer approximations since swapping $\inf$ and $\sup$ certainly isn't allowed without some thinking. If you knew that you can always take the same set $E$ in the $\inf$ (which I will show in \eqref{eqn:ast} below) then you'd be essentially done, but it is impossible to say if you got so far or not.
I'm going to ignore $\alpha_s$ since it only is a scalar factor that doesn't play a rôle in the argument and the formulas will already get cumbersome enough without it.
The following argument is essentially the one from Fremlin's measure theory, Volume 4II, 471D, page 102. $\DeclareMathOperator{\diam}{diam}$ $\newcommand{\hsd}[1]{\mathfrak{h}_{s,#1}^\ast}$
Set $\delta_n = 2^{-n}$ and choose $(A_{i}^n)_{i \in \mathbb{N}}$ of diameter $\leq 2^{-n}$ such that $A \subset \bigcup_{i=1}^\infty A_{i}^n$ and that
\begin{equation}\tag{$1$}\label{eqn:eq1}
\sum_{i=1}^\infty (\diam{A_{i}^n})^s \leq \hsd{2^{-n}}(A) +2^{-n},
\end{equation}
which is possible because of the definition of $\hsd{2^{-n}}$ as infimum.
Observe that there's no reason for the $A_{i}^n$ to be $\hsd{2^{-n}}$-measurable, let alone Borel, so we would like to “blow them up” slightly, so as to get open sets still approximating the $\hsd{2^{-n}}$-measure of $A$ well. The problem is that by doing so we will lose the diameter condition which appears in the definition of $\hsd{2^{-n}}$, but this isn't a serious problem: we can simply choose a larger $n$ and work with the sets we obtain from there. Here are the gory details:
Choose $0 \lt r_{i}^n \lt 2^{-n}$ so small that
$$
(\diam{(A_{i}^n)} + 2r_{i}^n)^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
$$
Now put $U_{i}^n = \{x \in X\,:\,d(x,A_{i}^n) \lt r_{i}^n\}$ and note that $U_{i}^n \supset A_{i}^n$ is an open set whose diameter satisfies
\begin{equation}\tag{$2$}\label{eqn:eq2}
(\diam{U_{i}^n})^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
\end{equation}
Let
$$
B = \bigcap_{n=1}^\infty \bigcup_{i=1}^\infty U_{i}^n
$$
and observe that $B$ is a $G_{\delta}$-set (countable intersection of open sets) containing $A$.
Given any $\delta \gt 0$, we can find $N$ such that $3 \cdot 2^{-N} \leq \delta$ so that for all $n \geq N$ we have $\diam{U_{i}^n} \leq \delta$. As $B \subset \bigcup_{i = 1}^\infty U_{i}^n$ we see from \eqref{eqn:eq1} and \eqref{eqn:eq2} that for $n \geq N$
$$
\hsd{\delta}(B) \leq \sum_{i=1}^\infty (\diam{U_{i}^n})^s
\leq
\sum_{i=1}^\infty [(\diam{A_{i}^n})^s + 2^{-n-i}] \leq \hsd{2^{-n}}{(A)} + 2 \cdot 2^{-n}
$$
Since $\hsd{2^{-n}}(A) \leq \mathfrak{h}_{s}^\ast(A)$ we get for $n \geq N$
$$
\hsd{\delta}(B) \leq
\mathfrak{h}_{s}^\ast(A) + 2^{-n}
$$
and letting $n \to \infty$ this gives
\begin{equation}\tag{$\ast$}\label{eqn:ast}
\hsd{\delta}(B) \leq \mathfrak{h}_{s}^\ast(A)
\end{equation}
for every $\delta \gt 0$. [Note: this is stronger than your condition involving $\inf$ since we can specify the set $E = B$ and thus avoid the infimum]
Taking the $\sup$ over all $\delta$ in \eqref{eqn:ast} this yields
$$
\mathfrak{h}_{s}^\ast(B) \leq \mathfrak{h}_{s}^\ast(A)
$$
and since $B \supset A$ and $B$ is $\mathfrak{h}_{s}$-measurable (being Borel) we can finally conclude
that
$$
\mathfrak{h}_{s}(B) = \mathfrak{h}_{s}^\ast(A),
$$
as desired.
I can elaborate abit why it is sufficient to show it for open ones. It might be that this is not exactly what the author was going after. If you wish, I may also continue this answer into a full proof which shows that $H^{\alpha}(C)\geq \frac{1}{2}$.
Choose for starters a $\delta$-cover $\{E_{j}\}_{j=1}^{\infty}$ of $C$ with $\sum_{j=1}^{\infty}\mathrm{diam}(E_{j})^{\alpha}\leq H^{\alpha}(C)+\delta$. Then for each $j$ we may choose a closed interval $I_{j}$ with $E_{j}\subset \mathrm{int}I_{j}$ and $\mathrm{diam}(I_{j})<(1+\delta)\mathrm{diam}(E_{j})$. Hence $\{\mathrm{int}I_{j}\}_{j=1}^{\infty}$ is an open cover of $C$, and in particular
\begin{align*}
H^{\alpha}(C)+\delta\geq \sum_{j=1}^{\infty}\mathrm{diam}(E_{j})^{\alpha}\geq (1+\delta)^{-\alpha}\sum_{j=1}^{\infty}\mathrm{diam}(I_{j})^{\alpha}.
\end{align*}
Thus to establish a lower bound for $H^{\alpha}(C)$ it suffices to establish a lower bound for $\sum_{j=1}^{\infty}\mathrm{diam}(I_{j})^{\alpha}$. (You may consider $\mathrm{int}(I_{j})$'s instead, which are open, as their diameter is the same as $I_{j}$'s.)
Best Answer
It is important to be able to sort through families of sets that cover other sets. When the covering is a family of balls, there are many theorems available. When the covering is arbitrary it can be much more challenging to sort. The Hausdorff measure and spherical Hausdorff measure (in integer dimensions) coincide on rectifiable sets, so in many important cases one is just as good as the other.
If $\{E_j\}$ is a covering of $E$ with sets whose diameter does not exceed $\delta$, you can select points $x_j \in E_j$ and consider the balls $\newcommand{\diam}{\mathrm{diam}\, }B_j = B(x_j,\diam E_j)$. Then $E_j \subset B_j$ and $\diam B_j = 2\diam E_j$ so that $\{B_j\}$ covers $E$, $\diam B_j \le 2\delta$ for all $j$, and $$S_{2\delta}^k(E) \le \sum_j \alpha_j \frac{(\diam B_j)^k}{2^k} = 2^k \sum_j \alpha_j\frac{(\diam E_j)^k}{2^k}.$$ Take the infimum over all such families $\{E_k\}$ to get $$S_{2\delta}^k(E) \le 2^k H_\delta^k(E).$$
In fact, the constant $2^k$ can be improved to $\left( \dfrac{2n}{n+1} \right)^{k/2}$.