You only need to consider the case $\mathfrak{h}_{s}^\ast(A) \lt \infty$, but you need to be a bit careful in choosing the outer approximations since swapping $\inf$ and $\sup$ certainly isn't allowed without some thinking. If you knew that you can always take the same set $E$ in the $\inf$ (which I will show in \eqref{eqn:ast} below) then you'd be essentially done, but it is impossible to say if you got so far or not.
I'm going to ignore $\alpha_s$ since it only is a scalar factor that doesn't play a rôle in the argument and the formulas will already get cumbersome enough without it.
The following argument is essentially the one from Fremlin's measure theory, Volume 4II, 471D, page 102. $\DeclareMathOperator{\diam}{diam}$ $\newcommand{\hsd}[1]{\mathfrak{h}_{s,#1}^\ast}$
Set $\delta_n = 2^{-n}$ and choose $(A_{i}^n)_{i \in \mathbb{N}}$ of diameter $\leq 2^{-n}$ such that $A \subset \bigcup_{i=1}^\infty A_{i}^n$ and that
\begin{equation}\tag{$1$}\label{eqn:eq1}
\sum_{i=1}^\infty (\diam{A_{i}^n})^s \leq \hsd{2^{-n}}(A) +2^{-n},
\end{equation}
which is possible because of the definition of $\hsd{2^{-n}}$ as infimum.
Observe that there's no reason for the $A_{i}^n$ to be $\hsd{2^{-n}}$-measurable, let alone Borel, so we would like to “blow them up” slightly, so as to get open sets still approximating the $\hsd{2^{-n}}$-measure of $A$ well. The problem is that by doing so we will lose the diameter condition which appears in the definition of $\hsd{2^{-n}}$, but this isn't a serious problem: we can simply choose a larger $n$ and work with the sets we obtain from there. Here are the gory details:
Choose $0 \lt r_{i}^n \lt 2^{-n}$ so small that
$$
(\diam{(A_{i}^n)} + 2r_{i}^n)^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
$$
Now put $U_{i}^n = \{x \in X\,:\,d(x,A_{i}^n) \lt r_{i}^n\}$ and note that $U_{i}^n \supset A_{i}^n$ is an open set whose diameter satisfies
\begin{equation}\tag{$2$}\label{eqn:eq2}
(\diam{U_{i}^n})^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
\end{equation}
Let
$$
B = \bigcap_{n=1}^\infty \bigcup_{i=1}^\infty U_{i}^n
$$
and observe that $B$ is a $G_{\delta}$-set (countable intersection of open sets) containing $A$.
Given any $\delta \gt 0$, we can find $N$ such that $3 \cdot 2^{-N} \leq \delta$ so that for all $n \geq N$ we have $\diam{U_{i}^n} \leq \delta$. As $B \subset \bigcup_{i = 1}^\infty U_{i}^n$ we see from \eqref{eqn:eq1} and \eqref{eqn:eq2} that for $n \geq N$
$$
\hsd{\delta}(B) \leq \sum_{i=1}^\infty (\diam{U_{i}^n})^s
\leq
\sum_{i=1}^\infty [(\diam{A_{i}^n})^s + 2^{-n-i}] \leq \hsd{2^{-n}}{(A)} + 2 \cdot 2^{-n}
$$
Since $\hsd{2^{-n}}(A) \leq \mathfrak{h}_{s}^\ast(A)$ we get for $n \geq N$
$$
\hsd{\delta}(B) \leq
\mathfrak{h}_{s}^\ast(A) + 2^{-n}
$$
and letting $n \to \infty$ this gives
\begin{equation}\tag{$\ast$}\label{eqn:ast}
\hsd{\delta}(B) \leq \mathfrak{h}_{s}^\ast(A)
\end{equation}
for every $\delta \gt 0$. [Note: this is stronger than your condition involving $\inf$ since we can specify the set $E = B$ and thus avoid the infimum]
Taking the $\sup$ over all $\delta$ in \eqref{eqn:ast} this yields
$$
\mathfrak{h}_{s}^\ast(B) \leq \mathfrak{h}_{s}^\ast(A)
$$
and since $B \supset A$ and $B$ is $\mathfrak{h}_{s}$-measurable (being Borel) we can finally conclude
that
$$
\mathfrak{h}_{s}(B) = \mathfrak{h}_{s}^\ast(A),
$$
as desired.
Best Answer
Subset $A$ of a measure 0 set $B$ would have outer measure 0. Given any $E$ then $\mu(A\bigcap E)=0$ because $A\bigcap E\subset B$; and $\mu(A^{c}\bigcap E)=\mu(E)$ because $B^{c}\bigcap E\subset A^{c}\bigcap E\subset E$ and $\mu(B^{c}\bigcap E)=\mu(E)$ since $B$ is measurable with measure 0.
Hence $\mu(A\bigcap E)+\mu(A^{c}\bigcap E)=\mu(E)$ so $A$ is measurable.
A more intuitive way to look at it is that a null set basically have no effects whatsoever, and definition of measurable sets depend only on what kind of harm the set could do.