You only need to consider the case $\mathfrak{h}_{s}^\ast(A) \lt \infty$, but you need to be a bit careful in choosing the outer approximations since swapping $\inf$ and $\sup$ certainly isn't allowed without some thinking. If you knew that you can always take the same set $E$ in the $\inf$ (which I will show in \eqref{eqn:ast} below) then you'd be essentially done, but it is impossible to say if you got so far or not.
I'm going to ignore $\alpha_s$ since it only is a scalar factor that doesn't play a rôle in the argument and the formulas will already get cumbersome enough without it.
The following argument is essentially the one from Fremlin's measure theory, Volume 4II, 471D, page 102. $\DeclareMathOperator{\diam}{diam}$ $\newcommand{\hsd}[1]{\mathfrak{h}_{s,#1}^\ast}$
Set $\delta_n = 2^{-n}$ and choose $(A_{i}^n)_{i \in \mathbb{N}}$ of diameter $\leq 2^{-n}$ such that $A \subset \bigcup_{i=1}^\infty A_{i}^n$ and that
\begin{equation}\tag{$1$}\label{eqn:eq1}
\sum_{i=1}^\infty (\diam{A_{i}^n})^s \leq \hsd{2^{-n}}(A) +2^{-n},
\end{equation}
which is possible because of the definition of $\hsd{2^{-n}}$ as infimum.
Observe that there's no reason for the $A_{i}^n$ to be $\hsd{2^{-n}}$-measurable, let alone Borel, so we would like to “blow them up” slightly, so as to get open sets still approximating the $\hsd{2^{-n}}$-measure of $A$ well. The problem is that by doing so we will lose the diameter condition which appears in the definition of $\hsd{2^{-n}}$, but this isn't a serious problem: we can simply choose a larger $n$ and work with the sets we obtain from there. Here are the gory details:
Choose $0 \lt r_{i}^n \lt 2^{-n}$ so small that
$$
(\diam{(A_{i}^n)} + 2r_{i}^n)^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
$$
Now put $U_{i}^n = \{x \in X\,:\,d(x,A_{i}^n) \lt r_{i}^n\}$ and note that $U_{i}^n \supset A_{i}^n$ is an open set whose diameter satisfies
\begin{equation}\tag{$2$}\label{eqn:eq2}
(\diam{U_{i}^n})^s \leq (\diam{A_{i}^n})^s + 2^{-n-i}.
\end{equation}
Let
$$
B = \bigcap_{n=1}^\infty \bigcup_{i=1}^\infty U_{i}^n
$$
and observe that $B$ is a $G_{\delta}$-set (countable intersection of open sets) containing $A$.
Given any $\delta \gt 0$, we can find $N$ such that $3 \cdot 2^{-N} \leq \delta$ so that for all $n \geq N$ we have $\diam{U_{i}^n} \leq \delta$. As $B \subset \bigcup_{i = 1}^\infty U_{i}^n$ we see from \eqref{eqn:eq1} and \eqref{eqn:eq2} that for $n \geq N$
$$
\hsd{\delta}(B) \leq \sum_{i=1}^\infty (\diam{U_{i}^n})^s
\leq
\sum_{i=1}^\infty [(\diam{A_{i}^n})^s + 2^{-n-i}] \leq \hsd{2^{-n}}{(A)} + 2 \cdot 2^{-n}
$$
Since $\hsd{2^{-n}}(A) \leq \mathfrak{h}_{s}^\ast(A)$ we get for $n \geq N$
$$
\hsd{\delta}(B) \leq
\mathfrak{h}_{s}^\ast(A) + 2^{-n}
$$
and letting $n \to \infty$ this gives
\begin{equation}\tag{$\ast$}\label{eqn:ast}
\hsd{\delta}(B) \leq \mathfrak{h}_{s}^\ast(A)
\end{equation}
for every $\delta \gt 0$. [Note: this is stronger than your condition involving $\inf$ since we can specify the set $E = B$ and thus avoid the infimum]
Taking the $\sup$ over all $\delta$ in \eqref{eqn:ast} this yields
$$
\mathfrak{h}_{s}^\ast(B) \leq \mathfrak{h}_{s}^\ast(A)
$$
and since $B \supset A$ and $B$ is $\mathfrak{h}_{s}$-measurable (being Borel) we can finally conclude
that
$$
\mathfrak{h}_{s}(B) = \mathfrak{h}_{s}^\ast(A),
$$
as desired.
Let $A\subseteq(0,1]$. It is easy to see that
$$0\leq\delta_x^*(A)\leq1.$$
Suppose first that $x\in A$. Then for any sequence $\{A_j\}_{j\in\mathbb{N}}$ in $B$ with $A\subseteq\bigcup_{j\in\mathbb{N}}A_j$ we have that $x\in A_k$ for some $k\in\mathbb{N}$. In particular then
$$\sum_{j=0}^\infty\delta_x(A_j)\geq\delta_x(A_k)=1.$$
Taking the infimum over all such sequences $\{A_j\}_{j\in\mathbb{N}}$ we get that $\delta_x^*(A)\geq1$, and so $\delta_x^*(A)=1$.
Suppose now that $x\notin A$. Choose a sequence $\{A_j\}_{j\in\mathbb{N}}$ in $B$ such that $A\subseteq\bigcup_{j\in\mathbb{N}}A_j=(0,1]\setminus\{x\}$ (this can easily be done explicitly if you're unsure about why such a sequence exists). This means that $\delta_x(A_j)=0$ for all $j\in\mathbb{N}$, and so
$$0\leq\delta_x^*(A)\leq\sum_{j=0}^\infty\delta_x(A_j)=0,$$
from which it follows that $\delta_x^*(A)=0$.
So we have now proven than $\delta_x^*(A)=\delta_x(A)$ for any $A\subseteq(0,1]$. I'll leave the measurability part of the question to you, as that should be a very short and straightforward argument.
Best Answer
If $x\in E$ the formula $1=\delta_x(E)=\inf \{\delta_x(U):E\subseteq U, U \text{ open}\}$ holds because $x\in U$ for every relevant $U$ and so $\delta_x(U)=1$. If $x\notin E$, take the set $U=X\backslash \{x\}$ (open because $\{x\}$ is closed because $X$ is Hausdorff).