In a metric space $X$ with metric $d$, a subset $A$ of $X$ is open (relative to the metric) if and only if for every $a\in A$ there exists $\delta\gt 0$ such that
$$B_X(a,\delta) = \{x\in X\mid d(x,a) \lt \delta\} \subseteq A.$$
Note that everything happens inside of $X$. You don’t get to “go outside” of $X$, even if such an outside naturally exists.
Now suppose you have a subset $S\subseteq X$. The metric of $X$ restricts to $S$, and so $(S,d|_S)$ is also a metric space, and hence we can ask about subsets of $S$ being open or not. When is $A\subseteq S$ open in this space? When for any $a\in A$, there exists $\delta\gt 0$ such that
$$B_S(a,\delta) = \{x\in S\mid d(x,a)\lt \delta\}\subseteq A.$$
That is, the same condition, but now we only look at elements of $S$, not at elements of $X$. Why? because you aren’t looking at “open-in-$X$”, you are looking at open-in-$S$. You only look at elements of $S$, you ignore the elements that are not in $S$.
So when $X=[0,1]$ with the usual metric, the set $A=[0,x)$, $0\lt x\lt 1$, is open. Why? Because if we take $a\in A$, then $0\leq a\lt x$. Letting $\delta = \frac{1}{2}(x-a)$, we have that
$$B_X(a,\delta) = \{ r\in [0,1)\mid d(a,r)\lt \delta\}$$
is completely contained in $A$.
Second question: $B$ is closed (in $M$), so its complement is open. $A=M-B$ is open, and then $B=M-A$ with $A$ open (in $M$).
Now, I claim that $S\cap A$ is open in $S$: if $a\in S\cap A$, then because $A$ is open in $M$, there exists $\delta\gt 0$ such that $B_M(x,\delta)\subseteq A$. Now consider $B_S(x,\delta)$. We have
$$\begin{align*}
B_S(x,\delta) &= \{s\in S\mid d(s,x)\lt \delta\}\\
&\subseteq \{s\in M\mid d(s,x)\lt \delta\} &\text{(since }S\subseteq M)\\
&= B_M(x,\delta)\\
&\subseteq A.
\end{align*}$$
On the other hand, $B_S(x,\delta)\subseteq S$. Since $B_S(x,\delta)$ is contained in both $S$ and $A$, then $B_S(x,\delta)\subseteq S\cap A$.
Thus we have shown that for every $x\in S\cap A$, there exists $\delta$ such that $B_S(x,\delta)\subseteq S\cap A$. Hence $S\cap A$ is open in $S$.
Note that “boundary” doesn’t enter into it. This is all about balls in the appropriate ambient set.
Okay, now, since $S\cap A$ is open in $S$, then $S-(S\cap A)$ is closed in $S$. And now it is an easy check to verify that
$$S-A = S-(S\cap A).$$
Thus, $S-A$ is closed in $S$, as claimed.
Again, note that boundary doesn’t enter into it.
Your error throughout is thinking that when you work in the relative metric of the subset, you should consider all points in the ambient space. You don’t; you just take points in the subset. Same way when we don’t take spheres in $\mathbb{R}^3$ when we look at open sets in $\mathbb{R}^2$.
Best Answer
... then you take a smaller radius.
A set $Z$ is open if for every point $x$ you can find an open ball (which might be very small) centered at $x$ that is contained in $Z$.
It doesn't mean every ball will work. It only means some balls will work. Actually it only needs to mean at least one ball will work. [That usually means the smaller balls will work too but sometimes there aren't any smaller balls (don't worry about that now.)]
If for example if you take the interior of a unit circle in a plane centered at $(0,0)$. That's open because "it has fuzzy edges"... Anyway if you take any point in it. Say $(0, 0.9999)$ then I can find an open ball centered at $(0,0.9999)$ entirely in the circle. To do that I have to take a radius $r \le 0.0001$; but I can do it if I take a radius small enough.
Now what if instead I had taken a radius of $27$ and that's way too big to fit in the circle? Well, I shrug my shoulders and give the person asking a confused look and say "Who cares about that radius? I can take a radius of $0.0001$ and that does work. I dont care about radii that don't work. I just care that there are radii the do work."