These can be solved at the same time, the closed roll is just a special case of the open ended.
Let
$$d = \text{Number of sides on each die}$$
$$n = \text{Number of dice rolled}$$
$$s = \text{Score required for success}$$
$$m = \text{number of successes required}$$
Also, let
$$s = n^k+r$$
where
$$k\in 0,1,2,\dots$$
$$r\in 1,2,\dots,d$$
And, let
$$\begin{align}
p_o &= \text{chance of getting open ended}\\
&= \begin{cases}
\frac{1}{d}&,\text{if roll is open}\\
0&,\text{otherwise}\\
\end{cases}
\end{align}$$
So, for 1 die, we have a geometric distribution to get $k$ open ended results followed by a single roll to get $r$ (note that this is not exactly a geometric distribution since we need $k$ failures with $q=1-p_0$ followed by 1 success with a different probability). Also we will require $x^0=1$.
$$p_1=\left(p_0\right)^{k}\left(\frac{d-r+1}{d}\right)$$
To get $m$ success from $n$ dice, we have a binomial distribution as you noticed so
$$\begin{align}
p_{m,n}&=\sum_{i=m}^n\binom{n}{i}p_1^i\left(1-p_1\right)^{n-i}\\
\end{align}$$
Some examples:
- $d=6$, $n=1$, $s=4$, $m=1$ & open ended; so $k=0$, $r=4$
$$\begin{align}
p_0=\frac{1}{6}\\
\end{align}$$
$$\begin{align}
p_1&=\left(\frac{1}{6}\right)^{0}\left(\frac{6-4+1}{6}\right)\\
&=1\left(\frac{3}{6}\right)\\
&=1\times\frac{1}{2}\\
&=\frac{1}{2}\\
\end{align}$$
$$\begin{align}
p_{1,1}&=\sum_{i=1}^1\binom{1}{i}p_1^i\left(1-p_1\right)^{1-i}\\
&=\binom{1}{1}\left(\frac{1}{2}\right)^{1}\left(1-\frac{1}{2}\right)^{1-1}\\
&=1\times\frac{1}{2}\times{1}\\
&=\frac{1}{2}\\
\end{align}$$
- $d=6$, $n=1$, $s=10$, $m=1$ & open ended; so $k=1$, $r=4$
$$\begin{align}
p_0=\frac{1}{6}\\
\end{align}$$
$$\begin{align}
p_1&=\left(\frac{1}{6}\right)^{1}\left(\frac{6-4+1}{6}\right)\\
&=\frac{1}{6}\times\frac{3}{6}\\
&=\frac{1}{12}\\
\end{align}$$
$$\begin{align}
p_{1,1}&=\sum_{i=1}^1\binom{1}{i}p_1^i\left(1-p_1\right)^{1-i}\\
&=\binom{1}{1}\left(\frac{1}{12}\right)^{1}\left(1-\frac{1}{12}\right)^{1-1}\\
&=1\times\frac{1}{12}\times{1}\\
&=\frac{1}{12}\\
\end{align}$$
- $d=6$, $n=3$, $s=4$, $m=2$ & closed; so $k=0$, $r=4$
$$\begin{align}
p_0=0\\
\end{align}$$
$$\begin{align}
p_1&=\left(0\right)^{0}\left(\frac{6-4+1}{6}\right)\\
&=1\left(\frac{3}{6}\right)\\
&=\frac{1}{2}\\
\end{align}$$
$$\begin{align}
p_{3,2}&=\sum_{i=2}^3\binom{1}{i}p_1^i\left(1-p_1\right)^{1-i}\\
&=\binom{3}{2}\left(\frac{1}{2}\right)^{2}\left(1-\frac{1}{2}\right)^{3-2}+\binom{3}{3}\left(\frac{1}{2}\right)^{3}\left(1-\frac{1}{2}\right)^{3-3}\\
&=3\times\frac{1}{4}\times\frac{1}{2}+1\times\frac{1}{8}\times1\\
&=\frac{4}{8}\\
&=\frac{1}{2}\\
\end{align}$$
- $d=6$, $n=3$, $s=10$, $m=2$ & open ended; so $k=1$, $r=4$
$$\begin{align}
p_0=\frac{1}{6}\\
\end{align}$$
$$\begin{align}
p_1&=\left(\frac{1}{6}\right)^{1}\left(\frac{6-4+1}{6}\right)\\
&=\left(\frac{1}{6}\right)\left(\frac{3}{6}\right)\\
&=\frac{1}{12}\\
\end{align}$$
$$\begin{align}
p_{3,2}&=\sum_{i=2}^3\binom{1}{i}p_1^i\left(1-p_1\right)^{1-i}\\
&=\binom{3}{2}\left(\frac{1}{12}\right)^{2}\left(1-\frac{1}{12}\right)^{3-2}+\binom{3}{3}\left(\frac{1}{12}\right)^{3}\left(1-\frac{1}{12}\right)^{3-3}\\
&=3\times\frac{1}{144}\times\frac{11}{12}+1\times\frac{1}{1,728}\times1\\
&=\frac{34}{1,728}\\
&=\frac{17}{864}\\
&\approx0.02\\
\end{align}$$
ETA: OK, I think I've fixed the problem. Off-by-one error...
I think this can be done with generating functions. The generating function for a single die is given by
$$
F(z) = \frac{t-1}{d} + \frac{(d-t)z}{d} + \frac{zF(z)}{d}
$$
We can interpret this as follows: The probability that there are no hits on the one die is $\frac{t-1}{d}$, so $F(z)$ has that as the coefficient for $z^0 = 1$. The probability that there is one hit and the die doesn't "explode" (repeat) is $\frac{d-t}{d}$, so $F(z)$ has that as the coefficient for $z^1 = z$. In the remaining $\frac{1}{d}$ of the cases, the die explodes and the situation is exactly as it was at the start, except that there is one hit already to our credit, which is why we have $zF(z)$: the $F(z)$ takes us back to the beginning, so to speak, and the multiplication by $z$ takes care of the existing hit.
This expression can be solved for $F(z)$ via simple algebra to yield
$$
F(z) = \frac{t-1+(d-t)z}{d-z}
$$
whose $z^h$ coefficient gives the probability for $h$ hits. For example, for the simple case $n = 1, d = 20, t = 11$:
\begin{align}
F(z) & = \frac{10+9z}{20-z} \\
& = \frac{10+9z}{20} \left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
& = \left( \frac{1}{2} + \frac{9}{20}z \right)
\left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
\end{align}
and then we obtain the probability that there are $h$ hits from the $z^h$ coefficient of $F(z)$ as
$$
P(H = h) = \frac{1}{2\cdot20^h}+\frac{9}{20^h} = \frac{19}{2\cdot20^h}
\qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{1}{2}
$$
In general, we can obtain the expectation of the number of hits $\overline{H}$ as
$$
\overline{H} = F'(1) = \frac{d(d-t)+t-1}{(d-1)^2} = \frac{d+1-t}{d-1}
$$
Now, for $n$ dice, we have
$$
[F(z)]^n = \left[ \frac{t-1+(d-t)z}{d-z} \right]^n
$$
We can write this as $N(z)M(z)$, where
\begin{align}
N(z) & = [t-1+(d-t)z]^n \\
& = \sum_{k=0}^n \binom{n}{k} (t-1)^{n-k}(d-t)^kz^k
\end{align}
and
\begin{align}
M(z) & = \left(\frac{1}{d-z}\right)^n \\
& = \frac{1}{d^n} \left( 1+\frac{z}{d}+\frac{z^2}{d^2}+\cdots \right)^n \\
& = \sum_{j=0}^\infty \binom{n+j-1}{j} \frac{z^j}{d^{n+j}}
\end{align}
so we can obtain a closed form for $P(H = h)$ from the $z^h$ coefficient of $[F(z)]^n = N(z)M(z)$ as
\begin{align}
P(H = h) & = \sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\frac{(t-1)^{n-k}(d-t)^k}{d^{n+h-k}} \\
& = \frac{(t-1)^n}{d^{n+h}}
\sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\left[ \frac{d(d-t)}{t-1} \right]^k
\end{align}
For example, for $n = 1, d = 6, t = 5$ (the example in the OP), the above expression yields
$$
P(H = h) = \frac{5}{3 \cdot 6^h} \qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{2}{3}
$$
which coincides with the conclusions drawn in the comments to the OP.
The expectation for the number of hits could be obtained by evaluating $\frac{d}{dz} [F(z)]^n$ at $z = 1$, but owing to the linearity of expectation, it is obtained more straightforwardly as $n$ times the expected number of hits for one die, namely
$$
\overline{H} = \frac{n(d+1-t)}{d-1}
$$
I think this all checks out, but some independent verification (or disproof, as appropriate) would be nice.
Best Answer
The method closest to the one of the link is to write the generating function as $F(z,r)$.
So, expanding on the scenario of the linked answer from 7 years ago, we have $$ F(z,r) =\begin{cases} \frac{t-1}{d} + \frac{(d-t+1)z}{d} &, r=0\\ \frac{t-1}{d} + \frac{(d-t)z}{d} + \frac{zF(z,r-1)}{d} &, else \end{cases} $$
Unrolling the recursion then gives $$ F(z,r)=\frac{z^r}{d^r}\cdot F(z,0)+\sum_{i=0}^{r-1}\left(\frac zd\right)^i·\left(\frac{t - 1}d + \frac{(d - t)·z}d\right) $$
As geometric series, we can compute its closed form: $$ F(z,r)=\frac{d^{-r-1}\cdot z^{r+1}\cdot (d-t+1)\cdot (z-1)}{z-d}+\frac{z\cdot (d-t)+t-1}{d-z} $$