ETA: OK, I think I've fixed the problem. Off-by-one error...
I think this can be done with generating functions. The generating function for a single die is given by
$$
F(z) = \frac{t-1}{d} + \frac{(d-t)z}{d} + \frac{zF(z)}{d}
$$
We can interpret this as follows: The probability that there are no hits on the one die is $\frac{t-1}{d}$, so $F(z)$ has that as the coefficient for $z^0 = 1$. The probability that there is one hit and the die doesn't "explode" (repeat) is $\frac{d-t}{d}$, so $F(z)$ has that as the coefficient for $z^1 = z$. In the remaining $\frac{1}{d}$ of the cases, the die explodes and the situation is exactly as it was at the start, except that there is one hit already to our credit, which is why we have $zF(z)$: the $F(z)$ takes us back to the beginning, so to speak, and the multiplication by $z$ takes care of the existing hit.
This expression can be solved for $F(z)$ via simple algebra to yield
$$
F(z) = \frac{t-1+(d-t)z}{d-z}
$$
whose $z^h$ coefficient gives the probability for $h$ hits. For example, for the simple case $n = 1, d = 20, t = 11$:
\begin{align}
F(z) & = \frac{10+9z}{20-z} \\
& = \frac{10+9z}{20} \left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
& = \left( \frac{1}{2} + \frac{9}{20}z \right)
\left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
\end{align}
and then we obtain the probability that there are $h$ hits from the $z^h$ coefficient of $F(z)$ as
$$
P(H = h) = \frac{1}{2\cdot20^h}+\frac{9}{20^h} = \frac{19}{2\cdot20^h}
\qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{1}{2}
$$
In general, we can obtain the expectation of the number of hits $\overline{H}$ as
$$
\overline{H} = F'(1) = \frac{d(d-t)+t-1}{(d-1)^2} = \frac{d+1-t}{d-1}
$$
Now, for $n$ dice, we have
$$
[F(z)]^n = \left[ \frac{t-1+(d-t)z}{d-z} \right]^n
$$
We can write this as $N(z)M(z)$, where
\begin{align}
N(z) & = [t-1+(d-t)z]^n \\
& = \sum_{k=0}^n \binom{n}{k} (t-1)^{n-k}(d-t)^kz^k
\end{align}
and
\begin{align}
M(z) & = \left(\frac{1}{d-z}\right)^n \\
& = \frac{1}{d^n} \left( 1+\frac{z}{d}+\frac{z^2}{d^2}+\cdots \right)^n \\
& = \sum_{j=0}^\infty \binom{n+j-1}{j} \frac{z^j}{d^{n+j}}
\end{align}
so we can obtain a closed form for $P(H = h)$ from the $z^h$ coefficient of $[F(z)]^n = N(z)M(z)$ as
\begin{align}
P(H = h) & = \sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\frac{(t-1)^{n-k}(d-t)^k}{d^{n+h-k}} \\
& = \frac{(t-1)^n}{d^{n+h}}
\sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\left[ \frac{d(d-t)}{t-1} \right]^k
\end{align}
For example, for $n = 1, d = 6, t = 5$ (the example in the OP), the above expression yields
$$
P(H = h) = \frac{5}{3 \cdot 6^h} \qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{2}{3}
$$
which coincides with the conclusions drawn in the comments to the OP.
The expectation for the number of hits could be obtained by evaluating $\frac{d}{dz} [F(z)]^n$ at $z = 1$, but owing to the linearity of expectation, it is obtained more straightforwardly as $n$ times the expected number of hits for one die, namely
$$
\overline{H} = \frac{n(d+1-t)}{d-1}
$$
I think this all checks out, but some independent verification (or disproof, as appropriate) would be nice.
Best Answer
To see why the probability is $\frac78,$ let's slightly rephrase the question: let's say instead of simulating the flips until we get a success (let's say heads) let's just flip the coin three times straight away and then look at the results. A set of three flips is successful if and only if there is at least one heads in the three flips, so the probability of a success must be equal to the probability of getting at least one heads.
So, letting $X \sim B(3, 0.5)$ be the number of heads, we want $\text{P}(X \geq 1).$ By the law of complements, this is equal to $1 - \text{P}(X = 0) = 1 - \left(\frac12\right)^3 = \frac78.$
Alternatively, using a sample space, there are $2^3 = 8$ possible sequences of three flips, and only $1$ has all three tails, so the other $7$ have at least one heads for a probability of $\frac78.$