ETA: OK, I think I've fixed the problem. Off-by-one error...
I think this can be done with generating functions. The generating function for a single die is given by
$$
F(z) = \frac{t-1}{d} + \frac{(d-t)z}{d} + \frac{zF(z)}{d}
$$
We can interpret this as follows: The probability that there are no hits on the one die is $\frac{t-1}{d}$, so $F(z)$ has that as the coefficient for $z^0 = 1$. The probability that there is one hit and the die doesn't "explode" (repeat) is $\frac{d-t}{d}$, so $F(z)$ has that as the coefficient for $z^1 = z$. In the remaining $\frac{1}{d}$ of the cases, the die explodes and the situation is exactly as it was at the start, except that there is one hit already to our credit, which is why we have $zF(z)$: the $F(z)$ takes us back to the beginning, so to speak, and the multiplication by $z$ takes care of the existing hit.
This expression can be solved for $F(z)$ via simple algebra to yield
$$
F(z) = \frac{t-1+(d-t)z}{d-z}
$$
whose $z^h$ coefficient gives the probability for $h$ hits. For example, for the simple case $n = 1, d = 20, t = 11$:
\begin{align}
F(z) & = \frac{10+9z}{20-z} \\
& = \frac{10+9z}{20} \left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
& = \left( \frac{1}{2} + \frac{9}{20}z \right)
\left(1+\frac{z}{20}+\frac{z^2}{20^2}+\cdots\right) \\
\end{align}
and then we obtain the probability that there are $h$ hits from the $z^h$ coefficient of $F(z)$ as
$$
P(H = h) = \frac{1}{2\cdot20^h}+\frac{9}{20^h} = \frac{19}{2\cdot20^h}
\qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{1}{2}
$$
In general, we can obtain the expectation of the number of hits $\overline{H}$ as
$$
\overline{H} = F'(1) = \frac{d(d-t)+t-1}{(d-1)^2} = \frac{d+1-t}{d-1}
$$
Now, for $n$ dice, we have
$$
[F(z)]^n = \left[ \frac{t-1+(d-t)z}{d-z} \right]^n
$$
We can write this as $N(z)M(z)$, where
\begin{align}
N(z) & = [t-1+(d-t)z]^n \\
& = \sum_{k=0}^n \binom{n}{k} (t-1)^{n-k}(d-t)^kz^k
\end{align}
and
\begin{align}
M(z) & = \left(\frac{1}{d-z}\right)^n \\
& = \frac{1}{d^n} \left( 1+\frac{z}{d}+\frac{z^2}{d^2}+\cdots \right)^n \\
& = \sum_{j=0}^\infty \binom{n+j-1}{j} \frac{z^j}{d^{n+j}}
\end{align}
so we can obtain a closed form for $P(H = h)$ from the $z^h$ coefficient of $[F(z)]^n = N(z)M(z)$ as
\begin{align}
P(H = h) & = \sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\frac{(t-1)^{n-k}(d-t)^k}{d^{n+h-k}} \\
& = \frac{(t-1)^n}{d^{n+h}}
\sum_{k=0}^{\max\{h, n\}} \binom{n}{k} \binom{n+h-k-1}{h-k}
\left[ \frac{d(d-t)}{t-1} \right]^k
\end{align}
For example, for $n = 1, d = 6, t = 5$ (the example in the OP), the above expression yields
$$
P(H = h) = \frac{5}{3 \cdot 6^h} \qquad h > 0
$$
with the special case
$$
P(H = 0) = \frac{2}{3}
$$
which coincides with the conclusions drawn in the comments to the OP.
The expectation for the number of hits could be obtained by evaluating $\frac{d}{dz} [F(z)]^n$ at $z = 1$, but owing to the linearity of expectation, it is obtained more straightforwardly as $n$ times the expected number of hits for one die, namely
$$
\overline{H} = \frac{n(d+1-t)}{d-1}
$$
I think this all checks out, but some independent verification (or disproof, as appropriate) would be nice.
Best Answer
If you have several dice, and you want to know what chance you have of getting a specific formula, then the solution is P = (h+m)^n, where h is the chance of hits, m is the chances of missing, and n is the number of dice involved.
This expands out to a polynomial, eg for 3, h³ + 3h²m + 3hm² + m³.
So if your chance of hitting is say 2 (ie 5, 6), and your miss is 4, (ie 1,2,3,4), this comes out to
8 tri-hits, 3*4*4 = 48 bi-hits, 3*2*16 = 96 hits and 4*4*4 = 64 misses. The sum of these numnbers ought be 216 tosses.
You use the row of the pascal triangle, along with powers of the hit/miss ratio, as
n! means 1 * 2 * 3 * 4 * 5 * ... n! It is the factorial.
The first line counts all the different ways of arranging say two hits and six misses (at any probability), ie hmmmhmmm and hmhmmmm are both counted.
The second line counts the possible ways of getting 6 hits, eg 5,6,5,5,6,5 six hits. There are 64 ways of doing this.
The third line counts the possible ways of getting misses, so 2,2,4,3,1 is a way of getting 5 misses.
The total of the row is (H+M)^n, the total possible throws over all the dice. In the present case, H=2, M=4, and H+M=6 possible values, so you have 6^8 = 1679616 possible throws.
The number of 4 hits from 8 die is then 70*16*256/1679616 = 0.170705685
This formula can be implemented in code to handle a large number of throws.