Problem 1: The chance for a seven is $6/36 = 1/6$ and the chance for an eight is $5/36$. The chance for neither is $25/36$.
We want to compute the chance of getting neither $k$ times and then getting a seven, where $k$ goes from $0$ up. The chance is thus:
$$\frac{1}{6} + \frac{25}{36}·\frac{1}{6} + \frac{25^2}{36^2}·\frac{1}{6} + \ldots = \\ \frac{1}{6}\sum_{k=0}^\infty \left(\frac{25}{36}\right)^k = \frac{1}{6}·\frac{1}{1-\frac{25}{36}} = \frac{6}{11}$$
Problem 2
This is a bit more tricky. We are going to compute the chance for the alternative: to het two sevens but not both a six and an eight. Then, if $p$ is the chance for that to happen, what we want is $1-p$.
We have two cases:
We have a sequence of $k$ numbers, none of wich are seven or six (but can be eight), then a seven, then another sequence of $k'$ numbers that are not six nor seven, and then another seven.
OR
We have a sequence of $k$ numbers, none of wich are seven or eight (but can be six), then a seven, then another sequence of $k'$ numbers that are not eight nor seven, and then another seven.
(note that both cases are mutually exclusive)
The chance for a six is $5/36$ and the chance for nor six nor seven is $25/36$.
So we basically have to add all the numbers of the form:
$$\underbrace{\frac{25}{36}·\frac{25}{36}\ldots \frac{25}{36}}_{k \text{times}}·\frac{1}{6}·\underbrace{\frac{25}{36}·\frac{25}{36}\ldots \frac{25}{36}}_{k' \text{times}}·\frac{1}{6}$$
$$\sum_{k=0}^\infty \sum_{k'=0}^\infty \left(\frac{25}{36}\right)^k\frac{1}{6}\left(\frac{25}{36}\right)^{k'}\frac{1}{6}=\frac{1}{6}\sum_{k=0}^\infty \left(\frac{25}{36}\right)^k·\frac{1}{6}\sum_{k'=0}^\infty \left(\frac{25}{36}\right)^{k'} = \left(\frac{6}{11}\right)^2 = \frac{36}{121}$$
That's the chance for one of the cases, but since both cases have the same chance, the total chance of the alternative is $\displaystyle\frac{72}{121}$, therefore, the chance we want is $1-\displaystyle\frac{72}{121} = \frac{49}{121}$
This is a revision of my previous answer (see below).
Well, I was being stupid. Or at least, I guessed wrong about the definitions of $R_{6,7,8}$ and $R_{7,6,8}.$ It looks like we are meant to define events as follows:
- the sequence ends in $8$ with no $7$ having been rolled. Call this event $R_{6,8}.$
- the sequence ends in $8$ with a $7$ having been rolled after the first $6.$ Call this event $R_{6,7,8}.$
- the sequence ends in $8$ with a $7$ having been rolled before the first $6.$ Call this event $R_{7,6,8}.$
These definitions differ from my previous ones in the replacement of "last $6$" by "first $6.$"
An outcome in $R_{6,8}$ consists of
- a sequence of rolls containing no $6,$ $7,$ or $8$,
- a $6$,
- a sequence of rolls containing no $7$ or $8$,
- an $8$.
The probability that the first two stages will unfold in this way is
$$
\sum_{n=0}^\infty \left(\frac{20}{36}\right)^n\cdot\frac{5}{36}=\frac{1}{16/36}\cdot\frac{5}{36}=\frac{5}{16}.
$$
This can be reformulated in terms of conditional probability by considering that a sequence containing no $6,$ $7,$ or $8$ has zero probability of continuing forever. So a $6,$ $7,$ or $8$ eventually comes up with probability $1.$ Given that a $6,$ $7,$ or $8$ comes up, the probability that it is a $6$ is $5/16.$ Applying this reasoning to the last two stages as well, we get
$$
\mathbb{P}(R_{6,8})=\mathbb{P}(6\mid6,7,8)\cdot\mathbb{P}(8\mid7,8)=\frac{5}{16}\cdot\frac{5}{11}=\frac{25}{176}.
$$
Now an outcome in $R_{6,7,8}$ consists of
- a sequence of rolls containing no $6,$ $7,$ or $8$,
- a $6$,
- a sequence of rolls containing no $7$ or $8$,
- a $7$,
- a sequence of rolls containing no $7$ or $8$,
- an $8$.
We get
$$
\mathbb{P}(R_{6,7,8})=\mathbb{P}(6\mid6,7,8)\cdot\mathbb{P}(7\mid7,8)\cdot\mathbb{P}(8\mid7,8)=\frac{5}{16}\cdot\frac{6}{11}\cdot\frac{5}{11}=\frac{150}{1936}.
$$
Finally, an outcome in $R_{7,6,8}$ consists of
- a sequence of rolls containing no $6,$ $7,$ or $8$,
- a $7$,
- a sequence of rolls containing no $6,$ $7,$ or $8$,
- a $6$,
- a sequence of rolls containing no $7$ or $8$,
- an $8$.
We get
$$
\mathbb{P}(R_{7,6,8})=\mathbb{P}(7\mid6,7,8)\cdot\mathbb{P}(6\mid6,7,8)\cdot\mathbb{P}(8\mid7,8)=\frac{5}{16}\cdot\frac{6}{16}\cdot\frac{5}{11}=\frac{150}{2816}.
$$
Previous answer
[This answer uses different definitions of $R_{6,7,8}$ and $R_{7,6,8}$ than does the answer above, leading to the interchange of the values of these probabilities. Interestingly, this difference also seems to result in a formulation that does not lend itself to a conditional probability interpretation. Obviously the definitions in the answer above are the ones intended by the book, so I retract my statement that the book is wrong.]
The answer your book gives is right, but, unless I'm missing something, the solution method is incorrect. In particular, the probabilities they obtain for $\mathbb{P}(R_{6,7,8})$ and $\mathbb{P}(R_{7,6,8})$ are switched. Intuitively, $\mathbb{P}(R_{7,6,8})$ should be larger than $\mathbb{P}(R_{6,7,8})$ because in the event $R_{7,6,8}$ you may continue to roll $6$s after a $7$ has been rolled whereas in the event $R_{6,7,8}$ you may not. Unfortunately, the book's solution says that $\mathbb{P}(R_{7,6,8})$ is the smaller of the two. The solution I give below verifies that the intuition is correct, and I have checked this by simulation as well. I don't actually understand how to use conditional probability to solve this problem in the manner that the book was attempting, and would be interested if anyone could shed light on that.
Here's my approach: by symmetry, the probability of ending in $8$ is equal to the probability of ending in $6.$ So we compute the probability of ending in $8$, and then double it. To end in $8,$ an $8$ must be rolled exactly once, a $7$ may be rolled at most once, and a $6$ may be rolled any number of times as long as it is rolled at least once. There are three cases:
- end in $8$ with no $7$ having been rolled. Call this event $R_{6,8}.$
- end in $8$ with a $7$ having been rolled after the last $6.$ Call this event $R_{6,7,8}.$
- end in $8$ with a $7$ having been rolled before the last $6.$ Call this event $R_{7,6,8}.$
Now
$$\begin{aligned}
\mathbb{P}(R_{6,8})=&\sum_{m=0}^\infty\left[\mathbb{P}(\text{not 7 or 8})\right]^m\cdot\mathbb{P}(6)\cdot\sum_{n=0}^\infty\left[\mathbb{P}(\text{not 6, 7, or 8})\right]^n\cdot\mathbb{P}(8)\\
=&\sum_{m=0}^\infty\left(\frac{25}{36}\right)^m\cdot\frac{5}{36}\cdot\sum_{n=0}^\infty\left(\frac{20}{36}\right)^n\cdot\frac{5}{36}\\
=&\frac{1}{11/36}\cdot\frac{5}{36}\cdot\frac{1}{16/36}\cdot\frac{5}{36}\\
=&\frac{5}{11}\cdot\frac{5}{16}=\frac{25}{176}.
\end{aligned}$$
Similarly,
$$\begin{aligned}
\mathbb{P}(R_{6,7,8})=&\sum_{\ell=0}^\infty\left[\mathbb{P}(\text{not 7 or 8})\right]^\ell\cdot\mathbb{P}(6)\cdot\sum_{m=0}^\infty\left[\mathbb{P}(\text{not 6, 7, or 8})\right]^m\cdot\mathbb{P}(7)\\&\cdot\sum_{n=0}^\infty\left[\mathbb{P}(\text{not 6, 7, or 8})\right]^n\cdot\mathbb{P}(8)\\
=&\sum_{\ell=0}^\infty\left(\frac{25}{36}\right)^\ell\cdot\frac{5}{36}\cdot\sum_{m=0}^\infty\left(\frac{20}{36}\right)^m\cdot\frac{6}{36}\cdot\sum_{n=0}^\infty\left(\frac{20}{36}\right)^n\cdot\frac{5}{36}\\
=&\frac{1}{11/36}\cdot\frac{5}{36}\cdot\frac{1}{16/36}\cdot\frac{6}{36}\cdot\frac{1}{16/36}\cdot\frac{5}{36}\\
=&\frac{5}{11}\cdot\frac{6}{16}\cdot\frac{5}{16}=\frac{150}{2816}.
\end{aligned}$$
Finally,
$$\begin{aligned}
\mathbb{P}(R_{7,6,8})=&\sum_{\ell=0}^\infty\left[\mathbb{P}(\text{not 7 or 8})\right]^\ell\cdot\mathbb{P}(7)\cdot\sum_{m=0}^\infty\left[\mathbb{P}(\text{not 7 or 8})\right]^m\cdot\mathbb{P}(6)\\
&\cdot\sum_{n=0}^\infty\left[\mathbb{P}(\text{not 6, 7, or 8})\right]^n\cdot\mathbb{P}(8)\\
=&\sum_{\ell=0}^\infty\left(\frac{25}{36}\right)^\ell\cdot\frac{6}{36}\cdot\sum_{m=0}^\infty\left(\frac{25}{36}\right)^m\cdot\frac{5}{36}\cdot\sum_{n=0}^\infty\left(\frac{20}{36}\right)^n\cdot\frac{5}{36}\\
=&\frac{1}{11/36}\cdot\frac{6}{36}\cdot\frac{1}{11/36}\cdot\frac{5}{36}\cdot\frac{1}{16/36}\cdot\frac{5}{36}\\
=&\frac{6}{11}\cdot\frac{5}{11}\cdot\frac{5}{16}=\frac{150}{1936}.
\end{aligned}$$
The probabilities of $R_{6,7,8}$ and $R_{7,6,8}$ are the reverse of those stated by the book. Of course, the book does not actually define $R_{6,7,8}$ and $R_{7,6,8}$ (or $R_6,$ $R_7,$ $R_8$), at least not in the excerpt provided. So maybe under some interpretation, the book's method is correct.
Best Answer
Your numerator and denominator are both correct. You can cancel $7\choose 2$ and $\left( \frac16 \right)^7$ top and bottom, leaving $\frac{5!}{5^5}=\frac{4!}{5^4}$