Added: Please see the end for a very simple way to solve the problem.
We assume that cards are removed from the deck until only face cards are left. (So a referee tells us when to stop.) We want the probability that at that time, the Ace of Hearts is still in the deck.
It is clear that if the referee lets us go through the whole deck, because the last card isn't a face card, the Ace of Hearts is gone.
There are $16$ individual cases to examine: (i) the last card is a face card, but the one before is not; (ii) the last $2$ cards are face cards, but the one before is gone; (iii) the last $3$ cards are face cards, but the one before is not. Continue in this way until case (xvi), the last $16$ cards are face cards.
Case (i): The probability that the last card is a facecard (F) is $\frac{16}{52}$. Given it is an F, the probability the previous card is NF is $\frac{36}{51}$. So the probability we end with NF, F is $\frac{16}{52}\cdot\frac{36}{51}$. Given that this has happened the probability the last card is the Ace of Hearts is $\frac{1}{16}$, for a probability of $\frac{16}{52}\cdot\frac{36}{51}\cdot\frac{1}{16}$.
Case (ii): The probablity the last card is F is $\frac{16}{52}$. Given that, the probability the previous card is an F is $\frac{15}{16}$. Given that, the probability the previous card is NF is $\frac{36}{50}$. Given all that, the probability that the Ace of Hearts is still in the deck is $\frac{2}{16}$. So our case (ii) probability is $\frac{16}{52}\cdot\frac{15}{51}\cdot\frac{36}{50}\cdot\frac{2}{16}$.
Case (iii): The probability that the last $3$ cards are $F$, but the fourth from the end is NF, is $\frac{16}{52}\cdot\frac{15}{51}\cdot\frac{14}{50}\cdot\frac{36}{49}$.
Given this has happened, the probability the Ace of Hearts is among the last $3$ is $\frac{3}{16}$. So the probability of case (iii) is $\frac{16}{52}\cdot\frac{15}{51}\cdot\frac{14}{50}\cdot\frac{36}{49}\cdot \frac{3}{16}$.
I hope the pattern is clear. One has to add up all of these. There is undoubtedly a much simpler way to view things. Certainly one can express the products above in terms of binomial coefficients. That part is not very interesting, but I think there is a way to avoid summing.
A much much simpler way: Don't worry about all the face cards except the Ace of Hearts. Confine attention to these $37$ cards.
The Ace of Hearts survives precisely if it is the last card among the $36$ non-face cards and the Ace of Hearts. Since any order of these $37$ cards is just as likely as any other order, the probability the Ace of Hearts is last among them is $\dfrac{1}{37}$.
The question is essentially solved in ยง3.2. Use the equation: $$posterior = \frac{likelihood \times \ prior}{evidence}$$ and work it through.
Assume a uniform prior - $P(f_H) = 1$.
The evidence $P(n_H| \ N)$ is the normalizing constant - $\int_0^1 df_H \ f_H^{n_H} {(1-f_H)}^{N-n_H}$.
Based on the hint about the beta integral, $\int_0^1 df_H \ f_H^{n_H} {(1-f_H)}^{N-n_H} = \frac{n_H! \ (N-n_H)!}{(N+1)!}$.
The likelihood is $P(n_H| \ f_H,N) = f_H^{n_H} {(1-f_H)}^{N-n_H}$
So the posterior probability is just the ratio of the likelihood over the evidence.
$$P(f_H | \ n_H, N) = \frac{f_H^{n_H} {(1-f_H)}^{N-n_H}}{\frac{n_H! \ (N-n_H)!}{(N+1)!}}$$
To find the probability of the $N+1^{th}$ toss, integrate over $f_H$. By the sum rule, $$P(h \ |\ n_H, N) = \int df_H P(h | \ f_H) \ P(f_H | \ n_H, N) $$
$P(h| \ f_H) = f_H$, so we have $\int_0^1 df_H\frac{f_H^{n_H+1} {(1-f_H)}^{N-n_H}}{\frac{n_H! \ (N-n_H)!}{(N+1)!}}$
Using the beta integral again, this becomes $$\frac{(n_H+1)! (N-n_H)!}{(N+2)!} \ \times \ \frac{(N+1)!}{n_H! (N-n_H)!} = \frac {n_H +1}{N+2}$$
Best Answer
In general $X_{m+n,p}$ and $Y_{n,p}$ are all in different probability spaces, but we can define versions of them on the same probability space (technically this is known as coupling), by considering an infinite sequence of independent coin-flips with probability $p$ of heads in each one.
BTW, there seems to be some confusion here, due to different conventions for the negative binomial. The definition as given in the text has $Y_{n,p}$ as the number of flips until the $n$'th head, while the question appears to use a different definition where $Y_{n,p}$ is the number of tails until the $n$'th head. With the definition as given, the correct statement is that $\text{Pr}(Y_{n,p} \le m) = \text{Pr}(X_{m,p} \ge n)$.
EDIT: Oh, I see the problem. I was looking at (8.59), and they're taking (8.60) as the definition of the negative binomial. So it is the number of tails.