I cannot prove that $P(win) \approx {1 \over \sqrt{n}}$, but here is a proof that:
Claim: $P(win) \le {2 \over \sqrt{n}}$
which of course implies $P(win) \to 0$, answering the OP question.
Proof: For convenience, let $G =$ the OP's original game. Consider some large, fixed $n$. Define:
We will separately bound $P(A), P(B)$ both $\le {1 \over \sqrt{n}}$. The main claim then follows because $P(win) = P(A)+P(B)$.
Lemma: $P(A) \le {1\over \sqrt{n}}$: Imagine a modified game $G'$ where the die is always rolled for all $n$ rounds, and the game result is the first winning or losing roll. This modified game $G'$ is clearly equivalent to the OP's truncated version $G$.
Let random variable $X=$ no. of winning rolls among the initial $\sqrt{n}$ rounds of $G'$. To win on or before $\sqrt{n}$ rounds, it is necessary (though not sufficient) that $X\ge 1$, i.e. $P(A) \le P(X\ge 1)$. Meanwhile, for any non-negative integer r.v.s, we have:
$$P(X\ge 1) = \sum_{k=1}^\infty P(X=k) \le \sum_{k=1}^\infty kP(X=k) = E[X]$$
In this case, by linearity, $E[X] = {1\over n} \sqrt{n} = {1 \over \sqrt{n}}.$ Combining, we have:
$$P(A) \le P(X\ge 1) \le E[X] = {1 \over \sqrt{n}} \;\;\; \square$$
Lemma: $P(B) \le {1 \over \sqrt{n}}$: First, define:
- event $E= $ game $G$ is inconclusive (neither won nor lost) after $\sqrt{n}$ rounds
Note that event $B$ requires event $E$, i.e. $P(B) = P(B \cap E) = P(E) P(B \mid E)$.
Now imagine two different modified games. In $G_1$, the game starts with $1$ to $\sqrt{n}$ as the losing numbers and adds one more losing number every round. By construction, $P(\text{win }G_1) = P(B \mid E)$.
In $G_2$, the game starts with $1$ to $\sqrt{n}$ as the losing numbers and the set of losing numbers doesn't change for the rest of the game. Clearly, $G_2$ is easier to win than $G_1$ (e.g. via a sample-point by sample-point dominance argument).
The modified game $G_2$ is not limited to $n$ rounds, but it does terminate with probability $1$. Therefore, the ratio:
$${P(\text{win }G_2) \over P(\text{lose }G_2)} = {P(\text{win }G_2 \mid G_2 \text{ terminates}) \over P(\text{lose }G_2 \mid G_2 \text{ terminates})} = {\text{no. of winning rolls} \over \text{no. of losing rolls}} = {1 \over \sqrt{n}}$$
The easiest proof of the above is to consider $G_2$ as a $3$-state Markov chain with $2$ absorbing states, for win and loss. Alternately one can consider there to be $1+\sqrt{n}$ terminating states, by symmetry all equally likely, and then color exactly $1$ of them as "winning" and the others as "losing".
Combining everything, we have:
$$P(B) = P(E) P(B \mid E) \le P(B \mid E) = P(\text{win }G_1) \le P(\text{win }G_2) = {1 \over 1 + \sqrt{n}} < {1 \over \sqrt{n}} \;\;\; \square$$
Your case is easier because you do not have to track all the rolls to wait for pairs which might appear. Indeed, every roll is independent of the previous. For a fixed number of rolls you could write this down as a tree (win, continue, lose). The win or lose branches stop the game, the "continue" branch again branches in (win, continue, lose) with the same probabilities.
Starting with the second question: This shows directly that the number of rolls until the game ends (either win or lose) is geometrically distributed with parameter $p=\frac{3}{36}+\frac{6}{36}=\frac{1}{4}$. The expected value of the geometric distribution is just $p^{-1}$.
Concerning the first question: To win after $n$ rolls you need to roll anything but a 7 or a 10 exactly $n-1$ times and then roll a 10. So
$$\mathbb{P}[\text{win after n rolls}] = \left(1-\frac{1}{4}\right)^{n-1}\frac{3}{36}$$
and therefore
$$\mathbb{P}[\text{win}]= \mathbb{P}[\exists n\in\mathbb{N}:\text{win after n rolls}]=\sum_{n=1}^{\infty}\mathbb{P}[\text{win after n rolls}]=\sum_{n=1}^{\infty} \left(1-\frac{1}{4}\right)^{n-1}\frac{3}{36}=\frac{12}{36}=\dfrac{1}{3}.$$
I hope that helps.
Best Answer
The probability of winning directly is, as you calculated, $8/36$, and the probability of losing directly is $(1+2+1)/36=4/36$.
For the remaining cases, you need to sum over all remaining rolls. Let $p$ be the probability of rolling your initial roll, and $q=6/36=1/6$ the probability of rolling a $7$. Then the probability of rolling your initial roll before rolling a $7$ is $p/(p+q)$, and the probability of rolling a $7$ before rolling your initial roll is $q/(p+q)$. Thus, taking into account the probability of initially rolling that roll, each roll that doesn't win or lose directly yields a contribution $p^2/(p+q)$ to your winning probability.
For $p=5/36$, that's
$$ \frac{\left(\frac5{36}\right)^2}{\frac{5+6}{36}}=\frac{25}{11\cdot36}\;, $$
and likewise $16/(10\cdot36)$ and $9/(9\cdot36)$ for $p=4/36$ and $p=3/36$, respectively. Each of those cases occurs twice (once above $7$ and once below), so your overall winning probability is
$$ \frac8{36}+\frac2{36}\left(\frac{25}{11}+\frac{16}{10}+\frac99\right)=\frac{244}{495} = \frac12-\frac7{990}\approx\frac12-0.007\;. $$