In the following I shall treat the continuous version of the game: The rolling of the die is modeled by the drawing of a real number uniformly distributed in $[0,1]$.
Only the bettor can have a strategy, and this strategy is completely characterized by some number $\xi\in\ ]0,1[\> $. It reads as follows: Draw once more if the current sum $x$ is $<\xi$, and stay if the current sum is $>\xi$. The problem is to find the optimal $\xi$.
Claim: When the bettor stays at some $x\in[0,1]$ his chance $q(x)$ of losing is $=(1-x)e^x$.
Proof. The bettor loses if the successive drawings of the dealer produce a partial sum $s_{n+1}$ in the interval $[x,1]$. This means that there is an $n\geq0$ with $s_n\leq x$ and $x\leq s_{n+1}\leq1$. The probability for $s_n\leq x$ is easily seen to be ${x^n\over n!}$, and the probability that the next drawing causes $s_{n+1}$ to lie in the given interval of length $1-x$ is $1-x$. It follows that
$$q(x)=\sum_{n=0}^\infty{x^n\over n!}\>(1-x)=(1-x)e^x\ .$$
From this we draw the following conclusion: If the bettor has accumulated $x$ and decides to make one additional draw then his chances of losing are
$$\tilde q(x)=\int_x^1 q(t)\>dt\>+x =(2-x)e^x+x\ ,$$
whereby the $+x$ at the end is the probability of busting.
The function $x\mapsto q(x)$ is decreasing in $[0,1]$, and $\tilde q$ is increasing in this interval, whereby $q(x)=\tilde q(x)$ when $x=\xi$ with $\xi\doteq0.570556$. When $x<\xi$ then $\tilde q(x)<q(x)$. This means that an additional drawing at $x$ decreases the probability of losing. When $x>\xi$ then the converse is true: An additional drawing would increase the probability of losing.
Before deciding whether to stop or roll, suppose you have a non-negative integer number of points $n$.
How many more rolls should you make to maximise the expected gain over stopping (zero)?
Suppose that further number of rolls is another non-negative integer $k$. Now consider the $6^k$ possible sequences of $k$ rolls:
- In $5^k$ of those sequences there is no six and you win some points. The sum $D_k$ over all such sequences of the sum of dice rolls within each sequence satisfies the recurrence relation
$$D_0=0\quad D_{n+1}=5D_n+15\cdot5^n$$
It turns out that this has a closed form:
$$D_k=15k\cdot5^{k-1}=3k\cdot5^k$$
- In the remaining $6^k-5^k$ sequences there is at least one six and you lose the $n$ points you had beforehand.
So the expected gain when you have $n$ points and try to roll $k$ more times before stopping is
$$G(n,k)=\frac{D_k-n(6^k-5^k)}{6^k}=\frac{3k\cdot5^k-n(6^k-5^k)}{6^k}$$
For a fixed $n$, the $k$ that maximises $G(n,k)$ is $m(n)=\max(5-\lfloor n/3\rfloor,0)$; if $3\mid n$ then $k=m(n)+1$ also forms a maximum.
Suppose we fix the maximum number of rolls before starting the game. At $n=0$, $k=5$ and $k=6$ maximise $G(n,k)$ and the expected score with this strategy is
$$G(0,5)=\frac{15625}{2592}=6.028163\dots$$
But what if we roll once and then fix the maximum rolls afterwards? If we roll 1 or 2, we roll at most 5 more times; if 3, 4 or 5, 4 more times. The expected score here is higher:
$$\frac16(1+G(1,5)+2+G(2,5)+3+G(3,4)+4+G(4,4)+5+G(5,4))=6.068351\dots$$
We will get an even higher expected score if we roll twice and then set the roll limit. This implies that the greedy strategy, outlined below, is optimal:
Before the start of each new roll, calculate $m(n)$. Roll if this is positive and stop if this is zero.
When $n\ge15$, $m(n)=0$. A naïve calculation that formed the previous version of this answer says that rolling once has zero expected gain when $n=15$ and negative expected gain when $n>15$. Together, these suggest that we should stop if and when we have 15 or more points.
Finding a way to calculate the expected score under this "stop-at-15" strategy took quite a while for me to conceptualise and then program, but I managed it in the end; the program is here. The expected score works out to be
$$\frac{2893395172951}{470184984576}=6.1537379284\dots$$
So this is the maximum expected score you can achieve.
Best Answer
If you play once you get $2/3$ of a dollar. If you play twice you either get two dollars or nothing, so the expected winnings are $(2/3)^2\times2=\$8/9$. If you play three times the expected winnings are $(2/3)^3\times 3=\$8/9$. If you play four times you get $(2/3)^4\times 4=\$64/81<\$8/9=\$72/81$. So the best strategy seems to be to play twice or three times.
Let $$f(x)=x(2/3)^x=x \exp(\log(2/3)x)$$ then $${df\over dx}=(1+x\log(2/3))\exp(\log(2/3)x)$$ which has a critical point when $x=-1/\log(2/3)\approx2.5$, which turns out to be a maximum from graphing.