Let $v$ denote the expected value of the game. If you roll some $x\in\{1,\ldots,100\}$, you have two options:
- Keep the $x$ dollars.
- Pay the \$$1$ continuation fee and spin the dice once again. The expected value of the next roll is $v$. Thus, the net expected value of this option turns out to be $v-1$ dollars.
You choose one of these two options based on whichever provides you with a higher gain. Therefore, if you spun $x$, your payoff is $\max\{x,v-1\}$.
Now, the expected value of the game, $v$, is given as the expected value of these payoffs:
\begin{align*}
v=\frac{1}{100}\sum_{x=1}^{100}\max\{x,v-1\}\tag{$\star$},
\end{align*}
since each $x$ has a probability of $1/100$ and given a roll of $x$, your payoff is exactly $\max\{x,v-1\}$. This equation is not straightforward to solve. The right-hand side sums up those $x$ values for which $x>v-1$, and for all such values of $x$ that $x\leq v-1$, you add $v-1$ to the sum. This pair of summations gives you $v$. The problem is that you don't know where to separate the two summations, since the threshold value based on $v-1$ is exactly what you need to compute. This threshold value can be guessed using a numerical computation, based on which one can confirm the value of $v$ rigorously. This turns out to be $v=87\frac{5}{14}$.
Incidentally, this solution also reveals that you should keep rolling the dice for a $1 fee as long as you roll 86 or less, and accept any amount 87 or more.
ADDED$\phantom{-}$In response to a comment, let me add further details on the computation. Solving for the equation ($\star$) is complicated by the possibility that the solution may not be an integer (indeed, ultimately it is not). As explained above, however, ($\star$) can be rewritten in the following way:
\begin{align*}
v=\frac{1}{100}\left[\sum_{x=1}^{\lfloor v\rfloor-1}(v-1)+\sum_{x=\lfloor v\rfloor}^{100}x\right],\tag{$\star\star$}
\end{align*}
where $\lfloor\cdot\rfloor$ is the floor function (rounding down to the nearest integer; for example: $\lfloor1\mathord.356\rfloor=1$; $\lfloor23\mathord.999\rfloor=23$; $\lfloor24\rfloor=24$). Now let’s pretend for a moment that $v$ is an integer, so that we can obtain the following equation:
\begin{align*}
v=\frac{1}{100}\left[\sum_{x=1}^{v-1}(v-1)+\sum_{x=v}^{100}x\right].
\end{align*}
It is algebraically tedious yet conceptually not difficult to show that this is a quadratic equation with roots
\begin{align*}
v\in\left\{\frac{203\pm3\sqrt{89}}{2}\right\}.
\end{align*}
The larger root exceeds $100$, so we can disregard it, and the smaller root is approximately $87\mathord.349$. Of course, this is not a solution to ($\star\star$) (remember, we pretended that the solution was an integer, and the result of $87\mathord.349$ does not conform to that assumption), but this should give us a pretty good idea about the approximate value of $v$. In particular, this helps us formulate the conjecture that $\lfloor v\rfloor=87$. Upon substituting this conjectured value of $\lfloor v\rfloor$ back into ($\star\star$), we now have the exact solution $v=87\frac{5}{14}$, which also confirms that our heuristic conjecture that $\lfloor v\rfloor=87$ was correct.
In the following I shall treat the continuous version of the game: The rolling of the die is modeled by the drawing of a real number uniformly distributed in $[0,1]$.
Only the bettor can have a strategy, and this strategy is completely characterized by some number $\xi\in\ ]0,1[\> $. It reads as follows: Draw once more if the current sum $x$ is $<\xi$, and stay if the current sum is $>\xi$. The problem is to find the optimal $\xi$.
Claim: When the bettor stays at some $x\in[0,1]$ his chance $q(x)$ of losing is $=(1-x)e^x$.
Proof. The bettor loses if the successive drawings of the dealer produce a partial sum $s_{n+1}$ in the interval $[x,1]$. This means that there is an $n\geq0$ with $s_n\leq x$ and $x\leq s_{n+1}\leq1$. The probability for $s_n\leq x$ is easily seen to be ${x^n\over n!}$, and the probability that the next drawing causes $s_{n+1}$ to lie in the given interval of length $1-x$ is $1-x$. It follows that
$$q(x)=\sum_{n=0}^\infty{x^n\over n!}\>(1-x)=(1-x)e^x\ .$$
From this we draw the following conclusion: If the bettor has accumulated $x$ and decides to make one additional draw then his chances of losing are
$$\tilde q(x)=\int_x^1 q(t)\>dt\>+x =(2-x)e^x+x\ ,$$
whereby the $+x$ at the end is the probability of busting.
The function $x\mapsto q(x)$ is decreasing in $[0,1]$, and $\tilde q$ is increasing in this interval, whereby $q(x)=\tilde q(x)$ when $x=\xi$ with $\xi\doteq0.570556$. When $x<\xi$ then $\tilde q(x)<q(x)$. This means that an additional drawing at $x$ decreases the probability of losing. When $x>\xi$ then the converse is true: An additional drawing would increase the probability of losing.
Best Answer
1. What is the expected value for player $A$?
Let $A, B$ be the players' rolls and $W$ be player $A$'s winnings. Then $$ \begin{align} \tag{1} \mathbb{E}W &= \mathbb{E}[W|A \leq 20]\cdot\mathbb{P}\{A \leq 20\} \ +\ \mathbb{E}[W|A > 20]\cdot\mathbb{P}\{A > 20\} \end{align} $$ just by simply conditioning on the event $\{A \leq 20\}$ so far.
Putting it all together, $$ \begin{align} \mathbb{E}W &= -\frac{21}{40}\frac{2}{3} + \frac{51}{2}\frac{1}{3} \\ &= \frac{163}{20}\\ &= 8.15 \end{align} $$ as was derived in previous answers. So far so good.
2. How does the expected value of the game for player $A$ change when player $B$ can re-roll?
Here the wording becomes rather more vague. Nonetheless I believe both previous answers misinterpret the problem statement. Allowing player $B$ a re-roll does not mean $B$ chooses the maximum of two rolls: that would be worded as "player $B$ is allowed the greater of two rolls". Allowing player $B$ a re-roll means that if player $B$ is unhappy with her first roll she may substitute it with a second roll, which may turn out to be less than her first. The ambiguity is in whether the re-roll is offered before or after player $B$ has seen player $A$'s roll. I consider only the first case, as it permits us to offer numbers instead of functions (of player $A$'s roll) as answers to this question, and to question 4.
Clearly, player $B$ will re-roll after rolling $b$ or less iff $$ \tag{2} \mathbb{E}[W|B=b] > \mathbb{E}[W] = 8.15 $$ as she wishes to minimize $W$. At this point we may as well calculate: $$ \begin{align} \mathbb{E}[W|B=b] &= \mathbb{E}[W|B=b, A \leq b]\cdot\mathbb{P}\{A \leq b\} \ +\ \mathbb{E}[W|B=b, A > b]\cdot\mathbb{P}\{A > b\} \\ &=-b\frac{b}{30}+\frac{(b+1)+30}{2}\frac{30-b}{30} \\ &= \frac{-3b^2-b+930}{60}. \end{align} $$ Let $b^*$ be the maximum $b$ such that inequality $(2)$ holds. (I found $b^* = 11$. Is there a faster, less error-prone method of calculating this part?)
Let $W_1$ be the value of the game when player $B$ can re-roll. Then \begin{align} \mathbb{E}[W_1] &= \mathbb{E}W + \frac{b^*}{20}\big(\mathbb{E}W - \mathbb{E}[W|B \leq b^*]\big)\\ \end{align} as we modify the old $\mathbb{E}W$ by adding the effect of the re-roll. I express it in this way because $$\mathbb{E}[W|B \leq b^*]$$ is easy to calculate quickly as it expands just like equation $(1)$ does, by pretending player $B$ is rolling a $b^*$-sided die.