If the expected value of this game is $a$, then at a die roll of $X$ you have the choice of either collecting $X$ or paying a dollar and restart, which gives you an expected value of $a-1$.
To maximize the expected value, you should take $X$ if $X> a-1$ and start over if $X\le a-1$ (it does not really matter what we do when $X=a-1$).
We obtain therefore
$$ a = \frac1{100}\left(\lfloor a-1\rfloor\cdot a+\sum_{k=\lfloor a-1\rfloor+1}^{100}k\right)
=\frac1{100}\left(\lfloor a-1\rfloor\cdot a+\frac{100\cdot101}{2}-\frac{\lfloor a-1\rfloor \cdot\lfloor a\rfloor}{2}\right).
$$
I find numerically (didn't do much code checking, but the results are somewhat plausible)
$$a\approx87.3571 $$
which seems to be exactly (and of course the true result must be rational)
$$a=87\frac{5}{14}.$$
But I'm sure you can do the justification after the fact, i.e. show that the strategy that consists in continuing until you roll at least $87$ gives you $87\frac{5}{14}$ as expected value.
For your convenience, here is the PARI one-liner:
solve(a=1,100,sum(k=1,100,max(a-1,k))/100-a)
If an extra roll costs two dollars instead of one, the result would be
$$a=82\frac12$$
instead, and with a cost of only $0.1$ dollars it would be
$$a=96\frac1{10}.$$
Let $v$ denote the expected value of the game. If you roll some $x\in\{1,\ldots,100\}$, you have two options:
- Keep the $x$ dollars.
- Pay the \$$1$ continuation fee and spin the dice once again. The expected value of the next roll is $v$. Thus, the net expected value of this option turns out to be $v-1$ dollars.
You choose one of these two options based on whichever provides you with a higher gain. Therefore, if you spun $x$, your payoff is $\max\{x,v-1\}$.
Now, the expected value of the game, $v$, is given as the expected value of these payoffs:
\begin{align*}
v=\frac{1}{100}\sum_{x=1}^{100}\max\{x,v-1\}\tag{$\star$},
\end{align*}
since each $x$ has a probability of $1/100$ and given a roll of $x$, your payoff is exactly $\max\{x,v-1\}$. This equation is not straightforward to solve. The right-hand side sums up those $x$ values for which $x>v-1$, and for all such values of $x$ that $x\leq v-1$, you add $v-1$ to the sum. This pair of summations gives you $v$. The problem is that you don't know where to separate the two summations, since the threshold value based on $v-1$ is exactly what you need to compute. This threshold value can be guessed using a numerical computation, based on which one can confirm the value of $v$ rigorously. This turns out to be $v=87\frac{5}{14}$.
Incidentally, this solution also reveals that you should keep rolling the dice for a $1 fee as long as you roll 86 or less, and accept any amount 87 or more.
ADDED$\phantom{-}$In response to a comment, let me add further details on the computation. Solving for the equation ($\star$) is complicated by the possibility that the solution may not be an integer (indeed, ultimately it is not). As explained above, however, ($\star$) can be rewritten in the following way:
\begin{align*}
v=\frac{1}{100}\left[\sum_{x=1}^{\lfloor v\rfloor-1}(v-1)+\sum_{x=\lfloor v\rfloor}^{100}x\right],\tag{$\star\star$}
\end{align*}
where $\lfloor\cdot\rfloor$ is the floor function (rounding down to the nearest integer; for example: $\lfloor1\mathord.356\rfloor=1$; $\lfloor23\mathord.999\rfloor=23$; $\lfloor24\rfloor=24$). Now let’s pretend for a moment that $v$ is an integer, so that we can obtain the following equation:
\begin{align*}
v=\frac{1}{100}\left[\sum_{x=1}^{v-1}(v-1)+\sum_{x=v}^{100}x\right].
\end{align*}
It is algebraically tedious yet conceptually not difficult to show that this is a quadratic equation with roots
\begin{align*}
v\in\left\{\frac{203\pm3\sqrt{89}}{2}\right\}.
\end{align*}
The larger root exceeds $100$, so we can disregard it, and the smaller root is approximately $87\mathord.349$. Of course, this is not a solution to ($\star\star$) (remember, we pretended that the solution was an integer, and the result of $87\mathord.349$ does not conform to that assumption), but this should give us a pretty good idea about the approximate value of $v$. In particular, this helps us formulate the conjecture that $\lfloor v\rfloor=87$. Upon substituting this conjectured value of $\lfloor v\rfloor$ back into ($\star\star$), we now have the exact solution $v=87\frac{5}{14}$, which also confirms that our heuristic conjecture that $\lfloor v\rfloor=87$ was correct.
Best Answer
Here's a solution based on the discussion in the comments for completeness
We have $\text{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2$. We know that $\mathbb{E}[X]^2=(87+\frac{5}{14})^2\approx 7631$ (see here for a proof of this).
To get $\mathbb{E}[X^2]$, the trick is to condition on the number of rolls. Note that we can write $X=Y_1+Y_2+\dots+Y_N$ where $Y_i$ is a random variable for the amount of money made in the $i$th roll and $N$ is a random variable for how many rolls until we stop, e.g. roll an 87 or above, according to our optimal strategy (see that link above for a proof of why this is the optimal strategy). Specifically, note that $Y_1,Y_2,\dots,Y_{N-1}$ are all deterministic given $N$ (they must all be $-1$, as they represent re-rolls where each re-roll costs one dollar). However, note $N\sim\text{Geo}(\frac{14}{100})$ and conditioned on $N=n$, $Y_N\sim \text{Unif}\{88-n,89-n,90-n,\dots,101-n\}$. Thus, we can compute (could probably be done by hand, but I just plugged into a calculator)
$$\mathbb{E}[X^2]=\sum_{n=1}^{\infty}\mathbb{E}[X^2|N=n]P_N(n)=\sum_{n=1}^{\infty}\sum_{k=88-n}^{101-n}\frac{k^2}{14}\cdot \left(\frac{14}{100}\right)\cdot\left(\frac{86}{100}\right)^{n-1}\approx 7691$$
Thus we have $$\text{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X]^2\approx 7691-7631=60$$
This is consistent with a random sampling simulation.
Conclusion: With optimal play, we expect to make around $\$87.36$, with standard deviation around $\sqrt{60}\approx \$7.75$.