There is no simplified description of the Nash equilibrium of this game.
You can compute the best strategy starting from positions where both players are about to win and going backwards from there.
Let $p(Y,O,P)$ the probability that you win if you are at the situation $(Y,O,P)$ and if you make the best choices. The difficulty is that to compute the strategy and probability to win at some situation $(Y,O,P)$, you make your choice depending on the probability $p(O,Y,0)$. So you have a (piecewise affine and contracting) decreasing function $F_{(Y,O,P)}$ such that $p(Y,O,P) = F_{(Y,O,P)}(p(O,Y,0))$, and in particular, you need to find the fixpoint of the composition $F_{(Y,O,0)} \circ F_{(O,Y,0)}$ in order to find the real $p(O,Y,0)$, and deduce everything from there.
After computing this for a 100 points game and some inspecting, there is no function $g(Y,O)$ such that the strategy simplifies to "stop if you accumulated $g(Y,O)$ points or more". For example, at $Y=61,O=62$,you should stop when you have exactly $20$ or $21$ points, and continue otherwise.
If you let $g(Y,O)$ be the smallest number of points $P$ such that you should stop at $(Y,O,P)$, then $g$ does not look very nice at all. It is not monotonous and does strange things, except in the region where you should just keep playing until you lose or win in $1$ move.
Your answers for $1$ and $2$ are right.
For $3$, say you reroll if you roll at most $x$. Then your expected payoff is
$$
E=\frac x{100}(E-1)+\left(1-\frac x{100}\right)\frac{x+1+100}2\;,
$$
and solving for $E$ yields
$$
E=\frac{10100-3x-x^2}{2(100-x)}\;.
$$
Then setting the derivative with respect to $x$ to zero yields
$$
10100-3x-x^2=(2x+3)(100-x)
$$
with solution $x=10(10\pm\sqrt2)$. Only the smaller solution is feasible, $x=10(10-\sqrt2)\approx85.86$, and substituting the two adjacent integers, $85$ and $86$, into the expression for $E$ yields $\frac{262}3\approx87.33$ for $x=85$ and $\frac{1223}{14}\approx 87.36$ for $x=86$.
Thus a maximal profit of about $\$87.36$ is achieved if you reroll whenever you have at most $86$.
Best Answer
The probability that you are still in game after the n-th game is just $(\frac 56)^n$. The average amount of money per non losing game is $(1+2+3+4+5)/5=3$. Hence after $n$ games you are stick with an amount of Money $M(n)$: \begin{equation} M(n)= 3 \cdot n \cdot\left( \frac 56 \right)^n \end{equation}
Now you can maximise this function and get a maximum at $n=\frac{1}{\log(\frac 65)}\approx 5.48$.
So you know that either 5 or 6 is the best number of games. Calculating M(5) and M(6) yields the result that the expectation value of money after 5 or 6 games is acutally the same, namely:
\begin{equation} M(5) = M(6) = \frac{15625}{2592}\approx 6.03 \end{equation}
Edit: (due to other posts) When you however can use the knowledge of your current money, you could do better. You should just stop playing, when in average you lose more money than you win. This happens when the expectation value gets negative. Assume now, that you have already won $N$ coins. You lose per with the probability $\frac 16 \;\; N$ coins. In the same move you win 3 coins with probability $\frac 56$. Hence they are equal, when $\frac N6 = 3\cdot \frac 56$. This yields $N=15$. So you have to stop playing (or could play 1 more time; it doesn't matter) when you reached 15 coins.