Your calculation is correct.
I'd do it this way.
Let $E_n$ be the expected net if we roll until we get $n$ or greater.
Suppose $n=6$. We roll once. If it's a $6$, we are done. If not, we are in exactly the situation we started in, except that our "first" roll isn't free: it cost $1$ dollar. Hence
$$ E_6 = \frac{1}{6}(6) + \frac{5}{6}(E_6-1).$$
Solving, we find $E_6=1$. (Alternatively: on average it takes $6$ rolls to get a $6$, and all but the first roll is free, so $E_6=6-5=1$.)
Similarly,
\begin{align}
E_5 &= \frac{2}{6}\left(\frac{5+6}{2} \right) + \frac{4}{6}(E_5-1) & \text{ so } &E_5=\frac{7}{2}=3.5 \\
E_4 &= \frac{3}{6} \left( \frac{4+5+6}{3} \right) + \frac{3}{6}(E_4-1) & \text{ so } &E_4=4 \\
E_3 &= \frac{4}{6} \left( \frac{3+4+5+6}{4} \right) + \frac{2}{6}(E_3-1) & \text{ so} &E_3=4 \\
E_2 &= \frac{5}{6} \left( \frac{2+3+4+5+6}{5} \right) +\frac{1}{6}(E_2-1) & \text{ so } &E_2=\frac{19}{5} = 3.8 \\
\end{align}
and, of course, $E_1=3.5$.
So the best strategy appears to be "roll until $4$ or more" with an expected net of $4$. Sometimes you will roll, say, 10 times with this strategy and lose money, but the average net is $4$.
Here is the result of $10^6$ simulated plays with the $4$ strategy:
net/number of occurrences
-21 1
-20 0
-19 0
-18 0
-17 0
-16 1
-15 0
-14 1
-13 5
-12 6
-11 8
-10 19
-9 32
-8 80
-7 146
-6 293
-5 579
-4 1122
-3 2272
-2 4517
-1 9147
0 18256
1 36474
2 72912
3 145747
4 291215
5 250332
6 166835
with an average net of $4.000515$.
Both approaches look correct! The $5+{}$in the first equation does account for the winnings you get to keep.
The series you found in the second case is
$$
\frac52 \sum_{k=1}^\infty \bigg( \frac12 \bigg)^k k.
$$
This is indeed a notch more complicated than a simple geometric series, but we can still handle it! Starting from
$$
\sum_{k=0}^\infty x^k = \frac1{1-x},
$$
valid for $|x|<1$, we can take the derivative of both sides to get
$$
\sum_{k=0}^\infty kx^{k-1} = \frac1{(1-x)^2},
$$
which implies
$$
c\sum_{k=1}^\infty kx^k = \frac{cx}{(1-x)^2}.
$$
This is exactly the series you want, with $c=\frac52$ and $x=\frac12$, and therefore your series equals
$$
\frac{(5/2)(1/2)}{(1-1/2)^2} = 5,
$$
which is the correct expected value.
Best Answer
Let's suppose we have only 1 roll. What is the expected payoff? Each roll is equally likely, so it will show $1,2,3,4,5,6$ with equal probability. Thus their average of $3.5$ is the expected payoff.
Now let's suppose we have 2 rolls. If on the first roll, I roll a $6$, I would not continue. The next throw would only maintain my winnings of $6$ (with $1/6$ chance) or make me lose. Similarly, if I threw a $5$ or a $4$ on the first roll, I would not continue, because my expected payoff on the last throw would be a $3.5$. However, if I threw a $1,2$ of $3$, I would take that second round. This is again because I expect to win $3.5$.
So in the 2 roll game, if I roll a $4,5,6$, I keep those rolls, but if I throw a $1,2,3$, I reroll. Thus I have a $1/2$ chance of keeping a $4,5,6$, or a $1/2$ chance of rerolling. Rerolling has an expected return of $3.5$. As the $4,5,6$ are equally likely, rolling a $4,5$ or $6$ has expected return $5$. Thus my expected payout on 2 rolls is $.5(5) + .5(3.5) = 4.25$.
Now we go to the 3 roll game. If I roll a $5$ or $6$, I keep my roll. But now, even a $4$ is undesirable, because by rerolling, I'd be playing the 2 roll game, which has expected payout of $4.25$. So now the expected payout is $\frac{1}{3}(5.5) + \frac{2}{3}(4.25) = 4.\overline{66}$.
Does that make sense?