Your calculation represents the probability of getting the red ball ten times in a row with replacement: this is a totally different problem. It should be easy to see that it is false by how small it is: the probability of getting the red in just the first pick is one out of twenty, so your chances of getting the red ball in the process of taking ten balls out should be even greater.
Here's how the reasoning with $X$ could have gone: the chance the $n$-th ball came up red is the probability the other $n-1$ before it came up nonred and then you got the red ball out of the remaining ones (the latter colored red in the calculations below). The probability the first pick was nonred is $19/20$, the probability the second pick was nonred is $18/20$, and so on, so we get the pattern
- $P(X=1)=\color{Red}{\frac{1}{20}}=1/20$
- $P(X=2)=\frac{19}{20}\times\color{Red}{\frac{1}{19}}=1/20$
- $P(X=3)=\frac{19}{20}\frac{18}{19}\times\color{Red}{\frac{1}{18}}=1/20$
- $P(X=4)=\frac{19}{20}\frac{18}{19}\frac{17}{18}\times\color{Red}{\frac{1}{17}}=1/20$
- $\qquad\qquad\cdots\cdots\cdots\cdots\cdots\cdots$
- $P(X=10)=\frac{19}{20}\frac{18}{19}\cdots\frac{11}{12}\times\color{Red}{\frac{1}{11}}=1/20$
Hence $P(X=1)+P(X=2)+\cdots+P(X=10)=\frac{1}{20}+\cdots\frac{1}{20}=10(\frac{1}{20})=1/2.$
Alternatively, we could have reasoned with symmetry as follows: every way of picking twenty balls out of twenty balls and keeping track of what order they come out will be like putting the balls in some order. The probability a specific ball will end up in a specific position is equal to the same probability for any other specific ball, say $p$. Since at least one ball must be in the position we have that the sum of each ball's probability being there equals one, or $20p=1$, hence $p=1/20$. Sum this over the first ten positions for the red ball and you get $10(1/20)=1/2$ as the probability the red ball is in the first ten positions. Since it makes no difference whether we actually take out the last ten balls or keep them in the box, this must be our desired value.
Two quicker ways could have been the following:
Symmetry: For every way of taking 10 balls out of 20 and leaving the other 10 in the box, there is exactly one way of taking those other 10 out of the box and leaving the original 10 in, hence by symmetry the probability of choosing the red ball equals the probability of not choosing the red ball. Thus the probability is $1/2$.
Counting: The number of ways of getting a red ball in a choice of 10 out of 20 is equal to the number of ways the other 9 non-red balls can be chosen out of the total 19 non-red balls: in other words, $19\choose9$. And the total number of ways of picking 10 balls out of 20 is $20\choose10$. So the probability of getting the red ball is $$\frac{{19\choose9}}{{20\choose10}}=\frac{19!}{9!10!} \frac{10!10!}{20!}=\frac{10}{20}=\frac{1}{2}.$$
Having these two techniques in your probability toolkit is important because problems that can be solved easily with them are ubiquitous in combinatorial situations. However it is also important that you don't split a probability problem into a large number of smaller cases (originally, we split ours up into 10 different cases) when doing so isn't actually necessary - this saves a lot of energy.
Question 1: It is easiest to prove this by induction. If $r = 0$ (there are no red balls) and $b \geq 1$, then clearly the first ball will be blue. That is, the number of balls $N$ that we need to draw in order to draw a blue ball has expected value $E(N) = 1$. Note that we can write this as
$$
E(N) = \frac{b+1}{b+1} = 1
$$
in accordance with the desired formula. Now, consider any other value of $r$, and $b \geq 1$ still. Suppose that we already have shown that for $r-1$ red balls and $b$ blue balls, the expected number of balls until we draw a blue ball is $\frac{b+r}{b+1}$, in accordance with the desired formula. We now draw a ball from an urn with $r$ red balls and $b$ blue balls. With probability $\frac{b}{b+r}$, the first ball is blue, and $N = 1$. With probability $\frac{r}{b+r}$, the first ball is red, and we now have an urn with $r-1$ red balls and $b$ blue balls. By the premise, we already know what the expected number of additional balls that need to be drawn in order to produce a blue ball; it is $\frac{b+r}{b+1}$. Therefore, for the urn with $r$ red balls and $b$ blue balls, we have
$$
\begin{align}
E(N) & = \frac{b}{b+r} \cdot 1
+ \frac{r}{b+r} \cdot \left(1+\frac{b+r}{b+1}\right) \\
& = \frac{b}{b+r} + \frac{r}{b+r} + \frac{r}{b+1} \\
& = 1 + \frac{r}{b+1} = \frac{b+r+1}{b+1}
\end{align}
$$
and we are done.
Question 2: By symmetry, the expected number of remaining balls left when only balls of one color (either red or blue) remain is equal to the number of balls of the same color we draw at the beginning. (To see this, draw all the balls out from the urn, and lay them out on a table, in sequence. The sequence from left to right is exactly as probable as the sequence from right to left.)
With probability $\frac{r}{b+r}$, the first ball is red. We are then left with $r-1$ red balls and $b$ blue balls, and we want to know how many balls we draw before (not until) we draw the first blue ball. This is one less than the result from Question 1. To this we have to add the first red ball. So when the first ball is red, our answer is
$$
N_{red} = \frac{b+r}{b+1}-1+1 = \frac{b+r}{b+1}
$$
With probability $\frac{b}{b+r}$, the first ball is blue, and by symmetry, the number of consecutive blue balls at the start is
$$
N_{blue} = \frac{b+r}{r+1}
$$
Thus, the expected number of consecutive balls drawn of the same color (which is also the answer to the original Question 2) is
$$
E(N) = \frac{r}{b+r} \cdot N_{red} + \frac{b}{b+r} \cdot N_{blue}
= \frac{r}{b+1} + \frac{b}{r+1}
$$
Best Answer
In each bet, you either obtain $k\cdot r$ dollars when you pick the red ball, or you lose $r$ dollars if you pick the blue ball. So, using your notation, the expected payout value from a single bet is: $$ P[red]\cdot (k\cdot r) + P[blue]\cdot (-r) = \frac{n\cdot k\cdot r}{m+n} - \frac{m\cdot r}{m+n} = \frac{r}{m+n}(n\cdot k-m) $$
This is the expected payout, i.e. the average payout if you repeat the bet a number of times. For the bet to be profitable to the better, the expected payout must be positive - so you will on average earn money instead of losing it. So we just need to solve an inequality for $k$: $$ \frac{r}{m+n}(n\cdot k-m) > 0 $$ $$ n\cdot k - m > 0 $$ $$ k > \frac{m}{n} $$ We of course assume that $r$ and $n$ are positive.