I will suppose, that at least one of $a, b, c$ is at least $k$, otherwise it is impossible to have $k$ balls of the same color. Then the answer is $\min(a, k - 1) + min(b, k - 1) + min(c, k - 1) + 1$. You can think of it the follwing way: in the worst case one can take $min(a, k - 1)$ red balls, $min(b, k - 1)$ green balls and $min(c, k - 1)$ blue balls and still not have k balls of the same color. Then adding any ball will guarantee $k$ balls of the same color.
It is absolutely irrelevant to take $13!$ or $\frac{13!}{5!8!}$ events. You can either consider all permutations of balls of the same color as different or not. This does not effect on the ratio since both coefficients $5!8!$ in denominator and numerator are cancelled.
And what is relevant is how you calculates number of events where Jack picks blue ball. How the probability that first $8$ balls are red relates to this? Yes, if the first $8$ balls are red, then Jack cannot win. But this is not the unique case when he lost. If the first drawn ball is blue, he also lost. If the first two balls are red and the third is blue, Jack also lost. And so on.
First Jim picks ball. He must pick red ball ($8$ variants). Next Jack can pick blue or red. If he picked blue ($5$ variants) then the game is over and the other permutations of $5+8-1-1=11$ balls are irrelevant. If we will not distinguish the balls of the same color, the number of elementary events which correspond to the case when Jack wins by his first step is
$$
\frac{\color{red}8\cdot \color{blue}5\cdot 11!}{5!8!}.
$$
If Jack picked red ($7$ variants), then Jim should also pick red ($6$ variants) and then Jack can pick blue or red. If he picked blue ($5$ variants), then the game is over and the potential $5+8-4=9$ balls can permute in any order, so if we will not distinguish the balls of the same color, the number of elementary events which correspond to the case when Jack wins by his second step is
$$
\frac{\color{red}8\cdot \color{red}7\cdot \color{red}6\cdot \color{blue}5\cdot 9!}{5!8!}.
$$
Next, there are
$$
\frac{\color{red}8\cdot \color{red}7\cdot \color{red}6\cdot \color{red}5 \cdot \color{red}4 \cdot \color{blue}5\cdot 7!}{5!8!}
$$
variants where Jack wins by its third step. And there are the
$$
\frac{\color{red}8\cdot \color{red}7\cdot \color{red}6\cdot \color{red}5 \cdot \color{red}4 \cdot \color{red} 3 \cdot \color{red}2 \cdot \color{blue}5\cdot 5!}{5!8!}
$$
variants to win by fourth step of Jack.
All the other variants are impossible since there is not enough amount of red balls.
Best Answer
Apparently you have still not been taught the formula for dealing with permutations of objects when some of them are identical.
You created permutations by multiplication, you can remove them by division, so if you have, say, $8$ objects where $5$ are $A's$ and $3$ are $B's$,
the formula will be $\Large\frac{8!}{5!3!}$
So can you apply the correct formula to your specific problem ?
Read more about it here