If the univariate hypergeometric is your only tool you have to get it into something where you have two classes.
One approach (not the only one):
Break the total up as follows --
Draw 2 non-blue + draw 3 non-blue + ... + draw 5 non-blue.
Then work out the probability under each case; e.g. the first one is:
$P(\text{two non-blue balls in 5 draws}) \times$
$\hspace{0.5 cm} P(\text{exactly one red} |$ $\hspace{ 3cm} \text{two balls that are either green or red drawn from the original pool})$
So for the second part, you're essentially drawing two balls from (2 red, 3 green) and working out the probability of exactly 1 red. So it should be the product of two hypergeometric probabilities.
The second term would be
$P(\text{three non-blue balls in 5 draws}) \times$
$\hspace{0.5 cm} [P(\text{exactly one red} |$ $\hspace{ 3cm} \text{three balls that are either green or red drawn from the original pool})$ $\hspace{0.3cm}+P(\text{exactly two red} |$ $\hspace{ 3cm} \text{three balls that are either green or red drawn from the original pool})]$
This is the sum of two hypergeometric probabilities, times a hypergeometric probability; however, you can write it as the difference of two phyper calls (which doesn't save anything in this term, but will on the remaining ones.
So the overall thing would be a sum of a vector of terms of the form dhyper(...)*(phyper(...)-phyper(...)). Note that you should be able to do the whole thing with a single call of dhyper and two phyper calls (since you can pass vector arguments).
where the dhyper call covers the "draw $i$ non-blue balls" and the difference of phyper terms covers the range of how many reds are drawn out of $i$.
If you do have a multivariate hypergeometric pmf, you should be able to write it as a sum of terms.
You could also approach it in terms of
$P(\bar{B}\geq 2)*
[1-P(R=0|\bar{B}\geq 2)-P(G=0|\bar{B}\geq 2)+
P(R=0,G=0|\bar{B}\geq 2)]$
This, too, will involve a sum of terms, but you can generate these using vector arguments as well.
I may come back and try to make this answer more broadly useful.
Best Answer
$\max(0,k-r)$ describes the smallest possible number of blue marbles you draw. Clearly you can't have fewer than $0$ blue marbles, but if there are fewer red marbles available to be drawn in the bag than the number of marbles you draw, you can't have $0$ blue marbles (e.g. if there are two red marbles and you draw 3 marbles, at least one must be blue). So if $k-r$ is positive, $X$ can't be smaller than that.
Consequently whichever is the bigger of $0$ and $k-r$ is the smallest $X$ can be. Now $\max$ is the maximum function, which just returns the biggest of its arguments, so $\max(0,k-r)$ is simply how you express that lower limit mathematically.
For the upper limit, the number of blue marbles you draw cannot exceed the number of marbles you draw, nor the number of blue marbles that are there to be drawn. That is, the most $X$ can be is the smaller of $k$ and $b$, which you write as $\min(k,b)$.