The number of ways of placing $n$ labelled balls in $k$ unlabelled boxes with repetition allowed is $\left\{n\atop k\right\}$, a Stirling number of the second kind. There is a rather ugly explicit formula for them:
$$\left\{n\atop k\right\}=\frac1{k!}\sum_i(-1)^{k-i}\binom{k}ii^n\;.$$
They also satisfy a rather nice Pascal-like recurrence:
$$\left\{n\atop k\right\}=k\left\{n-1\atop k\right\}+\left\{n-1\atop{k-1}\right\}\;,$$
with initial condition $$\left\{n\atop 0\right\}=\left\{0\atop n\right\}=[n=0]\;,$$
where $[n=0]$ is an Iverson bracket. The explanation here of the recurrence is concise but reasonably clear.
If the balls are also unlabelled, you are in effect looking at partitions of $n$ into $k$ parts. The number of such partitions is sometimes denoted by $P(n,k)$ and satisfies the recurrence
$$P(n,k)=P(n-1,k-1)+P(n-k,k)$$
with initial conditions $P(n,k)=0$ for $k>n$, $P(n,n)=1$, and $P(n,0)=[n=0]$. The triangle of these numbers is OEIS A008284, and you’ll find more information and references there.
If you don’t allow repetition, clearly you must have $n\le k$, and since you can’t tell one box from another, there is only one way to distribute the balls, whether they’re labelled or not: $n$ boxes each containing a ball, and $n-k$ empty boxes.
If the balls and boxes are labelled, there are $\binom53$ ways to choose the three non-empty boxes. Suppose that we’ve chosen the three boxes that are to be non-empty. There are $3^5$ ways to put the $5$ balls into those boxes, but some of those ways leave one or more of the three boxes empty. To correct for this we use an inclusion-exclusion argument.
Call the three chosen boxes Box 1, Box 2, and Box 3. There are $2^5$ ways to put the $5$ balls into Boxes 2 and 3, so that Box 1 remains empty. Similarly, there are $2^5$ ways to put the balls into Boxes 1 and 3, and $2^5$ ways to put the balls into Boxes 2 and 3. Thus, out of the $3^5$ arrangements of the balls in the three boxes, $3\cdot2^5$ leave a box empty and should be subtracted from the total to leave $3^5-3\cdot2^5$. But the arrangement that puts all $5$ balls into Box 3 got removed twice, since it leaves both Box 1 and Box 2 empty, and the same goes for the other two arrangements that put all $5$ balls into one of the three boxes. These should have been deducted only once from the original $3^5$, so we must add the $3$ back in, and the final count is
$$3^5-3\cdot2^5+3=243-96+3=150\;.$$
If the balls and boxes are both unlabelled, we simply want the number of partitions of $5$ into $3$ parts. This problem is small enough to be done by brute force: there are just two, $5=3+1+1$ and $5=2+2+1$.
Best Answer
I will assume that you know how to count the number of ways to distribute $n$ identical candies among $r$ children. Compute this number for the blue balls, call it $B$. Compute this number for the red balls. Call it $R$. For every way of distributing the blue balls, there are $R$ ways to distribute the red balls, so our total is $BR$.
If you do not know how to solve the general $n$ candy $r$ child problem, I suggest you look at the Wikipedia article on Stars and Bars. The answer turns out to be $\binom{n+r-1}{r-1}$. The reasoning is well-described in the Wikipedia article, and in several MSE answers.
For our particular very small numbers, $B$ and $R$ can also be computed by making careful lists.