In your equation, if you solve for $a$ you get:
$$a = \frac{\sqrt{M^2-60b^2}}{3} - 4b$$
If we're looking for integer solutions then this means that $M^2-60b^2$ must be a square. Also it must be divisible by $3$, so at least you only need search over (pseudo-)triples of the form: $$9x^2 + 60b^2 = M^2$$
Moreover you want $a$ to be positive so you require $x \ge 4b$. There is only one primitive triple with this property which yields $\{a,b,M\} = \{2,3,48\}$. Any multiple of this triple would also work ($\{4,6,96\}, \{6,9,144\}, \ldots$).
To solve this without (much) algebra:
Note that green bags must hold $\frac 14$ of the balls (as we divided them, by color, in the ratio 3:1). Thus there are $14$ balls in the green bags, hence $42$ in the red bags.
Now we ask: how many might be in each bag? Well, whatever that number is it must divide both $14$ and $42$, hence it is a common factor of those two numbers. But we can list those common factors: $\{1,2,7,14\}$. Each of these gives us a solution!
I. 1 ball in each of 14 green and 42 red bags.
II. 2 balls in each of 7 green and 21 red bags.
III. 7 balls in each of 2 green and 6 red bags.
IV. 14 balls in each of 1 green and 3 red bags. (Unless you interpret the problem as requiring plural green bags.)
Unless we are told more information about the size of the party we really can't choose amongst these (though at a guess, we are after case II....I is too many and III and IV are too few).
To be clear: I would not assign this to a fifth grader unless I mentioned that there might be several acceptable answers (and probably not even then).
Best Answer
These kinds of problems are special because we're only looking for one solution, so that we can make educated guesses to stumble upon one.
In a magic square, 1 and the maximum number are usually paired. But in this puzzle, unlike a magic square, every number is included in exactly two sums. This gives us freedom we wouldn't normally have. We can pair 1 with 12 on the outside, but because of this freedom, we can put the next-most-extreme numbers together, to give us a place to start from: $$\begin{array}{ccccccc} & & & 1\\ ? & & ? & & 2 & & ?\\ & ? & & & & 11\\ ? & & ? & & ? & & 12\\ & & & ? \end{array}$$
There is one undetermined number in a sum with 11 and a sum with 12. That can probably be made as small as possible: 3. (We could have done this symmetrically with 1 and 2 needing a big number, but probably not both at the same time.)
$$\begin{array}{ccccccc} & & & 1\\ ? & & ? & & 2 & & ?\\ & ? & & & & 11\\ ? & & ? & & 3 & & 12\\ & & & ? \end{array}$$
To complete $3+12$, we need a sum of $11$, which we could only get as $6+5$ or $7+4$. To complete $3+11$, we need $5+7$ or $4+8$. Therefore, we can't pick $7+4$ because it would wipe out both possibilities to complete the diagonal (this feels like kakuro). As such, the row needs a 6 and a 5. Which should go in the bottom left corner? Probably the middling value 6 since it's in a sum with 1 and a sum with 12:
$$\begin{array}{ccccccc} & & & 1\\ ? & & ? & & 2 & & ?\\ & ? & & & & 11\\ 6 & & 5 & & 3 & & 12\\ & & & ? \end{array}$$
This leaves $4+8$ for the diagonal on the right. But putting 4 in the top right would make the top row have no options since the numbers would have to be too big: $$\begin{array}{ccccccc} & & & 1\\ ? & & ? & & 2 & & 8\\ & ? & & & & 11\\ 6 & & 5 & & 3 & & 12\\ & & & 4 \end{array}$$ The only pair for the top row is 7,9 and the only pair for the upper diagonal is 9 and 10, so 9 must go in the intersection: $$\begin{array}{ccccccc} & & & 1\\ 7 & & 9 & & 2 & & 8\\ & 10 & & & & 11\\ 6 & & 5 & & 3 & & 12\\ & & & 4 \end{array}$$
As made clear in How to solve this system of linear equations, this is just one of many other solutions.