The other answers are nice, so I won't reiterate them. It's worth pointing out, though, that when you say "I know there are other methods, i.e. factorisation and completing the square, but does this mean that this formula can only be used in specialised cases", you seem to be implying that these other methods might yield different results.
This isn't the case. If the quadratic formula fails to give you real-valued answers, then the other two methods (and every applicable method) will also fail, because the quadratic polynomial itself simply doesn't have any real-valued roots.
You can visualize this easily: as you may know, the graph of a quadratic polynomial is a parabola, and the real roots of a polynomial correspond to its graphs' zeros, i.e. points where the graph crosses the $x$-axis. So if you imagine a parabola with its vertex above the $x$-axis and opening upwards, you can see that it will never cross the $x$-axis -- it has no real zeros at all!
This is exactly the situation with $y = 3x^2 + x + 24$ and every other quadratic polynomial for which $b^2 - 4ac$ (the discriminant) is negative.
Why isn't taking the square root of a negative number allowed?
Lastly (since it seems as though this may be one of your first introductions to complex numbers) let me clarify something that confuses my students greatly: don't let the words "real" and "imaginary" mislead you into thinking that real numbers are somehow genuine or actual and imaginary numbers are somehow fake.
Just as a "negative" number isn't sad, cynical, or irritated, and just as an "irrational" number hasn't lost its mind, "real" and "imaginary" are technical terms, and not meant to carry with them their English connotations. Although they earned those names due to prior misunderstandings about numbers, today we know that imaginary numbers are just as actual and valid as real numbers, and in fact complex (not "complicated") numbers have huge real-world applications and consequences.
Along these same lines: you say that taking the square root of a negative number "isn't allowed". More accurate would be to say that the square root of a negative number isn't a real number ("real", again, in the technical sense) because the square of a real number is always non-negative. If you require the result of taking the square root to be a real number, then you're right, taking the square root of a negative isn't allowed. But otherwise it's perfectly fine.
By analogy: you're allowed to divide the whole number 6 by the whole number 3, and the result is the whole number 2. But if you divide the whole number 1 by the whole number 2, then the result, $\frac{1}{2}$, isn't a whole number. If you require your operations on a certain class of numbers to produce a result from that same class of numbers, then lots of operations (like division, and subtraction too, for that matter) will have situations in which they aren't "allowed" to occur.
This requirement is sometimes very desirable, and there are certainly contexts in which you will want to impose it. But you should understand that imposing this requirement is a choice that you make because it fits the context of the problems you're working on. It's not divine law. If you find yourself in a situation in which you would like to take the square root of a negative number and produce the corresponding imaginary number, you are definitely "allowed" to do so.
You and every other answer thus far have talked about starting by squaring both sides, which works, but there is a slightly easier path to take. Rewrite your equation so that there is only one square root on each side, for example: $$4\sqrt{x-3}=3+\sqrt{6x-17}$$ Now, when you square both sides, you'll still have another square root to deal with, but it's not the square root of a product, like the $\sqrt{(x-3)(6x-17)}$ you would have had: $$16(x-3)=9+6\sqrt{6x-17}+(6x-17)$$ Expand, collect like terms, and rearrange to get the square root by itself: $$5x-20=3\sqrt{6x-17}$$ Square both sides again: $$25x^2-200x+400=9(6x-17)$$ $$25x^2-254x+553=0$$ Solving this gives $$x=7\text{ or }x=\frac{79}{25}$$ but $x=\frac{79}{25}$ doesn't work in the original equation, so $x=7$ is the only solution to the original equation.
Best Answer
We can start small and improve a bit at a time. We can begin with $\frac{a^2}{8}$ which does give the top term when squared. Next, $$ \left( \frac{a^2}{8} + B a \right)^2 = \frac{a^4}{64} + B \frac{a^3}{4} + B^2 a^2. $$ To get the $a^3/8$ we take $B = 1/2.$ So far, we have $$ \left( \frac{a^2}{8} + \frac{a}{2} \right)^2 = \frac{a^4}{64} + \frac{a^3}{8} + \frac{ a^2}{4}. $$ Not bad. Next, $$ \left( \frac{a^2}{8} + \frac{a}{2} +C\right)^2 = \frac{a^4}{64} + \frac{a^3}{8} + \frac{ a^2}{4} + C\frac{a^2}{4} + C a + C^2. $$ To get rid of the $a^2$ term, we need only take $C = -1$ and get $$ \left( \frac{a^2}{8} + \frac{a}{2} -1\right)^2 = \frac{a^4}{64} + \frac{a^3}{8} - a + 1. $$