The other answers are nice, so I won't reiterate them. It's worth pointing out, though, that when you say "I know there are other methods, i.e. factorisation and completing the square, but does this mean that this formula can only be used in specialised cases", you seem to be implying that these other methods might yield different results.
This isn't the case. If the quadratic formula fails to give you real-valued answers, then the other two methods (and every applicable method) will also fail, because the quadratic polynomial itself simply doesn't have any real-valued roots.
You can visualize this easily: as you may know, the graph of a quadratic polynomial is a parabola, and the real roots of a polynomial correspond to its graphs' zeros, i.e. points where the graph crosses the $x$-axis. So if you imagine a parabola with its vertex above the $x$-axis and opening upwards, you can see that it will never cross the $x$-axis -- it has no real zeros at all!
This is exactly the situation with $y = 3x^2 + x + 24$ and every other quadratic polynomial for which $b^2 - 4ac$ (the discriminant) is negative.
Why isn't taking the square root of a negative number allowed?
Lastly (since it seems as though this may be one of your first introductions to complex numbers) let me clarify something that confuses my students greatly: don't let the words "real" and "imaginary" mislead you into thinking that real numbers are somehow genuine or actual and imaginary numbers are somehow fake.
Just as a "negative" number isn't sad, cynical, or irritated, and just as an "irrational" number hasn't lost its mind, "real" and "imaginary" are technical terms, and not meant to carry with them their English connotations. Although they earned those names due to prior misunderstandings about numbers, today we know that imaginary numbers are just as actual and valid as real numbers, and in fact complex (not "complicated") numbers have huge real-world applications and consequences.
Along these same lines: you say that taking the square root of a negative number "isn't allowed". More accurate would be to say that the square root of a negative number isn't a real number ("real", again, in the technical sense) because the square of a real number is always non-negative. If you require the result of taking the square root to be a real number, then you're right, taking the square root of a negative isn't allowed. But otherwise it's perfectly fine.
By analogy: you're allowed to divide the whole number 6 by the whole number 3, and the result is the whole number 2. But if you divide the whole number 1 by the whole number 2, then the result, $\frac{1}{2}$, isn't a whole number. If you require your operations on a certain class of numbers to produce a result from that same class of numbers, then lots of operations (like division, and subtraction too, for that matter) will have situations in which they aren't "allowed" to occur.
This requirement is sometimes very desirable, and there are certainly contexts in which you will want to impose it. But you should understand that imposing this requirement is a choice that you make because it fits the context of the problems you're working on. It's not divine law. If you find yourself in a situation in which you would like to take the square root of a negative number and produce the corresponding imaginary number, you are definitely "allowed" to do so.
Best Answer
To minimize $y^2=(x-a)^2+(\tfrac{h}{w}\sqrt{w^2-x^2}-b)^2$ algebraically requires finding its turning points by solving $\frac{dy^2}{dx}=0$. This reduces to solving a quartic equation. Although quartics admit a solution in radicals, it's rarely worth using. It's probably best to use numerical minimization instead; it's not as expensive as you might think. Here's a Python solution:
For me, this took 1 millisecond to solve the special case you mentioned originally. You should be able to adapt it to C++ using Boost, which will be even faster.