I don't think it's really all that mysterious. The motivation behind using the Tschirnhaus Transformation seems clear; it's better to have fewer terms and fewer coefficients when solving an equation. It next probably occurred to Cardano to try $y = u + v$ and see what happens. Rewrite the equation as
$$u^3 + v^3 + 3uv(u + v) + p(u + v) + q = 0$$
You have a little leeway in choosing $u$ and $v$, since you only need them to sum to a root of the equation. So you might want to impose a second condition on $u$ and $v$ to simplify things (and two equations in the two variables $u$ and $v$ should specify their values). To reduce the above to a quadratic equation, you can stipulate that $uv = -{p \over 3}$. Namely you have
$$u^3 + v^3 + q = 0$$
which is equivalent to
$$u^3 -{p^3 \over 27}{1 \over u^3} + q = 0$$
Or just
$$u^6 + qu^3 - {p^3 \over 27} = 0$$
This is a quadratic equation in $u^3$, which you can solve. $u$ can be any of the three cube roots of this, and $v$ is determined by the equation $3uv = -p$. Thus you have three possible values of $u + v$ which will be the three roots of your equation.
In Cardano's time they didn't really know much about complex numbers (i.e. that every number has $n$ complex $n$th roots) but he was smart enough to basically understand what was going on.
I don't know if this has any advantage over using the rational root theorem directly, but here is a way you can find the representation if you only have the cube root representation (but it basically finds a polynomial representation and then uses the rational root theorem).
Suppose you don't have the full polynomial but only the solution given by Cardano's formula:
$$ x =\sqrt[3]{a+\sqrt{b}}+\sqrt[3]{a-\sqrt{b}} $$
(where $b$ is possibly negative: I won't use imaginary numbers explicitly)
I suppose that $x$ is an integer for now. I will explain the rational case later (it works almost exactly the same). Take the cube, and simplify to
$$ x^3 = 2a + 3x\sqrt[3]{a^2 - b} $$
A first test is to see if $ a^2 - b $ equals a cube. If it doesn't, it's impossible to represent the value as an integer. Otherwise, we define $u := \sqrt[3]{a^2 - b}$ (please note that the value of $u$ is known).
$$ x^3 = 2a + 3xu $$
$$ x^3 - 3xu - 2a = 0 $$
You can now use the rational root theorem, and try plugging in divisors of $2a$ for $x$ to see if you found a root. If you find an equality ($ x(x^2 + 3) = 2a $), you've found an integer $x$ satisfying $ x = \sqrt[3]{a+\sqrt{b}}+\sqrt[3]{a-\sqrt{b}} $, and otherwise $ \sqrt[3]{a+\sqrt{b}}+\sqrt[3]{a-\sqrt{b}} $ is not an integer.
I think the same approach works for rational instead of integer solutions. The only thing is that you should factor out the denominator at the start so you can still use the rational root theorem.
So, when we want to find the value for
$$ x = \sqrt[3]{\frac{7}{25}+i\sqrt{\frac{972}{15625}}} + \sqrt[3]{\frac{7}{25}-i\sqrt{\frac{972}{15625}}} $$
factor out the denominator (it is $5$), and bring it to the left side:
$$ 5x = \sqrt[3]{35+i\sqrt{972}} + \sqrt[3]{35-i\sqrt{972}} $$
Substitute $ y := 5x $. Continue like we would as explained in the case that $a$, $b$ are integers. We find $ u = \sqrt[3]{35^2 - (-972)} = \sqrt[3]{2197} = 13 $
The next step is to try divisors of $2a = 70$ in the equation $f(y) := y^3 - 39y - 70 = 0$.
We try $y = 1, 2, 5, 7, 35, 70$
$f(1) = -108, f(2) = -140, f(5) = -140, f(7) = 0$
So
$$ y = \sqrt[3]{35+i\sqrt{972}} + \sqrt[3]{35-i\sqrt{972}} = 7 $$
$$ x = \sqrt[3]{\frac{7}{25}+i\sqrt{\frac{972}{15625}}} + \sqrt[3]{\frac{7}{25}-i\sqrt{\frac{972}{15625}}} = \frac{y}{5} = \frac{7}{5} $$
You can also use this method to determine if $ x =\sqrt[3]{a\pm\sqrt{b}} $ has a respresentation in the form $ \frac{c\pm\sqrt{d}}{2} $, since (after applying this method) you have:
$$ p_+p_- = u = \sqrt[3]{a^2-b} $$
$$ p_+ + p_- = x $$
So $ p_- = \frac{u}{p_+} $ and if we substitute this in the second equation we get a quadratic equation with $ p_+, p_- $ as roots. But again: it's probably easier to just use the rational root theorem on the original cubic equation.
I got started with this by this blog post on Euler, Erdős, however, in this post, the author assumes $b^2 - a = -1$, which makes the equations more elegant. However, this is not the most general case, so the author loses some generality. In the comments I found a link to "Cardan Polynomials and the Reduction of Radicals" by Thomas J. Osler. It looks like it goes in depth much more (I think he shows how to do something similar for radicals of even higher degree than 3), but I have only skimmed the paper.
Best Answer
This is the casus irreducibilis, first discussed in detail by Bombelli. We end up unavoidably needing to travel through the complex numbers to end up with real roots! This is important historically, since it was the first time that one needed to treat complex non-real numbers seriously. We do not need to worry about non-real numbers when solving quadratics, since after all we can say that there are no roots. But that is not the case here, since undeniably there are real roots.
When you are trying to find the cube roots of your complex expression, you can assume that a cube root of your first expression is $a+bi$. Cube this, and you will get some messy equations. But you can (in this case) "spot" a root, and then you are finished. But that is cheating, it is due entirely to the fact that the roots of the original cubic are "nice."
One workaround of sorts is to use the trigonometric solution to the cubic, which is uses the trigonometric identity $\cos 3\theta=4\cos^3\theta-3\cos\theta$. You may want to look at the Wikipedia article on the cubic, which is reasonably thorough.