These are the parametric equations of an epicycloid, where the large circle has radius $R=1$, and the smaller exterior rolling circle has radius $r=\frac{1}{4}$.
A possible simplification is writing
$$4a=4(x+i y) = 5 z - z^ 5\\
4b=4(x-i y) = \frac{5}{z} - \frac{1}{z^5}$$
where $z = \cos \theta + i \sin \theta$
and get a relation between $a$, $b$, that implies a relation between $x$ and $y$, see link. Or, we can just give the result provided by WA :
$$-81 - 45 x^2 + 365 x^4 - 15 x^6 - 480 x^8 + 256 x^{10} - 45 y^2 - 2395 x^2 y^2 - 45 x^4 y^2 - 1920 x^6 y^2 + 1280 x^8 y^2 + 365 y^4 - 45 x^2 y^4 - 2880 x^4 y^4 + 2560 x^6 y^4 - 15 y^6 - 1920 x^2 y^6 + 2560 x^4 y^6 - 480 y^8 + 1280 x^2 y^8 + 256 y^{10}=0$$
a curve of degree $10$
$4$ leaves" />
We can simplify the defining polynomial by writing it as a polynomial in $x^2 + y^2$ and $x^2 y^2$.
$\bf{Added:}$ Following the solution of @ACB: we have the polar representation ($x = r \cos \phi$, $y=r \sin \phi$)
$$\cos^2 2 \phi = \frac{(9 - 4 r^2)(16 r^4+ 3 r^2 + 6)^2}{3125 r^4}= f(r)$$
The function in $r>0$ on RHS is strictly decreasing on $(0, \infty)$. Indeed, we have
$$f'(r) = -24\, \frac{(-1 + r^2)^2 (9 + 16 r^2) (6 + 3 r^2 + 16 r^4)}{3125\, r^5}$$
Now, $f(1) = 1$, and $f(\frac{3}{2})=0$. We see that our curve is situated between the circles of radius $1$, and $\frac{3}{2} = 1 + 2\cdot \frac{1}{4}$, as it should.
So now we can write the curve in polar coordinates
$$r = f^{-1}(\cos^2 2\phi)$$
where $f^{-1}\colon [0,1]\to [1, \frac{3}{2}]$ is the inverse function of $f$.
Best Answer
$\def\¿{\mathcal}$ Maybe I am complicating this, yet another method.
I will use these identities:
$\cos^6x+\sin^6x=1-\frac34\sin^22x$
$\cos^6x-\sin^6x=\cos 2x(1-\frac14\sin^22x)$
$\cos^4x+\sin^4x=1-\frac12\sin^22x$
$\cos^4x-\sin^4x=\cos2x$
Let
$c=\cos x$,
$s=\sin x$,
$\¿ C=\cos 2x$,
$\¿S=\sin2x$
I use $d$ instead of $c$ in the original problem to avoid confusion.
Now we have $$c^3+ac=b\tag{1}$$ $$s^3+as=d\tag{2}$$
$\small\mathit{(1)^2+(2)^2}$, $$c^6+s^6+2a(c^4+s^4)+a^2(c^2+s^2)=b^2+d^2$$ $$\left(1-\frac34\¿S^2\right)+2a\left(1-\frac12\¿S^2\right)+a^2=b^2+d^2$$ $$\¿S^2=\frac{4\left[\left(a+1\right)^2-(b^2+d^2)\right]}{(3+4a)}$$
$\small\mathit{(1)^2-(2)^2}$, $$c^6-s^6+2a(c^4-s^4)+a^2(c^2-s^2)=b^2-d^2$$ $$\¿C\left(1-\frac14\¿S^2\right)+2a\¿C+a^2\¿C=b^2-d^2$$ $$\¿C\left[(a+1)^2-\frac14\¿S^2\right]=b^2-d^2$$ $$\¿C=\frac{(3+4a)(b^2-d^2)}{2(a+1)^2(2a+1)+(b^2+d^2)}$$ Now we can use $$\¿C^2+\¿S^2=1$$