First, sketch the support of $(X,Y)$, which is $$(-1 \le X \le 1) \cap (0 < Y \le |X|).$$ This describes a "bow-tie" shaped region in the Cartesian coordinate plane, consisting of two right triangles with vertices $$\{(0,0), (1,0), (1,1)\}, \quad \{(0,0), (-1,0), (-1,1)\}.$$ The symmetry of this region is clear, and the line of symmetry is the $y$-axis.
The plot below visualizes the joint density in three-dimensional space:
For a fixed $Y = y$, the permissible values of $X$ comprise the union of two intervals; that is to say, $$(X \mid Y = y) \in [-1, -y] \cup [y, 1].$$ So, the interval of integration for the marginal density of $Y$ is not $[-1,1]$ but the interval above: $$f_Y(y) = \int_{x=-1}^{-y} 5x^2 y \, dx + \int_{x=y}^1 5x^2 y \, dx.$$
The above animation plots $X$ for specific values of $Y = y$ for $y \in (0,1]$. The conditional density $f_{X\mid Y}(x \mid y)$ is proportional to this plot in such a way that the area under the curve is made equal to $1$. In other words, the function $f_{X \mid Y}$ is simply a scaled version of this animation such that the area under the curve remains $1$ regardless of the choice of $y$:
Compare this with the previous animation.
Another way to reason about this is to observe that the support requires that $|X| \ge Y$: hence, on the interval $X \in [-y,y]$, $f_{X,Y}(x,y) = 0$, thus writing $$f_Y(y) = \int_{x=-1}^1 f_{X,Y}(x,y) \, dx = \int_{x=-1}^1 5x^2 y \, dx$$ fails to reflect that the joint density is not $5x^2 y$ when $-y \le x \le y$. We can correct for this by the suitable use of indicator functions; for example, we could have instead written the joint density as $$f_{X,Y}(x,y) = 5x^2 y \;\mathbb 1(|x| \ge y), \quad (x,y) \in [-1,1] \times [0,1].$$ This reminds us that integration in the rectangular region $[-1,1] \times [0,1]$ comes with an extra condition for the integrand to be positive: $$f_Y(y) = \int_{x=-1}^1 5x^2 y \; \mathbb 1(|x| \ge y) \, dx = \int_{x=-1}^{-y} 5x^2 y \, dx + \int_{x=y}^1 5x^2 y \, dx.$$ We could go overboard and write the whole density as $$f_{X,Y}(x,y) = 5x^2 y \; \mathbb 1 (0 < y \le |x| \le 1), \quad (x,y) \in \mathbb R^2$$ but this isn't really necessary as we already intuitively grasp that the convex boundary of the support is rectangular, leading to a natural choice of the lower and upper limits of integration except when dealing with the nonconvex part.
Best Answer
Note that the expression for the joint density $f_{X,Y}(x,y)$ that you computed only holds for $x$ and $y$ satisfying $x \ge 0$ and $y \ge \max\{x, 4\}$; it is zero otherwise. Thus for $x \ge 0$ we have $$f_X(x) = \int_{-\infty}^\infty f_{X,Y}(x,z) \, dz = \int_{\max\{x, 4\}}^\infty \frac{192}{z^5} \, dz.$$
Continuing the computation leads to $$f_X(x) = 48 \max\{x, 4\}^{-4} = \begin{cases}48 / 4^4 & x \in [0, 4] \\ 48 / x^4 & x > 4\end{cases}$$ So, $$\int_0^\infty f_X(x) \, dx = 4 \cdot \frac{48}{4^4} + \int_4^\infty \frac{48}{x^4} \, dx = \frac{3}{4} + \frac{16}{4^3} = 1.$$