By independence, the joint density function of $X$ and $Y$ is $(3e^{-3x})(5e^{-5y})$ when $x$ and $y$ are positive, and $0$ elsewhere. Let $W=\dfrac{X}{Y}$. We want to find the cumulative distribution function $F_W(w)$ of $W$.
If $w\le 0$, then $F_W(w)=0$. Now let $w$ be positive. For a long time we will think of $w$ as being fixed, like $w=1.7$.
We have $\dfrac{X}{Y}\le w$ if and only if $Y\ge \dfrac{X}{w}$.
Draw the line $y=\dfrac{x}{w}$. Then $\dfrac{X}{Y}\le w$ if and only if $(X,Y)$ is above or on that line. Let $T$ be the first-quadrant region which is above or on that line. Then
$$\Pr(W\le w)=\iint_T (3e^{-3x})(5e^{-5y})\,dy\,dx.$$
Now we just have an integration problem. It is marginally easier to integrate first with respect to $y$, from $x/w$ to $\infty$.
We get $3e^{-3x}e^{-5x/w}$.
Finally, integrate with respect to $x$ (Edit: this originally said y, i believe it should be x?), from $0$ to $\infty$. The integration is straightforward, we are just integrating $3\exp(-(3+5/w)x)$, and a substitution does it.
Now that we have the cdf $F_W(w)$, differentiate it with respect to $w$ to get the density function $f_W(w)$ of $W$.
The independence of $X$ and $Y$ gives you the joint density: $$f_{X,Y}(x,y)=f(x)f(y).$$
Once you have the joint density,
$$
P(X^2>Y)=\iint_{\{(x,y)\in\mathbb{R}^2|x^2>y\}}f_{X,Y}(x,y)\ dxdy\\
=\iint_{\{(x,y)\in\mathbb{R}^2|x^2>y\}}f(x)f(y)\ dxdy.
$$
Now, it is an exercise to show (using the definition of $f$) that
$$
\iint_{\{(x,y)\in\mathbb{R}^2|x^2>y\}}f(x)f(y)\ dxdy=\int_{0}^{1} \int_{0}^{x^2} (2x)(2y) dydx.
$$
Best Answer
You could solve it this way: \begin{align}f_{XY}(z) & = \int_\Bbb R f_{XY\mid Y=y}(z)\,f_{Y}(y)\,dy\tag 1\\[1ex]& = \int_\Bbb R \tfrac{1}{y}f_{X\mid Y=y}(\tfrac{z}{y})\,f_{Y}(y)\,dy \tag 2\\[1ex]&= \int_\Bbb R \tfrac{1}{y}f_{X}(\tfrac{z}{y})\,f_{Y}(y)\,dy \tag 3\\[1ex]&= 25\int_\Bbb R \frac{1}{y}\frac{z^4}{y^4}y^4\,\mathbf 1_{0\leq z/y\leq 1,0\leq y\leq 1} dy \tag 4\\[1ex]& = 25z^4\mathbf 1_{0\leq z\leq 1}\int_z^1 \frac{1}{y}\,dy\tag 5\\[1ex]& = 25 z^4 \ln(\tfrac{1}{z})~\mathbf 1_{0\leq z\leq 1}\tag 6\end{align} In the first equality we use the "density version" of the law of total probability, the second equality is due to transformation formula for probability densities and the third equation is due to independence of $X$ and $Y$. In the forth equality we substitute the function, and including the indicators for the joint support, which is equivalent to $\mathbf 1_{0\leq z\leq y\leq 1}$, so explaining why the domain for the integral is $(z..1)$.