You're set up is spot on.
Indeed, we're finding the area bounded below the curve $y = \sqrt{2x - x^2}\,$ and above the curve $y = x^2,\;$ between $x = 0$ and $x = 1$:
![enter image description here](https://i.stack.imgur.com/tSL6S.gif)
So indeed, we need to integrate:
$$\begin{align} I & = \int_0^1 \left(\sqrt{2x - x^2} - x^2\right)\,dx \\ \\
& = \int_0^1 \sqrt{1 - 1 + 2x - x^2} \,dx - \int_0^1 x^2\,dx\\ \\
& = \int_0^1 \sqrt{1 - (x - 1)^2} \,dx - \int_0^1 x^2\,dx
\end{align}$$
Now, for the first integral, we use the trigonometric substitution $(x - 1) = \sin\theta$, so that $dx = \cos\theta\,d\theta$.
Finding the new bounds of integration for the first integral (so we can save ourselves the task of "back substitution" at the very end):
At $x = 0, \sin\theta = -1 \implies \theta = -\pi/2.\;$ At $x = 1, \sin\theta = 0 \implies \theta = 0$.
This gives you, after substituting for the first integral:
$$I = \int_{-\pi/2}^{0} \sqrt{1 - \sin^2 \theta}\cos\theta\,d\theta - \int_0^1 x^2\,dx $$
We will use the identities $$\begin{align}\;1 - \sin^2\theta & = \cos^2 \theta\tag{1} \\ \cos^2 \theta & = \dfrac {1 + \cos (2\theta)}{2}\tag{2}\end{align}$$
$$I =\int_{-\pi/2}^0 \left(\sqrt{\cos^2\theta}\right)\cos\theta\,d\theta = \int_{-\pi/2}^0 \cos^2\theta\,d\theta - \int_0^1 x^2\,dx \tag{1}$$
$$I = \dfrac 12\int_{-\pi/2}^0 \left(1 + \cos(2\theta)\right) \,d\theta - \int_0^1 x^2 \,dx\tag{2}$$
Integration should now be relatively straightforward:
$$I = \left[\dfrac \theta2 + \dfrac 14\sin(2\theta)\right]\Big|_{-\pi/2}^0 \;- \;\dfrac{x^3}3\Big|_0^1\quad = \quad \frac\pi4-\dfrac 13$$
You can calculate the area to the right of both curves and left of the $y$-axis between $y=0$ and $y=\frac {11}2$ by integrating the given functions. Then, you can substract the results to get the area.
Also, just mirroring the image in $x=y$ or rotating it by a quarter turn may help.
EDIT
One integral (the blue one) should be $\frac{121}{12}$ and the red one should be $\frac{1573}{24}$. The difference is the area. (I'm assuming we're only talking about positive areas here.)
Best Answer
you have to use the form $$\int_a^b f(x)-g(x)dx$$ since the first integral contains a negative part of the function $y=x$ and this part gives negative area i.e. the value 5 is not true
The right one is $$\int_{-1}^3 x^2+3-xdx=52/3$$