First of all, the Jacobian of the spherical coordinat transformation is $\rho^2\sin\phi$ not $\rho\sin\phi$.
Also, the upper bound for $\rho$ should have been $6\cos\phi$ not $3$. The equation $\rho = 3$ describes a sphere of radius $3$ centered at the origin while the equation $\rho = 6\cos\phi$ describes a sphere of radius $3$ centered at $(x,y,z) = (0,0,3)$.
After fixing those two errors, you should get an integral which evaluates to the correct answer.
I'm adding this answer because the other answers so far answer different questions (either a different region, or a different method) than what was originally asked. I'm also going to try to explain my work in more depth to make it easier to follow along.
Note that the cone $z = \sqrt{(x^2 + y^2)}$ can be represented in spherical coordinates as $\phi = \frac{\pi}{4}$. You already calculated that $z = r \sin \phi$, and we know from the spherical coordinate transform that $z = r \cos \phi$. Combining these two equations, we get $\sin \phi = \cos \phi$ or $tan \phi = 1$, which means $\phi = \frac{\pi}{4}$. Intuitively/geometrically, you can also imagine that the cone covers the full range from $\theta = 0$ to $\theta = 2 \pi$ of rotation around the z-axis, meaning $\theta$ can be any value, and the radius can be any value as well (the cone is unbounded above), so the only parameter needed to define the cone is $\phi$.
To set up the region of integration, we know that to calculate the volume of the sphere, we would have the radius ranging from 0 to 1, with $\theta$, the angle of rotation around the z-axis, ranging from 0 to $2 \pi$, and $\phi$, the angle with the z-axis, ranging from 0 to $\pi$. In this case however, we want to exclude the region of the cone, which is the region from $\phi = 0$ to $\phi = \frac{\pi}{4}$. Thus we want to include $\phi$ only over the range from $\frac{\pi}{4}$ to $\pi$.
Thus, we have the below integral:
$$\int_{0}^{1} \int_{0}^{2 \pi} \int_{\frac{\pi}{4}}^{\pi} r^2 sin(\phi) d\phi d\theta dr$$
where $r^2 sin(\phi)$ comes from the change of base formula, and can be calculated as the absolute value of the determinant of the derivative matrix of $x$, $y$, $z$ with respect to $r$, $\theta$, and $\phi$.
The integral can be evaluated in any order, probably $\theta$ then $r$ then $\phi$ to get $2 \pi \frac{1}{3} (1 + \frac{1}{\sqrt{2}}) \approx 3.58$.
A quick intuitive check confirms that this answer makes sense: the volume of the sphere is given by $\frac{4}{3}\pi r^3$ which in this case is just $\frac{4}{3}\pi \approx 4.19 $, and we're taking out a small fraction of the top half of the sphere.
Find volume of a solid figure that lies between:
Best Answer
Let's consider the region above $z = 0$. Due to symmetry the volume bound is same above and below $z = 0$.
The region is defined by,
a) $x^2 + y^2 + z^2 \leq 4$
In spherical coordinates, $\rho \leq 2$ (the sphere)
b) $2x^2+2y^2-2z^2\geq 1$
In spherical coordinates, $ - 2 \rho^2 \cos 2\phi\geq 1$
c) $2 z^2 \geq x^2 + y^2$
In spherical coordinates, $ \tan \phi \leq \sqrt2$
At intersection of hyperboloid and sphere,
$-8 \cos2\phi = 1$ (plugging in $\rho = 2$ in $(b)$)
So, $\cos 2\phi = - \frac{1}{8}$
Notice that this is the region which is outside the hyperboloid and inside the cone and the sphere.
So, $ \sqrt{- (\sec 2\phi) / 2} \leq \rho \leq 2$ (please note for the given limits of $\phi$, $\sec 2\phi$ is negative and so the value inside the square root is positive).
$\arccos \left(\frac{\sqrt7}{4}\right) \leq \phi \leq \arccos \left(\frac{1}{\sqrt3}\right)$ (the lower bound is same as $\cos 2 \phi \leq - 1/8$ and the upper bound is same as $\tan \phi \leq \sqrt2$. I have just rewritten them differently).
$0 \leq \theta \leq 2\pi$
Can you take it from here?