I'm assuming that you already know how to get the curl for a vector field in Cartesian coordinate system. When you try to derive the same for a curvilinear coordinate system (cylindrical, in your case), you encounter problems. Cartesian coordinate system is "global" in a sense i.e the unit vectors $\mathbb {e_x}, \mathbb {e_y}, \mathbb {e_z}$ point in the same direction irrepective of the coordinates $(x,y,z)$. On the other hand, the curvilinear coordinate systems are in a sense "local" i.e the direction of the unit vectors change with the location of the coordinates.
For example, in a cylindrical coordinate system, you know that one of the unit vectors is along the direction of the radius vector. The radius vector can have different orientation depending on where you are located in space. Hence the unit vector for point A differs from those of point B, in general.
I'll first try to explain how to go from a cartesian system to a curvilinear system and then just apply the relevant results for the cylindrical system. Let us take the coordinates in the new system as a function of the original coordinates. $$q_1 = q_1(x,y,z) \qquad q_2 = q_2(x,y,z) \qquad q_3 = q_3(x,y,z) $$
Let us consider the length of a small element
$$ ds^2 = d\mathbf{r}.d\mathbf{r} = dx^2 + dy^2 + dz^2 $$
The small element $dx$ can be written as
$$ dx = \frac{\partial x}{\partial q_1}dq_1 + \frac{\partial x}{\partial q_2}dq_2+\frac{\partial x}{\partial q_3}dq_3 $$
Doing the same for $dy$ and $dz$, we can get the distance element in terms of partial derivatives of $x,y,z$ in the new coordinate system. This will be of the form
$$ ds^2 = \sum_{i,j} \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_j} dq_i dq_j = \sum_{i,j} g_{ij} dq_i dq_j $$
Here $\frac{\partial \mathbf{r}}{\partial q_j}$ represents the tangent vectors for $q_i = $constant , $i\neq j$. For an orthoganal coordinate system, where the surfaces are mutually perpendicular, the dot product becomes
$$ \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_j} = c \delta_{ij}$$
where the scaling factor $c$ arises as we haven't considered unit vectors. These factors are taken as $$c = \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_i} = h_i^2$$
$$ ds^2 = \sum_{i=1}^3 (h_i dq_i)^2 = \sum_i ds_i^2$$
Hence the length element along direction $q_i$ is given by $ds_i = h_i dq_i $.
Now, we are equipped to get the curl for the curvilinear system. Consider an infinitesimal enclosed path in the $q_1 q_2$ plane. And evaluate the path integral of the vector field $\mathbf{V}$ along this path.
$$\oint \mathbf{V}(q_1,q_2,q_3).d\mathbf{r} = \oint \mathbf{V}.\left( \sum_{i=2}^3 \frac{\partial \mathbf{r}}{\partial q_i} dq_i\right)$$
(q1, q2+ds_2) (q1+ds_1, q2+ds_2)
-----------<------------
| |
| |
V ^
| |
|---------->-----------|
(q1, q2) (q1+ds_1, q2)
$$ \oint \mathbf{V}.d\mathbf{r} = V_1 h_1 dq_1 -
\left( V_1 h_1 + \frac{\partial V_1 h_1}{\partial q_2} dq_2\right)dq_1
- V_2 h_2 dq_2 +
\left( V_2 h_2 + \frac{\partial V_2 h_2}{\partial q_1} dq_1\right)dq_2$$
$$ = \left( \frac{\partial V_2 h_2}{\partial q_1} -
\frac{\partial V_1 h_1}{\partial q_2} \right)dq_1 dq_2$$
From Stokes theorem,
$$ \oint \mathbf{V}.d\mathbf{r} =
\int_S \nabla \times \mathbf{V} . d\mathbf{\sigma} =
\nabla \times \mathbf{V} . \mathbf{\hat{q_3}} (h_1 dq_1) (h_2 dq_2)
= \left( \frac{\partial V_2 h_2}{\partial q_1} -
\frac{\partial V_1 h_1}{\partial q_2} \right)dq_1 dq_2 $$
Hence the $3$ component of the curl can be written as
$$(\nabla \times \mathbf{V})_3 = \frac{1}{h_1 h_2} \left( \frac{\partial V_2 h_2}{\partial q_1} -\frac{\partial V_1 h_1}{\partial q_2} \right) $$
Similarly, other components can be evaluated and all the components can be assembled in the familiar determinant format.
$$\nabla \times \mathbf{V} = \frac{1}{h_1 h_2 h_3} \begin{vmatrix}
\mathbf{\hat{q_1}}h_1 & \mathbf{\hat{q_2}}h_2 & \mathbf{\hat{q_3}}h_3\\
\frac{\partial}{\partial q_1} & \frac{\partial}{\partial q_2} & \frac{\partial}{\partial q_3} \\
V_1 h_1 & V_2 h_2 & V_3 h_3 \\
\end{vmatrix}$$
Now the expression for the curl is ready. All we need to do is find the values of $h$ for the cylindrical coordinate system. This can be obtained, if we know the transformation between cartesian and cylindrical polar coordinates.
$$ (x,y,z) = (r\cos\phi, r\sin\phi, z)$$
Now the length element
$$ ds^2 = dx^2 + dy^2 + dz^2 = (d(r\cos\phi))^2 + (d(r\sin\phi))^2 + dz^2 $$
Simplifying the above expression, we get
$$ ds^2 = (dr)^2 + r^2(d\phi)^2 + (dz)^2 $$
From the above equation, we can obtain the scaling factors, $h_1 = 1$ , $h_2 = r$, $h_3 = 1$. Hence the curl of a vector field can be written as,
$$ \nabla \times \mathbf{V} = \frac{1}{r}
\begin{vmatrix}
\mathbf{\hat{r}} & r\mathbf{\hat{\phi}}& \mathbf{\hat{z}}\\
\frac{\partial}{\partial r} & \frac{\partial}{\partial \phi} & \frac{\partial}{\partial z} \\
V_r & r V_\phi & V_z \\
\end{vmatrix}
$$
$\require{cancel}$
The problem starts with the determinant, I'm going to highlight just the first term as an example
\begin{eqnarray}
{\rm curl} ({\bf F}) &=& \color{red}{\frac{1}{r^2\sin \theta}}\left|
\begin{array}{ccc}
\color{blue}{{\bf e}_r} & r{\bf e}_\theta & r\sin\theta{\bf e}_\phi \\
\partial/\partial_r & \color{orange}{\partial/\partial \theta} & \color{magenta}{\partial /\partial\phi} \\
F_r & \color{olive}{rF_\theta} & \color{purple}{r\sin\theta F_\phi}
\end{array}\right| \\
&=& \color{red}{\frac{1}{r^{\cancel{2}} \sin\theta}} \left[\color{orange}{\frac{\partial}{\partial\theta}}\left(\color{purple}{\cancel{r}\sin\theta F_\phi} \right) - \color{magenta}{\frac{\partial}{\partial\phi}}\left(\color{olive}{\cancel{r} F_\theta}\right)\right]\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
&=& \color{red}{\frac{1}{r \sin\theta}} \left[\color{orange}{\frac{\partial}{\partial\theta}}\left(\color{purple}{\sin\theta F_\phi} \right) - \color{magenta}{\frac{\partial}{\partial\phi}}\left(\color{olive}{ F_\theta}\right)\right]\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
\end{eqnarray}
Now it is time to replace your field. We have
\begin{eqnarray}
\color{purple}{F_\phi} &=& -r^2\sin^2\theta\sin\phi \\
\color{olive}{F_\theta} &=& 3r^2\sin^2\theta\cos\theta\cos\phi \\
\end{eqnarray}
So that
\begin{eqnarray}
{\rm curl}(F) &=& \color{red}{\frac{1}{r \sin\theta}} \left[\color{orange}{\frac{\partial}{\partial\theta}}\left(\color{purple}{-r^2\sin^3\theta\sin\phi} \right) - \color{magenta}{\frac{\partial}{\partial\phi}}\left(\color{olive}{ 3r^2\sin^2\theta\cos\theta\cos\phi}\right)\right]\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
&=& \color{red}{\frac{1}{r \sin\theta}} \left[\color{purple}{-r^2\sin\phi}\color{orange}{\frac{\partial}{\partial\theta}}\left(\color{purple}{\sin^3\theta} \right) - \color{olive}{3r^2\sin^2\theta\cos\theta}\color{magenta}{\frac{\partial}{\partial\phi}}\left(\color{olive}{ \cos\phi}\right)\right]\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
&=& \color{red}{\frac{1}{r \sin\theta}} \left[\cancel{\color{purple}{-3r^2\sin\phi\sin^2\theta\cos\theta}} + \cancel{\color{olive}{3r^2\sin^2\theta\cos\theta\sin\phi}}\right]\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
&=& 0\color{blue}{{\bf e}_r} + (\cdots){\bf e}_\theta + (\cdots){\bf e}_\phi \\
\end{eqnarray}
I will let you finish the other two components
Best Answer
You need to write your $1$-form in terms of unit length basis covectors, namely $dr$, $r\,d\phi$, and $dz$, and then figure out the $\star$ ... It looks to me like you said $\star(d\phi\wedge dz) = r\,dr$. In fact, $\star(r\,d\phi\wedge dz) = dr$, so $\star(d\phi\wedge dz) =\dfrac1r\,dr$. Similarly with $\star(dr\wedge d\phi)$. I haven't checked the rest. (P.S. You should work with an orthonormal basis for the tangent vectors, too.)