Before doing the derivation, I'd like to explain the origin of the scale factors $h_i$. We will assume throughout that our curvilinear coordinates $x_1$, $x_2$, and $x_3$ are orthogonal, i.e. that the gradients $\nabla x_1$, $\nabla x_2$, $\nabla x_3$ are orthogonal vectors. We will also assume that they are right-handed, in the sense that $\widehat{e}_1\times\widehat{e}_2=\widehat{e}_3$.
The Origin of the Scale Factors
One important difference between curvilinear coordinates $x_1,x_2,x_3$ and standard $x,y,z$ coordinates is that curvilinear coordinates do not change at unit speed. That is, if we start at a point and move in the direction of $\widehat{e}_i$, we should not expect $x_i$ to increase at unit rate.
One consequence of this is that the gradients $\nabla x_i$ of the curvilinear coordinates are not unit vectors. For $x,y,z$ coordinates, we know that
$$
\nabla x \;=\; \widehat{\imath},\qquad \nabla y \;=\; \widehat{\jmath},\qquad\text{and}\qquad \nabla z\;=\; \widehat{k}.
$$
However, for curvilinear coordinates, we get something like
$$
\nabla x_1 \;=\; \frac{1}{h_1}\widehat{e}_1,\qquad \nabla x_2 \;=\; \frac{1}{h_2}\widehat{e}_2,\qquad\text{and}\qquad \nabla x_3 \;=\; \frac{1}{h_3}\widehat{e}_3,
\tag*{(1)}$$
where $h_1$, $h_2$, and $h_3$ are scalars.
The reciprocal $1/h_i$ of each scale factor represents the rate at which $x_i$ will change if we move in the direction of $\widehat{e}_i$ at unit speed. Equivalently, you can think of $h_i$ as the speed that you have to move if you want to increase $x_i$ at unit rate. For spherical coordinates, it should be geometrically obvious that $h_1 = 1$, $h_2 = r$, and $h_3 = r\sin\theta$.
Formula for the Gradient
We can use the scale factors to give a formula for the gradient in curvilinear coordinates. If $u$ is a scalar, we know from the chain rule that
$$
\nabla u \;=\; \frac{\partial u}{\partial x_1}\nabla x_1 \,+\, \frac{\partial u}{\partial x_2}\nabla x_2 \,+\, \frac{\partial u}{\partial x_3}\nabla x_3
$$
Substituting in the formulas from (1) gives us
$$
\nabla u \;=\; \frac{1}{h_1}\frac{\partial u}{\partial x_1}\widehat{e}_1 \,+\, \frac{1}{h_2}\frac{\partial u}{\partial x_2}\widehat{e}_2 \,+\, \frac{1}{h_3}\frac{\partial u}{\partial x_3}\widehat{e}_3\tag*{(2)}
$$
This is the formula for the gradient in curvilinear coordinates.
Formula for the Curl
First, observe that the determinant formula you have given for the curl is equivalent to the following three formulas:
$$
\begin{gather*}
(\nabla\times A)\cdot\widehat{e}_1 \;=\; \frac{1}{h_2h_3}\left|\begin{matrix}\frac{\partial}{\partial x_2} & \frac{\partial}{\partial x_3} \\[8pt] h_2A_2 & h_3A_3\end{matrix}\right| \\[12pt]
(\nabla\times A)\cdot\widehat{e}_2 \;=\; \frac{1}{h_3h_1}\left|\begin{matrix}\frac{\partial}{\partial x_3} & \frac{\partial}{\partial x_1} \\[8pt] h_3A_3 & h_1A_1\end{matrix}\right| \\[12pt]
(\nabla\times A)\cdot\widehat{e}_3 \;=\; \frac{1}{h_1h_2}\left|\begin{matrix}\frac{\partial}{\partial x_1} & \frac{\partial}{\partial x_2} \\[8pt] h_1A_1 & h_2A_2\end{matrix}\right|
\end{gather*}
$$
We will prove the first of these formulas. Given any vector field $A$, we can write
$$
\begin{align*}
A \;&=\; A_1 \widehat{e}_1 \,+\, A_2 \widehat{e}_2 \,+\, A_3 \widehat{e}_3 \\[6pt]
&=\; h_1A_1\,\nabla x_1 \,+\, h_2A_2\,\nabla x_2 \,+\, h_3A_3\,\nabla x_3
\end{align*}
$$
Taking the curl gives
$$
\nabla \times A \;=\; \nabla(h_1A_1)\times (\nabla x_1) \,+\, \nabla(h_2A_2)\times(\nabla x_2) \,+\, \nabla(h_3A_3)\times(\nabla x_3)
$$
Here we have used the identity $\nabla\times(uF) = (\nabla u)\times F + u(\nabla\times F)$, as well as the fact that the curl of a gradient is zero. Applying formula (1), we get
$$
\nabla \times A \;=\; \frac{1}{h_1}\nabla(h_1A_1)\times \widehat{e}_1 \,+\, \frac{1}{h_2}\nabla(h_2A_2)\times\widehat{e}_2 \,+\, \frac{1}{h_3}\nabla(h_3A_3)\times\widehat{e}_3
$$
When we take the cross products, the $\widehat{e}_1$ component will be
$$
(\nabla \times A)\cdot\widehat{e}_1 \;=\; \frac{1}{h_3}\nabla(h_3A_3)\cdot\widehat{e}_2 \,-\, \frac{1}{h_2}\nabla(h_2A_2)\cdot\widehat{e}_3.
$$
But, by formula (2) for the gradient,
$$
\nabla(h_3A_3)\cdot\widehat{e}_2 \;=\; \frac{1}{h_2}\frac{\partial}{\partial x_2}(h_3 A_3)\qquad\text{and}\qquad\nabla(h_2A_2)\cdot\widehat{e}_3 \;=\; \frac{1}{h_3}\frac{\partial}{\partial x_3}(h_2 A_2)
$$
Therefore,
$$
\begin{align*}
(\nabla \times A)\cdot\widehat{e}_1 \;&=\; \frac{1}{h_2h_3}\frac{\partial}{\partial x_2}(h_3A_3) \,-\, \frac{1}{h_2h_3}\frac{\partial}{\partial x_3}(h_2A_2) \\[12pt]
&=\; \frac{1}{h_2h_3}\left|\begin{matrix}\frac{\partial}{\partial x_2} & \frac{\partial}{\partial x_3} \\[8pt] h_2A_2 & h_3A_3\end{matrix}\right|
\end{align*}
$$
as desired.
I'm assuming that you already know how to get the curl for a vector field in Cartesian coordinate system. When you try to derive the same for a curvilinear coordinate system (cylindrical, in your case), you encounter problems. Cartesian coordinate system is "global" in a sense i.e the unit vectors $\mathbb {e_x}, \mathbb {e_y}, \mathbb {e_z}$ point in the same direction irrepective of the coordinates $(x,y,z)$. On the other hand, the curvilinear coordinate systems are in a sense "local" i.e the direction of the unit vectors change with the location of the coordinates.
For example, in a cylindrical coordinate system, you know that one of the unit vectors is along the direction of the radius vector. The radius vector can have different orientation depending on where you are located in space. Hence the unit vector for point A differs from those of point B, in general.
I'll first try to explain how to go from a cartesian system to a curvilinear system and then just apply the relevant results for the cylindrical system. Let us take the coordinates in the new system as a function of the original coordinates. $$q_1 = q_1(x,y,z) \qquad q_2 = q_2(x,y,z) \qquad q_3 = q_3(x,y,z) $$
Let us consider the length of a small element
$$ ds^2 = d\mathbf{r}.d\mathbf{r} = dx^2 + dy^2 + dz^2 $$
The small element $dx$ can be written as
$$ dx = \frac{\partial x}{\partial q_1}dq_1 + \frac{\partial x}{\partial q_2}dq_2+\frac{\partial x}{\partial q_3}dq_3 $$
Doing the same for $dy$ and $dz$, we can get the distance element in terms of partial derivatives of $x,y,z$ in the new coordinate system. This will be of the form
$$ ds^2 = \sum_{i,j} \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_j} dq_i dq_j = \sum_{i,j} g_{ij} dq_i dq_j $$
Here $\frac{\partial \mathbf{r}}{\partial q_j}$ represents the tangent vectors for $q_i = $constant , $i\neq j$. For an orthoganal coordinate system, where the surfaces are mutually perpendicular, the dot product becomes
$$ \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_j} = c \delta_{ij}$$
where the scaling factor $c$ arises as we haven't considered unit vectors. These factors are taken as $$c = \frac{\partial \mathbf{r}}{\partial q_i} .
\frac{\partial \mathbf{r}}{\partial q_i} = h_i^2$$
$$ ds^2 = \sum_{i=1}^3 (h_i dq_i)^2 = \sum_i ds_i^2$$
Hence the length element along direction $q_i$ is given by $ds_i = h_i dq_i $.
Now, we are equipped to get the curl for the curvilinear system. Consider an infinitesimal enclosed path in the $q_1 q_2$ plane. And evaluate the path integral of the vector field $\mathbf{V}$ along this path.
$$\oint \mathbf{V}(q_1,q_2,q_3).d\mathbf{r} = \oint \mathbf{V}.\left( \sum_{i=2}^3 \frac{\partial \mathbf{r}}{\partial q_i} dq_i\right)$$
(q1, q2+ds_2) (q1+ds_1, q2+ds_2)
-----------<------------
| |
| |
V ^
| |
|---------->-----------|
(q1, q2) (q1+ds_1, q2)
$$ \oint \mathbf{V}.d\mathbf{r} = V_1 h_1 dq_1 -
\left( V_1 h_1 + \frac{\partial V_1 h_1}{\partial q_2} dq_2\right)dq_1
- V_2 h_2 dq_2 +
\left( V_2 h_2 + \frac{\partial V_2 h_2}{\partial q_1} dq_1\right)dq_2$$
$$ = \left( \frac{\partial V_2 h_2}{\partial q_1} -
\frac{\partial V_1 h_1}{\partial q_2} \right)dq_1 dq_2$$
From Stokes theorem,
$$ \oint \mathbf{V}.d\mathbf{r} =
\int_S \nabla \times \mathbf{V} . d\mathbf{\sigma} =
\nabla \times \mathbf{V} . \mathbf{\hat{q_3}} (h_1 dq_1) (h_2 dq_2)
= \left( \frac{\partial V_2 h_2}{\partial q_1} -
\frac{\partial V_1 h_1}{\partial q_2} \right)dq_1 dq_2 $$
Hence the $3$ component of the curl can be written as
$$(\nabla \times \mathbf{V})_3 = \frac{1}{h_1 h_2} \left( \frac{\partial V_2 h_2}{\partial q_1} -\frac{\partial V_1 h_1}{\partial q_2} \right) $$
Similarly, other components can be evaluated and all the components can be assembled in the familiar determinant format.
$$\nabla \times \mathbf{V} = \frac{1}{h_1 h_2 h_3} \begin{vmatrix}
\mathbf{\hat{q_1}}h_1 & \mathbf{\hat{q_2}}h_2 & \mathbf{\hat{q_3}}h_3\\
\frac{\partial}{\partial q_1} & \frac{\partial}{\partial q_2} & \frac{\partial}{\partial q_3} \\
V_1 h_1 & V_2 h_2 & V_3 h_3 \\
\end{vmatrix}$$
Now the expression for the curl is ready. All we need to do is find the values of $h$ for the cylindrical coordinate system. This can be obtained, if we know the transformation between cartesian and cylindrical polar coordinates.
$$ (x,y,z) = (r\cos\phi, r\sin\phi, z)$$
Now the length element
$$ ds^2 = dx^2 + dy^2 + dz^2 = (d(r\cos\phi))^2 + (d(r\sin\phi))^2 + dz^2 $$
Simplifying the above expression, we get
$$ ds^2 = (dr)^2 + r^2(d\phi)^2 + (dz)^2 $$
From the above equation, we can obtain the scaling factors, $h_1 = 1$ , $h_2 = r$, $h_3 = 1$. Hence the curl of a vector field can be written as,
$$ \nabla \times \mathbf{V} = \frac{1}{r}
\begin{vmatrix}
\mathbf{\hat{r}} & r\mathbf{\hat{\phi}}& \mathbf{\hat{z}}\\
\frac{\partial}{\partial r} & \frac{\partial}{\partial \phi} & \frac{\partial}{\partial z} \\
V_r & r V_\phi & V_z \\
\end{vmatrix}
$$
Best Answer
[This answer doesn't add much that is new, but it became too painful trying to post is as a comment.]
One doesn't need to talk about differential forms, or integral identities, in order to prove the validity of vector calculus formulas in spherical coordinates (although both those points of view are very nice!). If $\vec{F}(x,y,z)$ is a vector field expressed in Cartesian coordinates, say, and $\vec{F}(r,\theta,\varphi)$ is the same vector field but now expressed in spherical coordinates, then the formula for $\nabla$ in spherical coordinates is determined by the requirement that $\nabla\cdot \vec{F}(x,y,z)$ and $\nabla\cdot \vec{F}(r,\theta,\varphi)$ correspond to one another as vector fields when you change from Cartesian to shperical coordinates.
Similarly, if $\vec{A}(x,y,z)$ and $\vec{B}(x,y,z)$ are a pair of vector fields expressed in Cartesian coordinates, with $\vec{A}(r,\theta,\varphi)$ and $\vec{B}(r,\theta,\varphi)$ being the same vector fields but now expressed in spherical coordiantes, then $\vec{A}\times \vec{B}$ (expressed in Cartesian coordinates) will correspond to $\vec{A}\times \vec{B}$ (expressed in spherical coordinates). So the left-hand side of the identity to be checked, when computed in Cartesian coordinates, will match (under the change from spherical to Cartesian coordinates) with the same expression when computed in spherical coordiantes. Similarly for the right-hand side of the identity. Thus the identity is equally valid in Cartesian or spherical coordinates. (Again, the key point is that the formulas for grad, div, and curl under a coordinate change are defined so as to make the above argument valid.)