For the volume of the cone, fix a height $z$ between $0$ and $h$, and consider the radius $r(z)$ of the circular slice of the cone at that height. If you "flatten" the picture into two dimensions, you get two similar right triangles :
One has height $h$ and base $r$. The other has height $h-z$ and base $r(z)$. Taking ratios of these, you see that
$$
\frac{h}{r} = \frac{h-z}{r(z)}
$$
So
$$
r(z) = \frac{r(h-z)}{h}
$$
Now the area of that slice that you chose is $\pi r(z)^2$, so the area of the "washer" outside this slice is
$$
\pi r^2 - \pi r(z)^2
$$
when you "add" up these washers, you get
$$
\int_0^h \left [\pi r^2 - \pi \frac{r^2(h-z)^2}{h^2}\right] dz
$$
which gives the volume of the region outside of the cone.
Consider your cone lying in a xyz-space. Put the vertex of the cone at the origin, and imagine your base of cone lying in the plane $z=h$. Let $\alpha$ be apex angle so that $\tan\alpha=\cfrac{R}{h}$.
At height $z$ above the origin(vertex of cone), the radius of the base is $z\tan\alpha$. So lateral surface of the cone is represented by the region in $\mathbb{R}^3$, $R=\{(x,y,z):x^2+y^2=(z\tan\alpha)^2, 0\le z\le h\}$. To do the surface integral, we need to parameterize this, as follows:
$\gamma(r,\theta)=(r\cos\theta,r\sin\theta,\cfrac{r}{\tan\alpha})$, $r\in [0,h\tan\alpha]$, $\theta\in [0,2\pi)$
(the parameterization question asked for)
Now we find:
$\cfrac{\partial{\gamma}}{\partial{r}}=(\cos\theta,\sin\theta,\cfrac{1}{\tan\alpha})$
$\cfrac{\partial{\gamma}}{\partial{\theta}}=(-r\sin\theta,r\cos\theta,0)$
Then $\left|\cfrac{\partial{\gamma}}{\partial{r}}\wedge\cfrac{\partial{\gamma}}{\partial{\theta}}\right|=\left|\big(\cfrac{-r\cos\theta}{\tan\alpha},\cfrac{-r\sin\theta}{\tan\alpha},r\big)\right|=\cfrac{r\sec\alpha}{\tan\alpha}$.
Now do the surface integral:
$\iint\limits_R \,dx\,dy=\int\limits_{0}^{2\pi}\int\limits_{0}^{h\tan\alpha}\cfrac{r\sec\alpha}{\tan\alpha} \,dr \,d\theta=2\pi \cfrac{\sec\alpha}{\tan\alpha} \left[\cfrac{r^2}{2}\right]_0^{h\tan\alpha}=\pi h^2 \sec\alpha \tan\alpha=\pi h^2 \cfrac{R}{h} \cfrac{\sqrt{h^2+R^2}}{h}=\pi R\sqrt{h^2+R^2}$
Best Answer
Thinking geometrically can save a whole bunch of time here. Split the surface integral over the cone into two integrals, an integral over the circular base (call this surface $B$) and integral over the curved part of the cone's surface connecting base to apex (call this surface $C$).
Consider the surface integral over $C$ first. Note that the vector field whose surface integral we're evaluating is simply a scalar multiple of the radius vector. But since we placed the cone's apex at the origin, any radius vector from the origin to a point on the surface $C$ will also be a tangent vector to surface $C$ at that point. It follows that this surface integral vanishes, since the dot product of a surface tangent vector with the surface normal vector is zero at each point:
$$\iint_{C}\vec{F}\cdot\hat{n}\,\mathbb{d}S=\frac13\iint_{C}\vec{r}\cdot\hat{n}\,\mathbb{d}S=0.$$
Now for the other part of the surface integral, $\iint_{B}\vec{F}\cdot\hat{n}\,\mathbb{d}S$. Again, $\vec{F}=\frac13\vec{r}$, and since we've oriented the cone so that its axis of symmetry coincides with the $z$-axis we can immediately see that the unit normal vector to this part of the cone's surface is simply the Cartesian unit vector $$\hat{n}=\hat{z}=\langle0,0,1\rangle$$. Parametrizing this surface is also straightforward:
$$\vec{r}(\rho,\phi)=\langle\rho\cos{\phi},\rho\sin{\phi},h\rangle,~~~\text{where }0\leq\rho \leq a,0\leq\phi\leq 2\pi.$$
Hence,
$$\vec{F}\cdot\hat{n}=\frac13 h\\ \implies \iint_{B}\vec{F}\cdot\hat{n}\,\mathbb{d}S = \frac{h}{3}\iint_{B}\,\mathbb{d}S = \frac{h}{3}(\pi a^2).~~~QED$$