This is not an answer, as I don't know how to evaluate the integral in terms
of elementary functions. However,
just in case someone else does...
I am assuming that $r \le R$ and that $h$ is 'large' so that it completely
surrounds the bored hole. I am also assuming that the big cylinder has its centerline along the $y$ axis and that the drilled hole is along the $x$ axis.
If we can compute the volume of the material of the original cylinder that was removed by drilling then we can compute the volume of the cylinder with a hole.
The volume of removed material is given by the volume of
$D_{r,R} = \{ (x,y,z) | z^2+y^2 \le r^2, z^2+x^2 \le R^2 \}$. This can be written as
$D_{r,R} = \{ (x,y,z) | z \in [-r,r], |x| \le \sqrt{R^2-z^2}, |y| \le \sqrt{r^2-z^2} \}$
so we can write
$\operatorname{vol}D_{r,R} = \int_{z=-r}^r \int_{y=-{\sqrt{r^2-z^2}}}^{\sqrt{r^2-z^2}} \int_{x=-{\sqrt{R^2-z^2}}}^{\sqrt{R^2-z^2}} 1 \ dx dy dz = 8 \int_0^r \sqrt{r^2-z^2} \sqrt{R^2-z^2} dz$.
As a sanity check, if we fix $r$, then
$\lim_{R \to \infty} {D_{r,R} \over 2 R} = \pi r^2$.
However, I don't think that this has an elementary solution.
A numerical solution is straightforward, if that suffices.
Then the volume of the drilled cylinder is
$\pi R^2 h-D_{r,R}$.
You don't say so explicitly, but your formulas imply that the
symbol $r$ represents the radius of the cylinder and $h$ is the height
of the cylinder.
There are many differently-shaped cylinders that can fit inside a
given sphere. If the radius of the sphere is $6$,
near one extreme, you have a very skinny cylinder whose
height is very close to $12$ and whose radius is close to zero.
Near the other extreme, you have a cylinder that is almost a flat disk
with a very small "height" (or thickness) $h$ and a radius that is
barely less than $6$.
We can even take the cylinder all the way to the extreme where it is
only a degenerate cylinder: $h=12$, $r=0$ describes a line segment that
just exactly fits in the sphere, and $h=0$, $r=6$ describes a flat disk
that also exactly fits in the sphere.
There's not much point in considering $r < 0$, because what is a cylinder
with a negative radius? (Actually there is a way to interpret such a thing,
but if you allow, for example, a cylinder of radius $-2$, it's identical to
a cylinder of radius $2$; so you have found no new cylinders but you have
to rewrite formulas such as $A(r) = 2\pi rh$ because the lateral area
of a cylinder is not less than zero. It's easier, and we don't miss
any answers, if we consider only cylinders such that $r \geq 0$.)
There is certainly no point in considering a cylinder with radius $r > 6$
if the radius of your sphere is $6$, because there is no way such
a cylinder could possibly fit inside the sphere.
You could also come to these conclusions by examining the formula
you found,
$$A(r)=2\pi r \cdot 2\sqrt{36-r^2},$$
because $r < 0$ gives a negative area (which can't be correct)
and $r > 6$ asks you to take the square root of a negative number,
which doesn't work.
But I think it's a good idea to have reasons based in the original
problem statement that say why a variable $r$ should have only a certain
possible set of values, because there's no guarantee that every
restriction that the problem statement requires will also be
enforced by whatever formulas we derive.
So we are OK with $r=0$ (if we accept a line segment as a degenerate cylinder of radius zero), we are OK with $r=6$ (if we accept a circle as a degenerate cylinder of height zero), and we are OK with
anything in between--just set $h= 2\sqrt{36-r^2}$, as you discovered,
and the cylinder will fit perfectly.
But $r < 0$ is useless to consider and $r > 6$ is impossible.
So $0 \leq r \leq 6$ describes every cylinder that we could ever want
to inscribe in a sphere of radius $6$.
Best Answer
Hint...use similar triangles $$\frac{a-h}{r}=\frac ab$$ to eliminate either $r$ or $h$ so you have a function of one variable.