You have a problem of order: The limits on the $z$ integral depend on $r$, and therefore it has to be done where $r$ is defined. In other words, the integral is $$\int_0^{2\pi}\int_2^4\int_{-\sqrt{a^2-(r-c)^2}}^{\sqrt{a^2-(r-c)^2}} rdzdrd\theta$$
And you are correct that $c = 3$ and $a = 1$
But as noted in the comments, this approach does not work well with the cuts. The problem is, the cuts are flat, which leads to complicated limits of integration for $r$ and/or $\theta$, which will undo the advantages of going to cylindrical representation ("polar" coordinates are in the plane - the 3D analogs are either "cylindrical", as you are using here, or "spherical").
Let's look at the cuts. The first is by the plane $y = -3$ (which is automatically parallel to the x-axis and to the z-axis). This plane is tangent to the "center ring" of the torus. The other plane is $x = 1$, which passes through the hole. Note that these two planes meet inside the torus. To set the up as an integral in $x, y, z$, you need to consider that for given values of $x$, there are in general two separate ranges of values for $y$ that need integrated over, and vice versa. The way to handle this is to break the torus up into quadrants and work each quadrant separately. Within each quadrant, there is only one range for $x$ or $y$ to be integrated. We can also do the same for $z$, limiting attention to $z \le 0$ and $z \ge 0$ separately. But in this case, the geometry is symmetric, so we will pick up the same volume on either side. Thus we can calculate only $z \ge 0$, then double it to get the whole volume.
The equation of the torus surface is $$z^2 +(\sqrt{x^2 + y^2} - 3)^2 = 1$$ So we can set up the integration over the entire first quadrant as
$$2\int_0^4\int_0^{\sqrt{16 - x^2}}\int_0^{\sqrt{1-(\sqrt{x^2+y^2}-3)^2}} dzdydx$$
After cutting, for each quadrant we have:
- Quadrant 1: $0 \le x \le 1, 0 \le y$
$$2\int_0^1\int_0^{\sqrt{16 - x^2}}\int_0^{\sqrt{1-(\sqrt{x^2+y^2}-3)^2}} dzdydx$$
- Quadrant 2: $x \le 0, 0 \le y$
$$2\int_{-4}^0\int_0^{\sqrt{16 - x^2}}\int_0^{\sqrt{1-(\sqrt{x^2+y^2}-3)^2}} dzdydx$$
- Quadrant 3: $x \le 0, -3 \le y \le 0$
$$2\int_{-4}^0\int_{-3}^{\sqrt{4 - x^2}}\int_0^{\sqrt{1-(\sqrt{x^2+y^2}-3)^2}} dzdydx$$
- Quadrant 4: $0 \le x \le 1, -3 \le y \le 0$
$$2\int_0^1\int_{-3}^{\sqrt{4 - x^2}}\int_0^{\sqrt{1-(\sqrt{x^2+y^2}-3)^2}} dzdydx$$
The $z$ integration is easy, but for the $y$ and $x$ integrations, you will need some trigonmetric substitutions.
The surface of the upper floor is $$\pi\left((R+r)^2-(R-r)^2\right)
=4\pi Rr$$ and of the lower floor $$\pi\left((R+\frac{\sqrt{5}}{3}r)^2-(R-\frac{\sqrt{5}}{3}r)^2\right)=\frac{4\sqrt{5}}{3}\pi Rr.$$
Since $$4\pi Rr+\frac{4\sqrt{5}}{3}\pi Rr=864000$$ and $r$ is known, we find $$R=\frac{864000}{4\pi\left(1+\frac{\sqrt{5}}{3}\right)r}\approx 131.31024<r$$
so no such torus can exist, thus you can get arbitrary non-consistent equations (and even $R<0$ too) assuming it exists (like "false implies anything").
Edit:
Let all the points mentioned above lie on a vertical plane passing through the center of symmetry $O$ (if the floors are horizontal). Let $O,C,A,B$ be on the upper floor and $O_1,F,E,G$ be on the lower one. $OA=R$ and $AC=AF=AD=AG=AB=r$, $AD\perp BC$, $E\in AD$, $AE=\frac 23 AD$ thus (by Pythagorean theorem) $EF=r\sqrt{1-\frac 49}=r\frac{\sqrt{5}}{3}$. $OAEO_1$ is a rectangle, hence $O_1E=OA=R$ thus
$$O_1G=O_1E+EG=O_1E+EF=R+r\frac{\sqrt{5}}{3},\\
O_1F=O_1E-EF=R-r\frac{\sqrt{5}}{3}.$$
Best Answer
I found it easiest to use cylindrical coordinates to set up the integrals needed for the center of mass. Before we do so, however, I set my coordinate system up as follows. I have positive $x$ coming out of the screen, positive $y$ going to the right, and positive $z$ up. In cylindrical coordinates $(r,\phi,z)$:
$$x = r \cos{\phi}$$ $$y=r \sin{\phi}$$
where we have the limits defining the region $\Omega$:
$$\phi \in \left [ -\frac{\pi}{2},\frac{\pi}{2} \right]$$ $$z \in [-R_2,R_2]$$ $$r \in \left [ R_1 - \sqrt{R_2^2-z^2}, R_1 + \sqrt{R_2^2-z^2}\right]$$
Also, for an object of constant mass density, the expression for the $x$ component of the center of mass is
$$\bar{x} = \frac{\displaystyle \int_{\Omega} d^3 \vec{x} \, x}{\displaystyle \int_{\Omega} d^3 \vec{x}}$$
(Note that, by symmetry, we have $\bar{y}=0$ and $\bar{z}=0$.) Let's first evaluate the denominator:
$$\begin{align} \int_{\Omega} d^3 \vec{x} &= \int_{-\pi/2}^{\pi/2} d\phi \, \int_{-R_2}^{R_2} dz \, \int_{R_1 - \sqrt{R_2^2-z^2}}^{R_1 + \sqrt{R_2^2-z^2}} dr \, r \\ &= \frac{\pi}{2}\int_{-R_2}^{R_2} dz \left [\left (R_1 + \sqrt{R_2^2-z^2} \right )^2 - \left (R_1 - \sqrt{R_2^2-z^2} \right )^2 \right ]\\ &= 4 \pi R_1 \int_0^{R_2} dz \, \sqrt{R_2^2-z^2}\\ &= \pi^2 R_1 R_2^2 \end{align}$$
Now we evaluate the center of mass:
$$\begin{align}\bar{x} &= \frac{1}{\pi^2 R_1 R_2^2} \int_{-\pi/2}^{\pi/2} d\phi \, \int_{-R_2}^{R_2} dz \, \int_{R_1 - \sqrt{R_2^2-z^2}}^{R_1 + \sqrt{R_2^2-z^2}} dr \, r^2 \cos{\phi} \\ &= \frac{4}{3 \pi^2 R_1 R_2^2} \int_0^{R_2} dz \, \left [\left (R_1 + \sqrt{R_2^2-z^2} \right )^3 - \left (R_1 - \sqrt{R_2^2-z^2} \right )^3 \right ]\\ &= \frac{8}{3 \pi^2 R_1 R_2^2} \int_0^{R_2} dz \,\left [3 R_1^2 \sqrt{R_2^2-z^2} + \left (R_2^2-z^2 \right )^{3/2} \right ] \\ &= \frac{8}{3 \pi^2 R_1 R_2} \left ( \frac{3 \pi}{4} R_1^2 R_2 + \frac{3 \pi}{16} R_2^3 \right )\end{align}$$
Simplifying, I get
$$\bar{x} = \frac{4 R_1^2+R_2^2}{2 \pi R_1}$$
ADDENDUM
As a quick note, in the limits as $R_2 \to 0$, we find that the center of mass becomes
$$\bar{x}=\frac{2}{\pi} R_1$$
which agrees with the center of mass of a uniform wire bent into a semicircle.