The part of the statement of the Fundamental theorem of calculus on which the proof of the substitution theorem relies has no requirement that the change of variable be a bijection. This is a common error. (I made it myself until I first taught Calculus.) (I'll return to this point at the end of answering your Question because this Question gives the appearance that bijectivity is required.)
Another common error is to believe that, during trigonometric substitution of a definite integral, one "pushes" the bounds of integration from the old integral to the new integral along the change of variable. This is true when one uses the substitution theorem in the forward direction and false when used in reverse, as it is in trigonometric substitution. One must "pull" the bounds along the inverse function of the substitution (with whatever domain restriction it has) to the new bounds. In your case, $x = r \sin \theta$ gives the inverse $\theta = \arcsin(x/r)$. So you have no choice but to find the $\theta$s
\begin{align*}
\theta_{\text{lower}} &= \arcsin(-1) = -\pi/2 \\
\theta_{\text{upper}} &= \arcsin(1) = \pi/2 \text{.}
\end{align*}
You could use a different restriction of the domain of sine for your arcsine, but to apply the Fundamental Theorem of Calculus, this choice of inverse function must be continuous (hence defined) on the interval of integration. There is no such restriction giving an arcsine that contains the interval $[-5\pi/2, 5\pi/2]$. This is where bijectivity enters the picture -- to ensure the inverse function exists over the entire interval of integration. (And it only enters the picture when applying the theorem in reverse.)
We need to fix the exceedingly common algebra error in your derivation. In particular, $\sqrt{a^2} = |a|$ for any $a$. Let's take that square root correctly.
\begin{align*}
I &= \int_{-r}^r \; \sqrt{r^2 - x^2} \,\mathrm{d}x
& & \hspace{-1in}\begin{bmatrix}
x = r \sin \theta \\
\mathrm{d}x = r \cos \theta \,\mathrm{d}\theta \\
\theta = \sin^{-1}(x/r)
\end{bmatrix} \\
&= \int_{\sin^{-1}(-r/r)}^{\sin^{-1}(r/r)} \; \sqrt{r^2 - (r \sin \theta)^2}\, r \cos \theta \,\mathrm{d}\theta \\
&= \int_{-\pi/2}^{\pi/2} \; \sqrt{r^2(1 - \sin^2 \theta)}\, r \cos \theta \,\mathrm{d}\theta \\
&= \int_{-\pi/2}^{\pi/2} \; \sqrt{r^2 \cos^2 \theta}\, r \cos \theta \,\mathrm{d}\theta \\
&= \int_{-\pi/2}^{\pi/2} \; |r \cos \theta| r \cos \theta \,\mathrm{d}\theta \\
&= \int_{-\pi/2}^{\pi/2} \; |r| |\cos \theta| r \cos \theta \,\mathrm{d}\theta \\
&= \int_{-\pi/2}^{\pi/2} \; |\cos \theta| r^2 \cos \theta \,\mathrm{d}\theta & & \hspace{-1in}[ r \geq 0 ] \\
\end{align*}
Recall that cosine is nonnegative on our interval of integration, quadrants I and IV and the quadrantal angle $0$.
\begin{align*}
I &= r^2 \int_{-\pi/2}^{\pi/2} \; \cos^2 \theta \,\mathrm{d}\theta \\
&= r^2 \left.\left( \frac{\theta}{2} + \frac{\sin(2\theta)}{4} \right) \right|_{\theta = -\pi/2}^{\pi/2} \\
&= r^2 \left( \left( \frac{\pi}{4} + \frac{\sin(\pi)}{4} \right) - \left( \frac{-\pi}{4} + \frac{\sin(-\pi)}{4} \right) \right) \\
&= r^2 \left( \frac{\pi}{4} + \frac{\pi}{4} \right) \\
&= \frac{1}{2} \pi r^2 \text{.}
\end{align*}
An actual statement of the substitution theorem (footnote) is this:
Let $\varphi:[a,b] \rightarrow I$ be a differentiable function with a continuous derivative, where $I \subseteq \Bbb{R}$ is an interval. Suppose that $f:I \rightarrow \Bbb{R}$ is a continuous function. Then, if $u = \varphi(x)$
$$ \int_a^b \; f \left( \varphi(x) \right) \varphi'(x) \,\mathrm{d}x = \int_{\varphi(a)}^{\varphi(b)} \; f(u) \,\mathrm{d}u \text{.} $$
That English Wikipedia article also explains why trigonometric substitution is a little different from normal substitution.
The formula is used to transform one integral into another integral that is easier to compute. Thus, the formula can be read from left to right or from right to left in order to simplify a given integral. When used in the former manner, it is sometimes known as $u$-substitution or $w$-substitution in which a new variable is defined to be a function of the original variable found inside the composite function multiplied by the derivative of the inner function. The latter manner is commonly used in trigonometric substitution, replacing the original variable with a trigonometric function of a new variable and the original differential with the differential of the trigonometric function.
Let me give two examples to really clarify what is meant by right-to-left and left-to-right. First, left-to-right:
$$ J = \int_{-2}^{1} \frac{2 x \, \mathrm{d}x}{\sqrt{x^2 + 1}} \text{.} $$
Here, we choose $u$ to stand for some combination of the dummy variable, making parts of the integrand simpler and we hope that the resulting differential will consume part of the integrand, leaving something simpler. So we set $u = x^2 + 1$. That is, $\varphi(x) = x^2 + 1$. This is not a bijection. $\varphi(1) = \varphi(-1)$. Then, $\mathrm{d}u = 2 x \,\mathrm{d}x$. That is, $\varphi'(x) = 2x$. We are matching to the left side of the substitution equation. So we apply the theorem and obtain the version on the right-hand side of the equation:
$$ J = \int_{\varphi(-2) = 5}^{\varphi(1) = 2} \frac{\mathrm{d}u}{\sqrt{u}} \text{,} $$
which is easy to finish. Now, right-to-left:
$$ K = \int_{-r}^r \; \sqrt{r^2 - x^2} \,\mathrm{d}x \text{.} $$
Here, we set $x = r \sin \theta$, making no attempt to capture some combination of the dummy variable and making no attempt to capture a piece of the integrand by means of the differential. We start with the simpler looking integral on the right of the equation in the theorem and replace it with the elaborated integral on the left. This is why the inverse function has to work on the entire interval, because we have to be able to carry the endpoints of integration along the inverse function, to the endpoints of the interval $I$ (notation from the theorem).
When using the theorem right-to-left, one applies $\varphi^{-1}$ to the endpoints. In your Question, there is no choice of $\varphi^{-1}$ that gives an interval $I$ with the endpoints $-5\pi/2$ and $5\pi/2$. The maximal interval for any inverse of sine has width $\pi$.
(footnote) One might be (justifiably) concerned that the statement of the theorem at the English Wikipedia is incomplete.
Rogowski et al. "Calculus: Early Transcendentals, 4th ed.", p. 340:
Theorem 1: The Substitution Method: If $F'(x) = f(x)$, and $u$ is a differentiable function whose range includes the domain of $f$, $$ \int f(u(x))u'(x)\,\mathrm{d}x = F(u(x)) + C \text{.} $$
(Since $F$ is an antiderivative of $f$, the object on the right is $\int f(u) \,\mathrm{d}u$.)
No bijectivity requirement here. You will also not find a bijectivity requirement in other careful statements of the theorem because there is no explicit or implicit bijectivity requirement in the forward direction. Bijectivity is only relevant in the reverse direction and is expressed implicitly by means of the interval $I$, in the English Wikipedia quote, and in Rogowski by writing $F(u(x))$, which requires the domain of $F$ to include the range of $u$, which is given to include the domain of $f$. (Recall that in a definite integral, we may replace "$f$" with a function that is undefined outside the interval of integration.)
Best Answer
Use walli's formula given here to do it faster Reduction formula for integral $\sin^m x \cos^n x$ with limits $0$ to $\pi/2$
$I=3\displaystyle\int_{0}^{\pi} \sin^2t\cos^4t\ dt=6 \displaystyle\int_{0}^{\pi/2} \sin^2t\cos^4t\ dt=6\times \dfrac{(2-1)(4-1)(4-3)}{6(6-2)(6-4)}\times \dfrac{\pi}{2}=\dfrac{3\pi}{16}$
your mistake : Use of Power rule in wrong way
we know,
$\displaystyle\int x^m \ dx=\dfrac{x^{m+1}}{m+1}+C$ but this is not analogous to
$3 \displaystyle\int \cos^6t\ dt\neq\dfrac{3}{7}cos^7t + C $ as you've written in comments.
but ,
$\displaystyle\int \cos^6t\ d(cost)\ =-\displaystyle\int \cos^6t \ sint\ dt \ =\dfrac{3}{7}cos^7t + C $