So we have $$ \left(\frac{3x ^{3/2}y^3}{x^2 y^{-1/2}}\right)^{-2} $$
Yes you are right you can "flip" the fraction to remove the negative exponent.
$$ \left(\frac{x^2 y^{-1/2}}{3x^{3/2}y^3}\right)^2$$
Here you multiply every exponent by 2 to square the expression inside the brackets. This is where you went wrong here.
$$ \left(\frac{x^4 y^{-1}}{9x^{3}y^6}\right)$$
But $y^{-1}=1/y$
so the above is equal to
$$ \left(\frac{x^4}{9x^{3}y^7}\right)$$
Now cancel the $x^3$
$$ \left(\frac{x}{9y^7}\right)$$
And we're done.
You have put your finger precisely on the statement that is incorrect.
There are two competing conventions with regard to rational exponents.
The first convention is to define the symbol $a^x$ for $a > 0$ only. The symbol $\sqrt[n]{a}$ is defined for negative values of $a$ so long as $n$ is odd, but according to this convention, one wouldn't write $a^{1/n}$, for instance.
In defining $a^{p/q}$ to be $(\sqrt[q]{a})^p$, the author you quoted chose the fraction $p/q$ to be in lowest form so that the definition would be unambiguous. For example, $a^{10/15}$ is defined to be $(\sqrt[3]{a})^2$. However, it is preferable to define $a^{p/q}$ to be $(\sqrt[q]{a})^p$ in all cases and to prove that this definition is independent of the particular representation chosen for $p/q$; this is what more rigorous books tend to do. That is, you prove that if $p/q = r/s$, then $(\sqrt[q]{a})^p = (\sqrt[s]{a})^r$. There is no mention of lowest form.
The competing convention is to also allow $a^x$ to be defined for all $a \ne 0$ and all rational numbers $x = p/q$ that have at least one representation with an odd denominator. You then prove that $(\sqrt[q]{a})^p$ is independent of the particular representation $p/q$ chosen, so long as the denominator is odd. Thus you can write $a^{3/5} = (\sqrt[5]{a})^3 = (\sqrt[15]{a})^{9} = a^{9/15}$. All of that is fine. However, you cannot write $a^{6/10} = (\sqrt[10]{a})^6$, or even $a^{6/10} = \sqrt[10]{a^6}$. The number $a^{6/10}$ is well-defined, but to write down its definition, you must first select a fraction equivalent to $6/10$ that has an odd denominator, which could be $3/5$ or $9/15$ or something else. For $a^{1/2}$, this can't be done at all, so $a^{1/2}$ is undefined for $a < 0$.
The rules for exponents break down if you start allowing $a < 0$ and exponents that can't be written with an odd denominator. For example, the rule $a^{xy} = (a^x)^y$ is valid, but only so long as $x$ and $y$ are both rational numbers that can be written with an odd denominator. This is not the case if you write $a^1 = (a^2)^{1/2}$, despite the fact that both sides of the equation are defined since $a^2 > 0$.
Edit Reading the paper by Tirosh and Even, I was surprised to learn this matter has drawn serious attention from math educators.
A long time ago, I assumed that, apart from complex extensions, $a^x$ for non-integer $x$ should be defined only for $a > 0$. I reasoned that it made no sense to have a function $(-2)^x$ defined only for rational numbers $x$ with odd denominator. I objected strenuously to notations like $(-8)^{1/3}$.
But that was before I taught a calculus class, which is when I realized why some textbook authors are so happy to define $a^x$ for $a < 0$, following the second convention. The reason is that the formula $\frac{d}{dx}(x^r) = rx^{r-1}$ is perfectly valid for $x < 0$ and $r$ with odd denominator.
Best Answer
[THIS is only an intuitive argument!]
The definitions will make sense if you look at it this way...
This condition is true, since a fractional exponent is still an exponent. This can also be proved in calculus, and the fact that $a^r = e^{r \ln a}$.
Let's say you have a negative number (e.g. $-100$). Now, raise $-100$ to the $\frac 12$ power using the definition for a fractional exponent. You get: $$(-100)^{\frac 12} = \sqrt{-100}$$ which is imaginary. But, let's raise $-100$ to the $\frac 24$ power. You get: $$(-100)^{\frac 24} = \sqrt[4]{(-100)^2} = \sqrt[4]{10000} = 10$$ which is real. But, $\frac 12 = \frac 24$, so how could the two answers be different? So it is the mathematical convention to simplify the exponent first before doing anything else.
Let's say we have a negative number, $-64$. It is possible to have a cube root of $-64$, since every real number has a unique cube root ($\sqrt[3]{-64} = -4$). The index $n$ for the cube root is $3$, which is odd. $-64$ also has a unique 5th root ($-2.29739670999$) 7th root ($-1.81144732853$), etc. This shows that $\sqrt[n]a$ is defined for odd values of $n$.
Now, let us calculate the square root of $-64$. This number is $8i$, which is imaginary. So, $\sqrt[n]a$ will not be real if $n$ is even (the index of the square root is $2$.)