Suppose that there were two distinct real numbers $y_1,y_2>0$ such that $y_1^n=x$ and $y_2^n=x$.
Because $y_1\neq y_2$, without loss of generality we have that $y_1<y_2$ (in other words, if it were instead the case that $y_2<y_1$, just relabel them). But as is pointed out in Rudin's argument, this implies that $y_1^n<y_2^n$, and therefore $x<x$, which is a contradiction; therefore our assumption that there were two distinct real numbers whose $n$th power was $x$ was false. Therefore there can be at most one positive real number $y$ such that $y^n=x$.
Here is my interpretation of how Rudin argued the first point in your question (to contradict the case that $y^{n} < x$).
The idea is to find a $y^*$ that would lie in the "gap" of numbers created if $y$ were indeed less than the "true $\sqrt[n]{x}$". (Here $y$ refers to $y=\sup E$ where $E=\{t:t^n < x, t\in\mathbb{R^+}\}$ as defined by Rudin.)
In other words, we are trying to construct such a $y^*$ that has the two properties $(y^*)^n < x$ and $y^* > y$. The first property says that $y^* \in E$, the set which $y$ bounds above. The second property contradicts the fact that $y$ is an upper bound of $E$, since $y^* \not\le y$ and $y^* \in E$.
To construct $y^*$ we need to find a suitable $h$ such that $y^*=y+h$.
Intuitively, one would like to let $h$ be a positive quantity less than the difference $\sqrt[n]{x} - y$. However, we have not yet shown the existence of $\sqrt[n]{x}$ so this complicates our approach.
One alternative is to look at the difference $x - y^n$, which is a valid expression at this point in the proof. Graphically, this approach can be thought of as picking the position of $y+h$ on the horizontal axis based on the function value $(y+h)^n$ on the vertical axis.
In other words, instead of specifying $h$ directly we will try to specify $h$ in terms of what $y+h$ maps to under exponentiation by $n$.
Hence, we look for an $h$ such that $(y+h)^n - y^n < x - y^n$.
(Again, think of comparing function values of the curve $f(t) = t^n$ on a graph. The left-hand side $(y+h)^n - y^n$ is some positive quantity smaller than the height of the vertical "gap" assumed to exist between $y^n$ and $x$ on the graph.)
(Since $f(t) = t^n$ is a strictly increasing function (for positive $t$), inequalities are preserved, so the hypothetical horizontal "gap" (in which we were originally interested) will correspond with a vertical "gap" after this transformation. Intuitively, (pretending $\sqrt[n]{x}$ is defined) we could write,
$$
\sqrt[n]{x} - y > 0 \iff \sqrt[n]{x} > y \iff f(\sqrt[n]{x}) > f(y) \iff \sqrt[n]{x}^n > y^n \iff x > y^n \iff x - y^n > 0
$$
with the following justifications for each of the double arrows (in order from left to right): (1) addition/subtraction by $y$, (2) $f$ is strictly increasing, (3) definition of $f$, (4) definition of $\sqrt[n]{x}$, (5) addition/subtraction by $y^n$.)
Using the observation Rudin made$^1$ we can inject another expression into the inequality to get:
$(y+h)^n - y^n < hn(y+h)^n < x - y^n$
(This "injection" is okay for now because we have technically not defined $h$ yet, but instead are still working backwards to specify $h$.)
The trouble with this inequality is that the middle expression contains $h$ inside a binomial term raised to the power of $n$, which makes it hard to algebraically isolate. I believe Rudin makes the convenient assumption at this point that $h$ is small — in other words $0 < h < 1$ — in order to complete his definition of $h$.
We can now inject another expression into the inequality to yield:
$(y+h)^n - y^n < hn(y+h)^n < hn(y+1)^n < x - y^n$
Isolating just the rightmost two expressions and rearranging gives the final inequality:
$h < \frac{x - y^n}{n(y+1)^n}$
End Note
- Rudin's observation:
$b^n - a^n = (b-a)(b^{n-1} + ab^{n-2} + ... + a^{n-2}b + a^{n-1})$
so
$b^n - a^n < (b-a)n(b^{n-1})$
in the case $0 < a < b$.
(Rudin let $b = y+h$ and $a=y$ in the proof.)
Best Answer
I grabbed my copy of "Principles of Mathematical Analysis" and I believe your confusion is justified. Rudin made a (small) mistake in the proof.
Rudin is simply trying to show that $E$ is non-empty and bounded above in this first part of the proof, which is hinted at by the fact that $t$ plays no further part afterwards. To that end he is trying to construct a number $t$ which would qualify to live in $E$. He goes ahead and constructs this $t$ as follows: $$ t = \frac{x}{x+1}. $$ From this we already have that $t < x$, since $0 < \frac{1}{x+1} < 1$, but we want $t^n < x$. Luckily for us, from the way $t$ has been constructed we know $0 < t < 1$. This means (and here is where Rudin made his mistake), $t^n \leq t < x$. The equality happens exactly when $n = 1$ as you pointed out. We still have $t^n < x$ regardless, and so the conclusion is still valid: $E$ is non-empty.
He makes the same mistake, again forgetting about $n = 1$, when goes about showing that $E$ is bounded above. A strict inequality must be replaced by a non-strict one. However, just like in the case above we are rescued by the fact that the claim $t^n \geq t > x$ still means that $t^n > x$.
However, there is also a much larger mistake. We also require $y > 0$ for the theorem to be true. I was honestly surprised Rudin didn't specify this in his statement of the theorem. He uses it in the proof.