First, note that in (2), the naturals $\Bbb N$ are referred to as "nonnegative integers" (page one) which certainly includes $0$. But that is not the crux of the confusion here.
If $c=0$ then it doesn't contradict that $k=b^2$.
Nobody claims that "$c=0$" contradicts "$k=b^2$." Rather, it contradicts our original hypothesis that the integer $k$ is not a perfect square (clearly $b^2$ is a square!). The flow of the argument goes like this:
- Hypothesis (H): $k$ is not a perfect square and $S$ is nonempty.
- Givens (G): Wlog, let $(a,b)\in S$ with $a>b$ and $a+b$ minimal among $S$.
- Deduction 1: Quadratic formula says $\frac{x^2+b^2}{xb+1}=k$ has a root $x=c$ in addition to $x=a$.
- Deduction 2: By Vieta's $c=kb-a$ so $c$ is an integer.
- Hypothesis A: $c=0$.
- Deduction: Plugging $x=c$ the equation gives us $b^2=k$
- Contradiction: This contradicts our top-level assumption that $k$ is not a square.
- Deduction 3: $c\ne 0$.
- Hypothesis B: $c<0$.
- Deduction: $c^2-kbc+b^2-k\ge c^2+k(1)+b^2-k=c^2+b^2>0$.
- Contradiction: But the above is $0$, as it is $c$ plugged into the quadratic.
- Deduction 4: $c>0$, and so $(c,b)\in S$.
- Deduction 5: Since $a>b$, $b^2-k<a^2$, so $c=\frac{b^2-k}{a}<a$ hence $c+b<a+b$.
- Contradiction: Deduction 5 contradicts the given that $a+b$ was minimal among pairs.
- Deduction X: Either $S$ is empty or $k$ is a perfect square.
This is just a rough outline that concisely presents the structure of the argument. Notice that this amounts to a proof-by-contradiction which contains proofs-by-contradiction. Let me know if you have any questions about the dependencies between these hypotheses and deductions!
Remark. Technically in Deduction 1 we don't know whether or not $c\ne a$, but we don't need to know either. The rest of the argument works just the same, until Deduction 5 where we learn that $c+b<a+b$, which is impossible if $c=a$ (not that it matters!).
I suspect your paraphrasing (or the book!) has a minor typo. Let me put it in bullet points.
- Fix $k$ to be some positive integer. Consider the set $Q$ of pairs $(a,b)$ such that $a\ge b\color{Red}>0$ and $$\frac{a^2+b^2}{ab+1}=k.$$
- Then $a$ is a root of the quadratic equation $\frac{x^2+b^2}{xb+1}=k$. Let $c$ be the other root.
- By Vieta's formulas, $a+c=bk$, so $c$ is an integer.
- We have $k(bc+1)=a^2+b^2>0$, so we must have $bc\ge -1$. Since $b> 0$, we have $c\ge 0$.
- Suppose $c>0$. Then the product of roots is $ac=b^2-k<b^2$, which is impossible unless $b>c$, and if $b>c$ then the valid pair $(b,c)\in Q$ has $c$ smaller than $b$ in $(a,b)$, contra minimality.
- Therefore, $c=0$ and $k=b^2$ is a square. Since our choice of $k$ was arbitrary, whenever $\frac{a^2+b^2}{ab+1}$ is a positive integer it must also be a perfect square.
Don't confuse this argument, say (3), with the arguments in the original two links.
- Both (1) and (2) assume $k$ is not a perfect square in order to derive a contradiction, while in (3) this is not assumed (though its negation is derived nonetheless).
- In (1), $S(k)$ is defined by strictly positive pairs, and the pairs are not ordered by size. We deduce $c\ne0$ here by invoking the assumption $k$ is nonsquare, and thus $(b,c)\in S(k)$.
- In (2), $S$ is defined to possibly contain pairs with one or both being $0$, though it also doesn't force the pairs to be ordered - the order of $A,B$ is taken wlog. We assume by hypothesis that no pair in $S$ has a $0$ and obtain a contradiction, so it does contain a $0$, implying $k$ is square.
- In (3), $Q$ contains strictly positive pairs, and they are ordered by size. Similar to (2) we prove that (using contradiction on minimality) $k=b^2$, even though $(b,0)\not\in Q$.
The arguments are subtly different and go about the proof in different ways. I would have advised you to stick to one of the proofs and understand that one before moving on to another, let alone two others, but the cat is out of the bag now I guess.
Concerning your perception of ambiguity, let me just say this. We desire to say that whenever $\frac{a^2+b^2}{ab+1}$ is a positive integer, it is a perfect square. In order to prove this, we could fix $k$ positive and consider all positive, or alternatively nonnegative, pairs $a,b$ such that the fraction equals $k$, and go from there; in the end our desired claim follows because our choice of $k$ to fix was arbitrary.
Note that in (3), the set $Q$ does not contain every pair $x,y$ such that $\frac{x^2+y^2}{xy+1}=k$. It only contains the positive ones. We prove that either $(b,c)\in Q$ and get a contradiction on minimality, or else $c=0$ and plugging it in gives $k=\frac{0^2+b^2}{b\cdot 0+1}=b^2$ is a square. Again, it does not matter that $(b,0)\not\in Q$, the computation is valid regardless because $c=0$ is still a root of $\frac{x^2+b^2}{xb+1}=k$, just not one inside $Q$.
Best Answer
Hint: You are almost done. Observe that $$1 + 10^{2n} - 2\cdot10^n = (10^n - 1)^2.$$