At the heart of these so-called "Vieta-jumping" techniques are certain symmetries (reflections) on conics. These symmetries govern descent in the group of integer points of the conic. If you wish to develop a deeper understanding of these proofs then I highly recommend that you study them from this more general perspective, where you will find much beauty and unification.
The group laws on conics can be viewed essentially as special cases of the group law on elliptic curves (e.g. see Franz Lemmermeyer's "poor man's" papers), which is a helpful perspective to know. See also Sam Northshield's expositions on associativity of the secant method (both linked here).
If memory serves correct, many of these contest problems are closely associated with so-called Richaud-Degert quadratic irrationals, which have short continued fraction expansions (or, equivalently, small fundamental units). Searching on "Richaud Degert" etc should locate pertinent literature (e.g. Lemmermeyer's Higher descent on Pell Conics 1). Many of the classical results are couched in the language of Pell equations, but it is usually not difficult to translate the results into more geometric language.
So, in summary, your query about a "natural or canonical way to see the answer to the problem" is given a beautiful answer when you study the group laws of conics (and closely related results such as the theory of Pell equations). Studying these results will provide much motivation and intuition for generalizations such as group laws on elliptic curves.
See also Aubry's beautiful reflective generation of primitive Pythagorean triples, which is a special case of modern general results of Wall, Vinberg, Scharlau et al. on reflective lattices, i.e. arithmetic groups of isometries generated by reflections in hyperplanes.
Here is a bit of a hacky answer to your question in the affirmative
You observe that $k = \mathrm{gcd}(a,b)^2$. After plodding through various resources, it is because one can Viete Jump:
$$ (a,b) \mapsto \big(a_1, b_1\big) \mapsto \dots \mapsto (k,0) $$
and the gcd is conserved. Once I agree with you, let $a = \text{gcd}(a,b) \, c$ and $b = \text{gcd}(a,b) \, d$ so that
\begin{eqnarray} ab+1 &=& \frac{a^2+b^2}{\mathrm{gcd}(a,b)^2}= c^2 + d^2 \\
&=& \,\text{gcd}(a,b)^2 \, cd + 1
\end{eqnarray}
In this way there are two conditions $c, d$ might solve (where the two $\square$ are different) and $\text{gcd}(c,d)=1$:
\begin{eqnarray*}
c^2 - \square \, cd + d^2 &=& 1 \\
c^2 + d^2 &=& \square
\end{eqnarray*}
Hopefully these two equations leads to a contradiction.
Added 11/15 The answer is definitely no. Let $k = \mathrm{gcd}(a,b)$ We are trying to solve in integers:
\begin{eqnarray} c^2 - k\; cd + d^2 &=& 1 \\
c^2 + d^2 = \square
\end{eqnarray}
As I learned, the first can be solve with $(c,d) = (1,0)$ or $(k,1)$ and there are an infinite family of solutions using consecutive terms of a recursive sequence [2, 3]
$$ x_{n+1} = k \, x_n + x_{n-1} $$
There are sometimes ways to link Pythagorean triples to Pell equations [1] (Modular Tree of Pythagoras)
$$ x_{n+1}^2 + \frac{1}{2} x_n^2 < \sqrt{x_{n+1}^2 + x_n^2 } = x_{n+1}^2 \sqrt{1 + (x_n/x_{n+1})^2} < x_{n+1}^2 + \frac{1}{2} x_n^2 + 1$$
This cannot be an integer. So any time we solve the Pell equation, we cannot also solve Pythagoras. $\quad\quad\square$
Old Answer
This is discussed on Wikipedia's article on Vieta Jumping:
Nobody of the six members of the Australian problem committee could solve it. Two of the members were George Szekeres and his wife, both famous problem solvers and problem creators. [...] The problem committee submitted it to the jury of the XXIX IMO marked with a double asterisk, which meant a superhard problem, possibly too hard to pose. After a long discussion, the jury finally had the courage to choose it as the last problem of the competition. Eleven students gave perfect solutions.
Among the eleven was Bau Chau Ngรด (Fields Medal 2010). His work on the Fundamental Lemma also has a jumpy flavor [1, 2, 3] but is quite advanced.
The discussion on YouTube is helpful as well. These videos give a thorough discussion of different ways to solve
These may not directly solve your problem but provide historical context and indicate possible strategies.
In the Wikipedia article, the example of Viete Jumping is IMO 1988/6 -- the same as asked in the question:
Let $a,b$ be positive integers such that $ab+1$ divides $a^2 + b^2$ show that $ \frac{a^2 + b^2}{ab+1}$ is a perfect square.
and the solution goes in three steps
#1 Let $a, b \geq 0$ be solutions to $\frac{a^2 + b^2}{ab+1} = k$ such that $k$ is not a perfect square: $k \neq \square$
#2 Starting from $(a,b)$ we can try to generate another solution $(x,b)$ which solves the quadratic equation:
$$ x^2 - kb\, x +(b^2 - k) = 0 $$
The map $(a,b) \mapsto (a_1,b)$ is our Vieta jumping Since both $a, a_1$ are acceptable solutions we have:
$$ (x-a)(x-a_1) = x^2 - (a + a_1) x + aa_1 = 0$$
By the Viete equations (comparing the coefficients. We find out two things:
#3 If $a \geq b$ we can deduce that $a_1 \geq 0$ (is positive) and additionally $ a > a_1 \ge 0 $
Summary We've show that given two positive numbers $a,b$ solving $\frac{a^2+b^2}{ab+1}=k$ with $k \neq \square$ we can always find another solution $(a_1,b)$ solving the same equation with $a > a_1$.
Then the Viete jump consists of a map:
$$ (a,b) \mapsto \left\{\begin{array}{rc}
(b,a) & \text{ if } a \leq b \\
(\frac{b^2-k}{a},b) & \text{ if } a \geq b
\end{array}\right.$$
While this does not solve your problem -- to show that $ab+1 \neq \square$ -- it does indicate possibly where to start and some possible resources.
A quick use of Bezout formula shows that $ab+1$ should also divide
$$ \big[(a^2 + b^2) + 2(ab+1)\big] +
\big[(a^2 + b^2) - 2(ab+1)\big] = (a+b)^2 +(a-b)^2 $$
and this could lead to your contradiction.
Best Answer
Geometric solution to Q6: Consider a rectangle with sides ๐, ๐ Its diagonal has length $$ \sqrt{a^2+b^2} $$
This diagonal is a side of square A
Area A $$= a^2+b^2$$ Area B = $$ ๐. \sqrt{a^2+b^2}$$ Geometrically Q6 becomes equation (1) , $$ ๐ = Area A / Area B $$
Length ๐ is projected onto the side square A to yield a length, ๐
Now, $$ ๐ = ๐/cosฯด $$ And $$cosฯด = ๐/โ{a^2+b^2}$$ So $$๐ = ๐/๐. โ{a^2+b^2}$$ Area B $$= ๐/๐. {a^2+b^2}$$ Then from (1) $$๐ = ๐/๐ $$ (2)
Now if Area B = $$๐๐ + 1$$ as in Q6 then $$๐๐ + 1 = a/b.{a^2+b^2}$$ $$b^2 + ๐/๐ = {a^2+b^2}$$ $$๐ = a^3$$ From (2) $$๐ = a^2,$$ a perfect square