Problem 6 of the 1988 International Mathematical Olympiad notoriously asked:
Let $a$ and $b$ be positive integers and $k=\frac{a^2+b^2}{1+ab}$. Show that if $k$ is an integer then $k$ is a perfect square.
The usual way to show this involves a technique called Vieta jumping. See Wikipedia or this MSE post.
I can follow the Vieta jumping proof, but it seems a bit strained to me. You play around with equations that magically work out at the end. I don't see how anyone could have come up with that problem using that proof.
Is there a natural or canonical way to see the answer to the problem, maybe using (abstract) algebra or more powerful tools? In addition, how can someone come up with a problem like this?
Best Answer
At the heart of these so-called "Vieta-jumping" techniques are certain symmetries (reflections) on conics. These symmetries govern descent in the group of integer points of the conic. If you wish to develop a deeper understanding of these proofs then I highly recommend that you study them from this more general perspective, where you will find much beauty and unification.
The group laws on conics can be viewed essentially as special cases of the group law on elliptic curves (e.g. see Franz Lemmermeyer's "poor man's" papers), which is a helpful perspective to know. See also Sam Northshield's expositions on associativity of the secant method (both linked here).
If memory serves correct, many of these contest problems are closely associated with so-called Richaud-Degert quadratic irrationals, which have short continued fraction expansions (or, equivalently, small fundamental units). Searching on "Richaud Degert" etc should locate pertinent literature (e.g. Lemmermeyer's Higher descent on Pell Conics 1). Many of the classical results are couched in the language of Pell equations, but it is usually not difficult to translate the results into more geometric language.
So, in summary, your query about a "natural or canonical way to see the answer to the problem" is given a beautiful answer when you study the group laws of conics (and closely related results such as the theory of Pell equations). Studying these results will provide much motivation and intuition for generalizations such as group laws on elliptic curves.
See also Aubry's beautiful reflective generation of primitive Pythagorean triples, which is a special case of modern general results of Wall, Vinberg, Scharlau et al. on reflective lattices, i.e. arithmetic groups of isometries generated by reflections in hyperplanes.