My son presented me with an interesting problem:
Of all possible polygons inscribed in a circle of radius $R$, find the
one that has the sum $S$ of squared side lengths maximized:
$S=a_1^2+a_2^2+\dots+a_n^2$, with $a_i$ representing the length of the
$i$-th side. The number of sides is not fixed, you should consider all
triangles, quadrilaterals, pentagons…
It's not that complicated, at least in the beginning. It's easy to show that the optimal polygon (with $n>3$) cannot have obtuse ($>90^\circ$) angles. For example, if such an angle $A_{i-1}A_{i}A_{i+1}$ exists, by cosine theorem:
$$|A_{i-1}A_{i}|^2+|A_{i}A_{i+1}|^2<|A_{i-1}A_{i+1}|^2$$
So if you drop vertex $A_i$, you get a polygon with a bigger $S$. This quickly eliminates all polygons with $n>4$.
All candidate polygons with $n=4$ must be rectangles and if their sides are $a$ and $b$, the sum $S$ is $2a^2+2b^2=8R^2$. So with respect to $S$ all rectangles inscribed in the circle are equivalent. In fact, a right triangle with sides $a$, $b$ and $2R$ has the same $S$ as any inscribed rectangle.
But maybe there is an inscribed triangle with $S>8R^2$. I was able to show that for an inscribed triangle with sides $a,b,c$ and $b\ne c$, an isosceles triangle with all acute angles and base $a$ has better value of $S$. So the optimal triangle must be isosceles. Looking from all three sides, the only possible solution is the equilateral triangle and the sum $S$ in that case is $9R^2$.
However, to prove that fact I had to use trigonometry which is not so complicated (and I can present it here if you want so), but it leaves impression that there has to be some simpler explanation why the equilateral triangle is the best choice. My trigonometry proof takes a few lines of text, I want something more elegant.
Just an idea: if you draw lines through the center of the circle perpendicular to the sides of a triangle and denote the pedal lengths with $h_a,h_b,h_c$, it's easy to show that in order to maximize $a^2+b^2+c^2$ you have to minimize $h_a^2+h_b^2+h_c^2$. And then – what?
EDIT: I want to present the part of the proof that I don't like. Take an arbitrary triangle $ABC$ with sides $a,b,c$ inscribed in a circle. Consider side $a$ fixed and play with angle $\gamma$ to get different values of $b,c$. I want to prove that isosceles triangle $BCA_1$ has bigger $S$ than any other triangle with one side equal to $a$.
$$b=2R\sin\frac{\pi-\alpha+\gamma}{2}=2R\cos\left(\frac\alpha2-\frac\gamma2\right)$$
$$c=2R\sin\frac{\pi-\alpha-\gamma}{2}=2R\cos\left(\frac\alpha2+\frac\gamma2\right)$$
$$b^2=4R^2\cos^2\left(\frac\alpha2-\frac\gamma2\right)=2R^2(1+\cos(\alpha-\gamma))$$
$$c^2=4R^2\cos^2\left(\frac\alpha2+\frac\gamma2\right)=2R^2(1+\cos(\alpha+\gamma))$$
$$b^2+c^2=4R^2+2R^2(\cos(\alpha-\gamma)+\cos(\alpha+\gamma))=4R^2(1+\cos\alpha\cos\gamma)$$
And this sum achieves maximum obviously for $\gamma=0$, or for $A\equiv A_1$. So for any given side $a$, $b$ and $c$ must be of equal. But you can look at the optimal triangle from sides $b$ and $c$ as well. The only triangle which has no better option is equilateral triangle.
EDIT 2: This “moving vertex” procedure can be repeated infinite number of times and the result is an equilateral triangle! Check excellent proof by Noah Schweber here.
Best Answer
Yes, the maximal sum is the one of the equilateral triangle, that is $9R^2$.
Since Prove that in any triangle $ABC$, $\cos^2A+\cos^2B+\cos^2C\geq\frac{3}{4}$ then $$ \sin^2 A+\sin^2 B+\sin^2 C=3-\cos^2 A-\cos^2 B-\cos^2 C\leq \frac{9}{4} $$ where $A$, $B$ and $C$ are non negative numbers such that $A+B+C=\pi$. Hence, for any inscribed triangle, the sum of the squares of the sides is $$(2R\sin A)^2+(2R\sin B)^2+(2R\sin C)^2\leq 9R^2.$$