Probably the author does not intend to say that the expression for $l_4$ follows from the formula. Actually, it does, sort of. We have $l_2=2R$, and substitution in the formula gives $l_4=R\sqrt{2}$. We have used the formula starting from a "degenerate" polygon. It happens to work, but is liable, for good reason, to cause confusion.
However, $l_4$ can be computed directly, using basic geometry. The diagonal of the square inscribed in the circle is $2R$. So if $s$ is the length of a side of the square, then by the Pythagorean Theorem
$$s^2+s^2=(2R)^2=4R^2,$$
so $s^2=2R^2$ and therefore $s=R\sqrt{2}$.
Luckily I hang out with a bunch of people on Twitter and Mastodon who are way smarter than me, and they came up with some great ideas.
The core solution here is to concatenate one of the lists being compared to itself. For two equal-length sets $A$ and $B$, if $A \subset B^\frown B$ then $A$ must be the same sequence of $B$ with some rotation. This reduces the problem to a string search.
To demonstrate:
$$A = [1,2,3,5,2]$$
$$B = [3,5,2,1,2]$$
$$B' = B^\frown B = [3,5,2,\mathbf{1,2,3,5,2},1,2]$$
$$A \subset B'$$
For those not familiar with the notation (and for me when I inevitably come back to this answer in a year and I've forgotten set notation) the approach is essentially:
let a = "12352";
let b = "35212";
let b2 = b + b;
return b2.includes(a);
This can be optimised in a few ways.
First, the concatenation does not have to use $O(2n)$ memory, since the search can be performed on the original set by computing the search pointer (or array index) modulo the set length.
A KMP string search has a worst-case performance of $O(m)$ pre-processing and $O(n)$ matching, where $m$ is length of the string we're searching for and $n$ is the length of the string we're searching in. Since all polygons in each hash group need to be compared to each other, the pre-processing can be reused for repeated comparisons.
The KMP approach can then be modified a little to find a lexicographically minimal string rotation during the pre-processing step. This improves the average-case search performance a lot, because it is far more likely that $A$ and $B$ will have matching rotations. It's also a free optimisation, because it has the same $O(m)$ time complexity as the regular KMP pre-processing step.
Credit to @goopypanther for first suggesting the subset approach, and Jannis Harder (@jix) for suggesting the lexicographical rotation trick.
Best Answer
Let us first find $a$.
Let $O$ be the center of the regular polygon. Then, a circle centered at $O$ circumscribes the polygon. Each side is a chord of the circle and hence the perpendicular from the center bisects the side. Also, $\triangle AOM\cong\triangle BOM$ so $\angle AOM=\angle BOM=\dfrac{\pi}{n}$. Now, in $\triangle OAM $,$$\frac{AM}{MO}=\frac{x/2}{r}=\tan\frac{\pi}{n}$$ so $$r=\frac{x}{2\tan\frac{\pi}{n}}$$ so the area of triangle $OAB$ is $\dfrac12OM\times AB=\dfrac{x^2}{4\tan\frac{\pi}{n}}$. There are exactly $n$ such triangles so the total area is, $$\fbox{$a= \dfrac{nx^2}{4\tan\frac{\pi}{n}}$}.$$
Now,
to find the side length of the smaller second polygon, join BC. By the midpoint theorem on $\triangle ABC$, the side length of the second polygon is $\dfrac{BC}{2}$. Now, to find $BC$, note that $\angle CAB=\dfrac{(n-2)\pi}{n}$. By the cosine rule in $\triangle ABC$,$$\frac{x^2+x^2-BC^2}{2x^2}=\cos\angle CAB$$$$\implies 1-\frac{BC^2}{2x^2}=\cos\frac{(n-2)\pi}{n}=\cos\left(\pi-\frac{2\pi}{n}\right)$$$$=-\cos\frac{2\pi}{n}$$ so that $$\frac{BC^2}{2x^2}=1+\cos\frac{2\pi}{n}=2\cos^2\frac{\pi}{n}$$ which implies that $$BC=2x\cos\frac{\pi}{n}$$ or that the side of the second polygon is $\dfrac{BC}{2}= x\cos\dfracπn.$ Thus, its area will be (by similarity) $a\cos^2\dfrac{\pi}{n}$, which implies that the common ratio for the geometric series is $\cos^2\dfrac{\pi}{n}.$