I'm trying to derive a general formula for the side of a square inscribed in a regular polygon with $n$ sides.
I know there may be more than one, e.g. infinitely-many for an octagon, but I want to be able to extract the side for at least one of them.
I am aware of Dilworth and Mane's 2010 paper "Inscribing a regular $m$-gon in a regular $n$-gon" (PDF link via sc.edu), but there is no indication on how to calculate the side.
Second part:
Is it true that for an odd $n$ there is no inscribed square that is concentric with the polygon?
Best Answer
This problem (the first part) is a simple version of one posed some one hundred years ago, called the Inscribed Square Problem or the Square Peg Problem. In full generality that problem is still unsolved.
It is to our advantage that regular polygons are convex. For convenience we can assume the vertices of our regular $n$-gon are equally spaced points on the unit circle around the origin:
$$ (1,0),\;(\cos\left(\frac{2\pi}{n}\right),\sin\left(\frac{2\pi}{n}\right)),\; \ldots (\cos\left(\frac{2(n-1)\pi}{n}\right),\sin\left(\frac{2(n-1)\pi}{n}\right)) $$
Case: n divisible by four
The easiest case to "calculate" is when $n$ is a multiple of four, e.g. the regular octagon mentioned in the Question. We can choose any point on the perimeter of the regular $n$-gon and rotate around the polygon through $n/4$ edges to locate successive points (until we return to the originally chosen point). The four points so chosen will form a square.
If $(x,y)$ is the point initially chosen on the $n$-gon (where $4|n$), and the polygon is centered on the origin, then the next points appears at right-angle rotations, given by:
$$ (x,y),\;(y,-x),\;(-x,-y),\;(-y,x) $$
The maximum area is achieved in this case if the initial point $(x,y)$ is chosen to be a vertex of the (origin-centered) regular $n$-gon.
Case: n divisible by two but not by four
If $n$ is even but not a multiple of $4$, then the regular $n$-gon described by our vertices above is symmetric with respect to both the $x$-axis and the $y$-axis. Note that while vertices $(1,0)$ and $(-1,0)$ are on the $x$-axis, the $y$-axis intersects the polygon at midpoints of two parallel edges.
We are interested in the points where $y=x$ intersects the polygon. The vertices of our polygon which "straddle" this line correspond (in the first quadrant) to the angles:
$$ \frac{2\pi k}{n} \lt \frac{\pi}{4} \lt \frac{2\pi (k+1)}{n} $$
Stated more simply, this says $k \lt n/8 \lt k+1$, so $k= \lfloor n/8 \rfloor$.
The equation for the edge, passing through these points:
$$ (\cos\left(\frac{2k\pi}{n}\right),\sin\left(\frac{2k\pi}{n}\right)),\;(\cos\left(\frac{2(k+1)\pi}{n}\right),\sin\left(\frac{2(k+1)\pi}{n}\right)) $$
can then be solved with $y=x$ to determine a point of intersection $(r,r)$ on the polygon. An inscribed square is then given by corners:
$$ (r,r),\;(-r,r),\;(-r,-r),\;(r,-r) $$
which belong to the polygon by virtue of its symmetry with respect to the $x$-axis and the $y$-axis.
Case: n is odd (not divisible by two)
TBD