I was looking for a way to check if the regular polygon area formula of $\frac{ns^2}{4} \cot \frac{π}{n}$ where s is the side length and n is the number of sides could be extended to fractional and irrational n. Setting s to be 1, we get the constant in exact form of the area of the polygon. This also gives the area of the unit polygon as well. I am assuming the area of the entire polygon not using the even-odd rule just like the Wolfram Alpha example.
Let this area be called $A_n=\frac{n}{4}\cot\frac{π}{n}$ Testing n=$\frac{5}{2}$, we see that the interior angle is θ=$\frac{4π}{5}$. This creates the pentagram polygon with Schläfli Symbol of $\lbrace\frac{5}{2}
\rbrace$. When putting the formula into use, $A_\frac{5}{2}=\frac{5/2}{4}\cot\frac{π}{5/2}$= $\frac{\sqrt{5}}{8}\sqrt{5-2\sqrt{5}}$, but when searching in Wolfram Mathworld, the constant came out to be A(unit pentagram or area constant of pentagram)=$\frac{\sqrt{10}}{4}\sqrt{65-29\sqrt{5}}$. Similarly, the area of a pentagon can be used in the formula to give $A_5=\frac{5}{4}\cot\frac{π}{5}$= $\frac{\sqrt{5}}{4}\sqrt{5+2\sqrt{5}}$,
In addition, one interpretation of the number of sides can also be the perimeter as we are assuming the side length is $\frac{5}{2}$ meaning that each side has a length of $\frac{1}{2}$ or less which contradicts the assumption that we can only have one side.
However, if we allow the side length to be one without any scaling of the sides as a result of the n-value, the formula still has the constant incorrect. Only when solving for the area of the $36^\circ-126^\circ-18^\circ$ triangle, do we get the value. After multiplying by $10$, the number of triangles in the shape, we get the same value as in Wolfram Mathworld. This triangle is the triangle when the shape is radially divided into ten equal triangles.
My question is what the restrictions are for the formula $A=A(n,s)=\frac{ns^2}{4}\cot \frac{π}{n}$ where n is the number of sides and s is the side length. It definitely works for $n\in\Bbb Z$ an integer assuming negative values of n would correspond to negative areas. If $n=0$, the formula is undefined as well as for any values $n= \frac{1}{m}$ where $m\in\Bbb Z$ and the formula equals $0$ when n= $\frac{2}{2m+1}$ where $m\in\Bbb Z$.
My second question is if there are restrictions for this equation, what would be a general formula only in terms of $n$ and $s$ defined in the above paragraph? This means $n,s\in\Bbb R$. How about for $n,s\in\Bbb C$? How about $n$ and $s$ in the set of all matrices with complex entries $\Bbb M$?
It seems like for a regular complex polygon, the area would be in the form of A=2n*$\frac{1}{2}$bh=nbh=nc*h where 2n is the amount of dividing triangles there are in the polygon, $\frac{1}{2}$bh is the area of these triangles, and c is the circumradius being equal to the base, denoted by b, of each triangle. This premise assumes the regular complex polygon is “filled” so that the “star” outline was filled in completely without any self-intersections affecting the area definition.
Going onto seedcode and setting the stator rotor (sr) to the number of sides and leaving only one rotor to be inside the stator rotor (sr) which is named rotor 1 (r1) to be the turning number will give a shape reminiscent of the shape with Schläfli of $$\left\{ \frac{p}{q}\right\}=\left\{ \frac{sr}{r1}\right\}=\left\{ \frac{\text{number of sides}}{\text{turning number}}\right\}$$ Be sure to set the pen radius (r) in the app to be r1. In the app, r=r1.
Thanks to @Blue for figuring out the formula for a “filled” star polygon in terms of the circumradius and Schläfli number $$\{N\}=\text{ratio}=\left\{\frac{p}{q}\right\}$$. I also appreciate them for giving insight into the other convention. Going on the star polygon link below, gives the circumradius in terms of the ratio which then can be simplified to $\frac{1}{2} \csc\frac{π}{N}$. This can be intuitively and geometrically understood by imagining the cosecant operator in the diagram in their post. However, this assumes a unit length and therefore “area constant”.
Interestingly enough, substituting this value into their formula for r and multiplying by $s^2$ for s being the side length simplified down to:
$$\text{Area}\left(\text{“filled” polygon with side length }s\text{ and Schläfli symbol }\{N\}=\left\{\frac a b\right\}\right)=A(a,b,s)=A_N (s)= \frac{a s^2\csc^2 \frac{π}{N}}{4(\cot\frac{π}{a}+\tan\frac{π}{N})}$$. Testing the pentagram we get $$A_{5/2} = \frac{\sqrt{10}}{4}\sqrt{65-29\sqrt{5}}=.31027…$$ after a few steps, the same value as in Wolfram Mathworld.
Amazingly, when putting $$N=a=\frac a1 =\frac ab$$ we get $$A_N (s)=\frac {a s^2 }4 \cot \frac πa$$ which implies the cotangent formula assuming a “filled polygon” and b=1 can only have n as an integer and n not equal to 0. Even trying a pure imaginary number such as $\{N\}=\left\{\frac i2\right \} $ with side length $i$ experimentally gets:
$$A_{\frac i2} (i)=\frac 14\tanh(\pi)\coth(2\pi)\text{csch}(2\pi)=\frac{e^{2\pi}(e^{4 \pi}+1)}{2(e^{2 \pi}+1)^3(e^{2 \pi}-1)}=.00093025…$$ after a few steps. This experimental example has the hyperbolic trig functions.
New answers for complex $n,s$ are always welcomed. Please correct me and give me feedback!
Here are some links that were used in the making of these questions:
Generalized Overturned Polygons
Best Answer
First, I'll note that not everyone agrees about terminology and/or notation in this area. My biases are informed by Branko Grünbaum (my masters thesis advisor (name-drop!)), who's responsible for introducing relatively-new conventions. (See, for instance, the Regular Star Polygons section of Wikipedia's "Regular Polygon" entry.) Be that as it may ...
It's best to think of the "regular polygon" with Schläfli symbol $\{n/k\}$ as a closed polygonal path that shares vertices with a regular $n$-gon but whose edges jump to every $k$-th one. For instance, here's $\{7/3\}$, traversing the vertices of the regular $7$-gon in the order $P_{3m}$, for $m=0, 1, 2, \ldots, 6$, (with subscripts taken modulo $7$);
For $k=1$, the regular $\{n/k\}$-gon (I'll stop typing "regular") traces the boundary of the standard, convex $n$-gon. For $n$ and $k$ co-prime, it's a self-intersecting path with no discernible "interior". Non-co-primes create a path that traces over itself multiple times (eg, a $\{6/2\}$ is a hexagonal path that wraps twice around a triangle); it may-or-may not have an "interior", or a "multiply-covered interior", or whatever, but I won't dwell on that.
Assigning an "area" to such a thing may-or-may not make sense in every context. However, the Shoelace Formula for computing the area of a polygon from the coordinates of its vertices can't tell if the path is self-intersecting, so it generates "area" calculations that, while possibly non-sensical, are at least consistent with non-self-intersecting cases; this makes it as good a definition of "area" as any.
Taking vertex $P_m$ of the host $n$-gon to have coordinates $(r\cos(2\pi m/n), r\sin(2\pi m/n))$ so that $\{n/k\}$ traces a path of the vertices $P_{km}=(r\cos(2\pi km/n), r\sin(2\pi km/n))$, the Shoelace Formula ultimately yields exactly what one gets from OP's motivating Cotangent Formula, with $n/k$ substituting for the "number of sides": $$\text{"area"} = n\cdot\frac12\cdot r^2\sin\frac{2k\pi}{n} = \frac14 ns^2\cot\frac{\pi}{n/k} = \frac14 ns^2\cot\frac{k \pi}{n} \tag{$\star$}$$ where $s:=2r\sin(\pi k/n)$ is a segment of the polygonal path (eg, $\overline{P_0 P_3}$ in the above $\{7/3\}$).
But OP isn't interested in that. :)
Rather, OP wants a formula for the area of a "complex", "filled" polygon whose outline matches that of an $\{n/k\}$-gon. While sources (I'm looking at you, MathWorld) tend to conflate such a figure with the $\{n/k\}$-gonal path itself, I'll dub the figure the hull of the $\{n/k\}$-gon, and denote it $[n/k]$. (Here, we'll assume $n$ and $k$ are co-prime.) So, for instance, below is $[7/3]$, which is a pointy, non-convex, equilateral $14$-gon with a non-self-intersecting boundary; it is not a $7$-gon. (Generally, $[n/k]$ is a $2n$-gon.)
Calculating the hull's area is straightforward. In the $[7/3]$ example, symmetry tells us that the area is $2\cdot7$-times that of $\triangle OP_0 Q$; and that area we can compute with some angle chasing and a little help from the Law of Sines. In general, writing $r$ for the circumradius, we have ...
$$\begin{align} \text{area of}\;[n/k] &= 2n\cdot |\triangle OP_0Q| = 2n\cdot \frac12|OP_0||OQ|\sin\angle P_0OQ \\[8pt] &= n\cdot r\cdot \frac{r\sin\frac{\pi(n-2k)}{2n}}{\sin \frac{\pi(n+2k-2)}{2n}}\cdot\sin\frac{\pi}{n} \\[8pt] &= n r^2 \frac{\cos\frac{k\pi}{n}\sin\frac{\pi}{n}}{\cos\frac{(k-1)\pi}{n}} \\[8pt] &= \frac{n r^2}{\cot\dfrac{\pi}{n}+\tan\dfrac{k\pi}{n}} \tag{$\star\star$} \end{align}$$
Expressing $(\star\star)$ in terms of the $\{n/k\}$ path's segment length (eg, $|P_0P_3|$) or the $[n/k]$ hull's side length ($|P_0 Q|$) is left as an exercise to reader.
As for OP's questions about generalizing further, say to an irrational "numbers of sides" ...
Observe that the rational case (including the good ol' integer case), the vertices of the polygon lie along a circle's circumference with central angles that are rational multiples of $\pi$. If the irrational case simply substitutes irrational central angles, then any such path bounces unendingly about the circle, with vertices arbitrarily close to any any point on the circumference. In the "$\{n/k\}$ path" sense, it has infinitely many sides and thus infinite (Shoelace) "area"; in the "$[n/k]$ hull" sense, it effectively fills the circle, for an "area" of $\pi r^2$.
Then again, there may be alternative interpretations. This may be a discussion for another time (and question). I've done enough typing. :)