First, I'll note that not everyone agrees about terminology and/or notation in this area. My biases are informed by Branko Grünbaum (my masters thesis advisor (name-drop!)), who's responsible for introducing relatively-new conventions. (See, for instance, the Regular Star Polygons section of Wikipedia's "Regular Polygon" entry.) Be that as it may ...
It's best to think of the "regular polygon" with Schläfli symbol $\{n/k\}$ as a closed polygonal path that shares vertices with a regular $n$-gon but whose edges jump to every $k$-th one. For instance, here's $\{7/3\}$, traversing the vertices of the regular $7$-gon in the order $P_{3m}$, for $m=0, 1, 2, \ldots, 6$, (with subscripts taken modulo $7$);
For $k=1$, the regular $\{n/k\}$-gon (I'll stop typing "regular") traces the boundary of the standard, convex $n$-gon. For $n$ and $k$ co-prime, it's a self-intersecting path with no discernible "interior". Non-co-primes create a path that traces over itself multiple times (eg, a $\{6/2\}$ is a hexagonal path that wraps twice around a triangle); it may-or-may not have an "interior", or a "multiply-covered interior", or whatever, but I won't dwell on that.
Assigning an "area" to such a thing may-or-may not make sense in every context. However, the Shoelace Formula for computing the area of a polygon from the coordinates of its vertices can't tell if the path is self-intersecting, so it generates "area" calculations that, while possibly non-sensical, are at least consistent with non-self-intersecting cases; this makes it as good a definition of "area" as any.
Taking vertex $P_m$ of the host $n$-gon to have coordinates $(r\cos(2\pi m/n), r\sin(2\pi m/n))$ so that $\{n/k\}$ traces a path of the vertices $P_{km}=(r\cos(2\pi km/n), r\sin(2\pi km/n))$, the Shoelace Formula ultimately yields exactly what one gets from OP's motivating Cotangent Formula, with $n/k$ substituting for the "number of sides":
$$\text{"area"} = n\cdot\frac12\cdot r^2\sin\frac{2k\pi}{n} = \frac14 ns^2\cot\frac{\pi}{n/k} = \frac14 ns^2\cot\frac{k \pi}{n} \tag{$\star$}$$
where $s:=2r\sin(\pi k/n)$ is a segment of the polygonal path (eg, $\overline{P_0 P_3}$ in the above $\{7/3\}$).
But OP isn't interested in that. :)
Rather, OP wants a formula for the area of a "complex", "filled" polygon whose outline matches that of an $\{n/k\}$-gon. While sources (I'm looking at you, MathWorld) tend to conflate such a figure with the $\{n/k\}$-gonal path itself, I'll dub the figure the hull of the $\{n/k\}$-gon, and denote it $[n/k]$. (Here, we'll assume $n$ and $k$ are co-prime.) So, for instance, below is $[7/3]$, which is a pointy, non-convex, equilateral $14$-gon with a non-self-intersecting boundary; it is not a $7$-gon. (Generally, $[n/k]$ is a $2n$-gon.)
Calculating the hull's area is straightforward. In the $[7/3]$ example, symmetry tells us that the area is $2\cdot7$-times that of $\triangle OP_0 Q$; and that area we can compute with some angle chasing and a little help from the Law of Sines. In general, writing $r$ for the circumradius, we have ...
$$\begin{align}
\text{area of}\;[n/k] &= 2n\cdot |\triangle OP_0Q| = 2n\cdot \frac12|OP_0||OQ|\sin\angle P_0OQ \\[8pt]
&= n\cdot r\cdot \frac{r\sin\frac{\pi(n-2k)}{2n}}{\sin \frac{\pi(n+2k-2)}{2n}}\cdot\sin\frac{\pi}{n} \\[8pt]
&= n r^2 \frac{\cos\frac{k\pi}{n}\sin\frac{\pi}{n}}{\cos\frac{(k-1)\pi}{n}} \\[8pt]
&= \frac{n r^2}{\cot\dfrac{\pi}{n}+\tan\dfrac{k\pi}{n}} \tag{$\star\star$}
\end{align}$$
Expressing $(\star\star)$ in terms of the $\{n/k\}$ path's segment length (eg, $|P_0P_3|$) or the $[n/k]$ hull's side length ($|P_0 Q|$) is left as an exercise to reader.
As for OP's questions about generalizing further, say to an irrational "numbers of sides" ...
Observe that the rational case (including the good ol' integer case), the vertices of the polygon lie along a circle's circumference with central angles that are rational multiples of $\pi$. If the irrational case simply substitutes irrational central angles, then any such path bounces unendingly about the circle, with vertices arbitrarily close to any any point on the circumference. In the "$\{n/k\}$ path" sense, it has infinitely many sides and thus infinite (Shoelace) "area"; in the "$[n/k]$ hull" sense, it effectively fills the circle, for an "area" of $\pi r^2$.
Then again, there may be alternative interpretations. This may be a discussion for another time (and question). I've done enough typing. :)
Best Answer
The most basic statement along these lines is that the average width of any line segment is equal to $2L/\pi$, where $L$ is the length of the segment. This is highly related to the Buffon's needle problem, where the length of the needle is the same as the width of the strips. It is also related to the fact that the average value of $|x|$ on the unit circle is equal to $2/\pi$.
Now consider a convex polygon in the plane. No matter how you turn the polygon, its horizontal width is always equal to twice the sum of the horizontal widths of the edges. Specifically, the sum of the widths of the edges along the bottom is equal to the width of the polygon, and the same is true of the widths of the edges along the top. Thus, the average width of any convex polygon is equal to half the sum of the average widths of the side lengths: $$ \text{avg. width} \;=\; \frac{1}{2}\left(\frac{2s_1}{\pi} + \cdots + \frac{2s_n}{\pi}\right) \;=\; \frac{s_1+\cdots+s_n}{\pi}. $$
There is a similar formula in three dimensions involving area. Specifically, suppose we take a polyhedron and rotate it in all possible ways. For each rotation, we measure the area of the projection of the polyhedron onto the $xy$-plane. Then $$ \text{avg. $xy$-area} \;=\; \frac{A_1+\cdots+A_n}{4} $$ where $A_1,\ldots,A_n$ are the areas of the faces. Thus, if you take a unit cube outside on a sunny day and rotate it at random, the average area of its shadow will be $(1+1+1+1+1+1)/4 = 3/2$. Again, the factor of $4$ comes from the fact that a polyhedron has top faces and bottom faces, and the average $xy$-area of a flat object is $A/2$. The latter follows from the fact that the average value of $|z|$ on the unit sphere is $1/2$.