combinatoricsgeometry
On a circle there are $n$ evenly spaced points (i.e., they form a regular polygon). How many triangles can be formed when congruent triangles are considered the same?
This question is closely related to
How many triangles can be formed by the vertices of a regular polygon of $n$ sides?, but here congruent triangles are considered to be the same.
In the case where the isosceles triangles must have vertices drawn from the regular $n$-gon, let $k$ be the number of the $n$-gon sides that are outside one of the two legs of the isosceles triangle.
For every isosceles triangle $k$ must be an even number strictly between $0$ and $n$. On the other hand, if $k$ is given, then the side lengths of the isosceles triangle also follow, and the only choice we have is which of the $n$ vertices to choose as the apex. This analysis gives $\lfloor \frac{n-1}{2}\rfloor n$ possibilities. However, if $n$ is a multiple of $3$ then we have counted each of the $n/3$ equilateral inscribed triangles tree times, and we must correct for this.
In total the number of triangles is $$ \left\lfloor \frac{n-1}{2}\right\rfloor n - \begin{cases}(2/3)n&\text{if }3|n\\0& \text{otherwise}\end{cases}$$ Or, expressed by case analysis modulo 6: $$ \#\text{isosceles} = \begin{cases} n^2/2 - (5/3) n & \text{for } n\equiv 0 \pmod 6 \\ n^2/2 - (1/2) n & \text{for } n\equiv 1, 5 \pmod 6 \\ n^2/2 - n & \text{for } n\equiv 2, 4 \pmod 6 \\ n^2/2 - (7/6) n & \text{for } n\equiv 3 \pmod 6 \end{cases}$$
Consider a regular polygon with $n$ number of vertices $\mathrm{A_1, \ A_2,\ A_3, \ A_3, \ldots , A_{n-1}}$ & $\mathrm{A_{n}}$
Total number of triangles formed by joining the vertices of n-sided regular polygon $$N=\text{number of ways of selecting 3 vertices out of n}=\color{}{\binom{n}{3}}$$ $$N=\color{red}{\frac{n(n-1)(n-2)}{6}}$$ $\forall \ \ \color{blue}{n\geq 3}$
Consider a side $\mathrm{A_1A_2}$ of regular n-polygon. To get a triangle with only one side $A_1A_2$ common (As shown in figure-1 below)
(figure-1)
Join the vertices $A_1$ & $A_2$ to any of $(n-4)$ vertices i.e. $A_4, \ A_5,\ A_6, \ \ldots \ A_{n-1}$ to get triangles with only one side common. Thus there are $(n-4)$ different triangles with only one side $A_1A_2$ common. Similarly, there are $(n-4)$ different triangles with only one side $A_2A_3$ common & so on. Thus there are $(n-4)$ different triangles with each of $n$ sides common. Therefore, number of triangles $N_1$ having only one side common with that of the polygon $$N_1=\text{(No. of triangles corresponding to one side)}\text{(No. of sides)}=\color{blue}{(n-4)n}$$
(figure-2)
Now, join the alternate vertices $A_1$ & $A_3$ by a straight (blue) line to get a triangle $A_1A_2A_3$ with two sides $A_1A_2$ & $A_2A_3$ common. Similarly, join alternate vertices $A_2$ & $A_4$ to get another triangle $A_2A_3A_4$ with two sides $A_2A_3$ & $A_3A_4$ common & so on (as shown in above figure-2). Thus there are $n$ pairs of alternate & consecutive vertices to get $n$ different triangles with two sides common (Above fig-2 shows $n$ st. lines of different colors to join alternate & consecutive vertices). Therefore, number of triangles $N_2$ having two sides common with that of the polygon $$N_2=\color{blue}{n}$$ If $N_0$ is the number of triangles having no side common with that of the polygon then we have $$N=N_0+N_1+N_2$$ $$N_0=N-N_1-N_2$$ $$=\binom{n}{3}-(n-4)n-n$$ $$=\color{}{\frac{n(n-1)(n-2)}{6}-n^2+3n}$$ $$N_0=\color{red}{\frac{n(n-4)(n-5)}{6}}$$ The above formula $(N_0)$ is valid for polygon having $n$ no. of the sides such that $ \ \ \color{blue}{n\geq 6}$
Best Answer
For each triangle $A_iA_jA_k$, the angles $\angle A_iOA_j$, $\angle A_jOA_k$ and $\angle A_kOA_i$ are respectively equal to $|j-i|\frac{2\pi}{n}$, $|j-k|\frac{2\pi}{n}$ and $|k-i|\frac{2\pi}{n}$ with $1\le i\ne j\ne k \le (n-1)$.
Let denote $$ \begin{cases} x=|j-i|\\ y=|k-j|\\ z=|k-i| \end{cases} $$
Then $$ \begin{cases} x+y+z = n \\ x,y,z \in \Bbb N^* \\ \end{cases} \tag{1} $$
The number of triangles when congruent triangles are considered the same is equal to the number of solutions of $(1)$ (Two solutions that differ only in the order of their summands are considered the same, for example, with $n = 4$: $1+1+2$ is the same as $2 +1 +1$).
The problem $(1)$ is solved here and the number of solutions of $(1)$ is the number of partitions $p(3,n)$ of $n$ into $3$ non-zero parts. The number of partitions $p(k,n)$ satisfies $$ \begin{cases} p(0,0) &= 0 \\ p(k,n) &= p(k,n-k)+ p(k-1,n-1) & \text{otherwise}. \\ \end{cases} \tag{2} $$ Remark: It seems that the recurrent formula in the answer is not correct, the recurrent formula $(2)$ is taken from the wikipedia here. From $(2)$, we can deduce the general formula of $p(3,n)$ as follows
$$ \begin{align} \implies p(3,n) &=\left\lfloor \frac{n-1}{2} \right\rfloor+\left\lfloor \frac{n-4}{2} \right\rfloor+...+\left\lfloor \frac{n-1-3i}{2} \right\rfloor+... \\ & =\sum_{0 \le i \le \left\lfloor \frac{n-1}{3}\right\rfloor}\left\lfloor \frac{n-1-3i}{2} \right\rfloor \end{align} $$
Hence, the number of triangles is equal to $$p(3,n)=\sum_{0 \le i \le \left\lfloor \frac{n-1}{3} \right\rfloor}\left\lfloor \frac{n-1-3i}{2} \right\rfloor$$
Remark: The last steps have a lot of calculation, please feel free to correct if you find any mistake.