I guess the answer is: $C(n,n-3)/2$ since for $n$-sided polygon,there are $n$ vertices, and for each vertex, it cannot form diagonal with the adjacent points and itself. So, each vertex can form $n-3$ points. And half of the vertices have repeated diagonals and we need to divide by $2$.
It holds true for quadrilateral and pentagon,but why it does not fit triangles?
Best Answer
The number you have computed, $C(n,n-3)/2$, which I am interpreting as $\binom{n}{n-3}/2$ though this doesn't coincide with your argument, is too large. A different way:
We can choose any two points, except adjacent points are not allowed. There are $\binom{n}{2}$ pairs of points and $n$ pairs of adjacent points, so the answer is $\binom{n}{2}-n$.