Given $n$ people, where $n$ is even, you can choose the first pair of people ${n \choose 2}$ ways, where ${n \choose 2}=\frac{n!}{2!(n-2)!}$. The next pair can be chosen in ${n-2\choose 2}$ ways, etc... The end result will be $\frac{n}{2}$ pairs which can be arranged in $\left(\frac{n}{2}\right)!$ ways. So there are $$\frac{{n \choose 2}{n-2 \choose 2}\dots{2 \choose 2}}{\left(\frac{n}{2}\right)!}=\frac{n!}{\left(\frac{n}{2}\right)!\,2^{\left(\frac{n}{2}\right)}}$$ ways to arrange all $n$ people into sets of pairs.
So for 8 people there are $\frac{8!}{4!\,2^{4}}=105$ possible sets of pairs.
Now the question remains, how many sets are there that do not contain any pairs from the original set?
Let $$f(x)=\frac{x!}{\left(\frac{x}{2}\right)!\,2^{\frac{x}{2}}}$$
If $S=\left\{p_{1}, p_{2}, ..., p_{\frac{n}{2}}\right\}$ is our original set of pairs and $P_{k}$ is the set of all sets containing $p_{k}$, where $1\le k\le\frac{n}{2}$ then by the inclusion-exclusion principal the number of sets not containing any of the original pair is:
$$f(n)-\left(|P_{1}|+|P_{2}|+\dots+|P_{\frac{n}{2}}|\right)+\left(|P_{1}\cap P_{2}|+(|P_{1}\cap P_{3}|+\dots\right)-\dots$$
But for any $P_{k}$; $|P_{k}|=f(n-2)$ therefore, $|P_{1}|=|P_{2}|=\dots=|P_{\frac{n}{2}}|$and in general, given any $k$ where $1\le k\le \frac{n}{2}$, then $|P_{1}\cap P_{2}\cap\dots\cap P_{k}|=f(n-2k)-\dots$
So the number of sets not containing any of the original pair is:
$$f(n)-\left(f(n-2)+f(n-2)+\dots\right)+\left(f(n-4)+f(n-4)+\dots\right)-\dots$$
which equals:
$$f(n)-{\frac{n}{2}\choose 1}f(n-2)+{\frac{n}{2}\choose 2}f(n-4)-\dots$$
So in the case where $n=8$
$$f(8)-4f(6)+6f(4)-4f(2)+1f(0)=105-60+18-4+1=60$$
You came close. When you have the $3$ chairs to put in $6$ places it becomes as stars and bars problem, so the answer is ${6+3-1\choose6-1}={8\choose5}=56$, which does result in $6720.$
Your solution us incorrect, because ${6\choose3}$ is the number of ways to pick three of the spots in which to put a chair. This would be OK if it were required that there be no more than two chairs between any two of the kids, but nothing prevents us from putting all three chairs between the second at third kid, for example.
We have six spots where we can put chairs, and three chairs to distribute. This is the same problem as distributing three indistinguishable balls in six distinct bins, which is solved by stars and bars.
To answer the point you raise in your last comment, the spots are distinguishable because it makes a difference if Jane and Mary are separated by two chairs or three or four, but it doesn't matter which particular chairs we put between them; the red chair and the green chair are the same as far as we are concerned in this problem.
Best Answer
(1) Assuming they in a line, not a circle, then imagine $7$ stars a row (the "other" students). Among the $8$ gaps in between or outside of those stars, choose $3$ for the locations of the special $3$ students. This can be done in $C(8,3)=56$ ways.
(2) Now order (say, left to right) both of the two subsets of students arbitrarily, this can be done $(7!)(3!)$ ways.
Each admissible arrangement of students corresponds to a unique sequence of choices in (1) and (2). So there are $56(7!)(3!)$ ways to do it.