I would do the problem exactly the way you did it.
Here is an alternative approach to confirm your answer.
We can line up the five students in $5!$ ways, leaving spaces between them and at the ends of the row in which to insert the teachers. There are six such spaces, four between successive students and two at the ends of the row. We can insert the three teachers in $P(6, 3) = 6 \cdot 5 \cdot 4$ ways. This gives us $$5! \cdot 6 \cdot 5 \cdot 4$$ linear arrangements of students and teachers in which no two teachers are consecutive.
However, since we wish to arrange the students and teachers around a circular table so that no two teachers sit in consecutive seats, we must exclude those linear arrangements in which teachers are at both ends of the row. There are three ways to select the teacher at the left end of the row, two ways to select the teacher at the right end of the row, and four ways to place the remaining teacher in one of the four spaces between successive students. Hence, there are $$3 \cdot 2 \cdot 4 \cdot 5!$$ linear arrangements in which teachers are at both ends of the row.
Hence, there are $$6 \cdot 5 \cdot 4 \cdot 5! - 3 \cdot 2 \cdot 4 \cdot 5! = (120 - 24)5! = 96 \cdot 5!$$ linear arrangements of teachers and students so that no two teachers are consecutive and teachers are not at both ends of the row.
These linear arrangements correspond to the permissible ways we can seat the students and teachers around the table. To account for rotational invariance, we divide the number of linear arrangements by $8$, which yields
$$\frac{96 \cdot 5!}{8} = 12 \cdot 5! = 1440$$
permissible seating arrangements around a circular table, as you found.
$A$, $B$, $C$, $D$ and $E$ are five persons who are to be seated around a circular table such that $A$ and $B$ must sit together and $C$ and $D$ must never sit together. In how many ways can they be seated?
There are four possible seating arrangements.
Seat E. Since A and B sit together and C and D are separated, C and D must both be adjacent to E. Therefore, choosing whether C or D sits to E's immediate left also determines who sits to E's immediate right and choosing whether A or B sits two seats to E's left also determines who sits two seats to E's right. Hence, there are $2 \cdot 2 = 4$ permissible seating arrangements, as shown below.
Notice that none of these seating arrangements can be obtained from another by rotation.
Should the rotation of a particular arrangement be construed as the same or different?
By convention, a rotation of a particular arrangement is considered to be the same unless the seats are labeled or we are given a particular reference point (such as a special chair or the north end of the table).
Notice that we have already accounted for rotational invariance by measuring our seating arrangements relative to the position of E.
Say we have $10$ persons sitting around a circular table, with $3$ of them wanting to sit together and $4$ other persons who wish to be separated? In how many ways can they be seated?
We use the block of three people who wish to sit together as our reference point. Say the people are $A$, $B$, and $C$. In how many ways can they be arranged within the block?
$3!$
Suppose the four people who wish to be separated are $D$, $E$, $F$, and $G$. Since there are only seven seats left at the table, they must be seated in the seats that are $1$, $3$, $5$, and $7$ positions to the left of the block. In how many ways can they be seated?
$4!$
Let's call the remaining three people $G$, $H$, and $I$. In how many ways, can they be seated in the remaining three chairs?
$3!$
Therefore, by the Multiplication Principle, the number of permissible seating arrangements is
$3!4!3!$
Best Answer
The problem with your second working is that it leads to duplicates. To understand it, when you have chosen $9$ students of $14$ and arranged them in between $10$ students who cannot sit together, say there is a student A who is in the group of $9$. Now you are left with group of $5$ students. Say there is a student B in this group of $5$. Now you seat B in one of the twenty available slots.
But you also have arrangements where B is chosen to be in the first group of $9$ and A is in the second group of $5$. Now when you try and seat A as part of second group, it leads to duplicate arrangements.
If you want to seat $10$ students first and then seat $14$ students, here is one way to count arrangements -
First use stars and bars to count number of possible arrangements of seats. There are $11$ slots available after seating $10$ students. $9$ slots in between cannot be empty and that leaves only $5$ seats that must go to one of the $11$ slots. That can be done in $\displaystyle {15 \choose 5}$ ways.
Now that we have $14$ seats, all is left is to seat the students, which can be done in $14!$ ways.
So total number of permissible arrangements = $\displaystyle 10! \cdot {15 \choose 5} \cdot 14!$