We initially seat A, B, and D. Since A and B must sit together, there are only two ways to do this, depending on whether B sits to the left or right of A, as shown below.
To ensure that C and E do not sit together, we must place them to the left or right of the block. There are two ways to do this. Hence, there are four admissible arrangements, as shown below.
- Pick which four people will sit at the smaller round table in $\binom{10}{4}$ ways.
- Now... let the youngest person who you selected sit at the table wherever they like, it matters not where. Then, after the youngest has chosen his seat, arrange the remaining people around the rest of the table in $3!$ ways.
- The youngest person from those not selected, let him sit at the larger table wherever he likes, it matters not where. Then, have the remaining people sit around the outside of that table in the remaining seats in $5!$ ways.
The final answer is then:
$$\binom{10}{4}\times 3!\times 5!$$
Your error is in neglecting to fully account for who the people sitting at the first table is. The "multiplication by 1" makes some sort of sense if the selection of the people has already occurred, but you jumped the gun with it, having used it before we know the full list of people sitting there yet.
Summarizing comments from below for easier access:
Another popular way of thinking of this type of problem is via a "division by symmetry" argument. A division by symmetry argument generally goes as follows: If we count the number of outcomes while allowing ourselves to overcount and we get an answer of $n$ including the overcounts and we learn that every outcome was overcounted the exact same number of times, having been counted $k$ times each, then the corrected count without overcounting would be $n/k$.
I tend to avoid these style arguments as a matter of principle, as they are a common source of headaches for people just starting, are challenging to phrase correctly, and require the use of division which would if used incorrectly take us out of the realm of the integers into the realm of the natural numbers. To a beginner, it is not clear at a glance why $\dfrac{(kn)!}{(k!)^n}$ should be an integer, but writing it as $\binom{kn}{k,k,\dots,k}$ it is much clearer that it will be an integer. That all being said, there are times where a division by symmetry argument is helpful and perhaps even clearer than the alternatives.
For this problem, let us starting from the small table and starting from north going clockwise select a person to sit in each seat, then once that table is full do the same for the larger table again starting from north. You will find there are $(10\times 9\times 8\times 7) \times 6\times \cdots \times 2\times 1 = 10!$ such arrangements.
Then, we recognize that each way of arranging the small table was overcounted by a factor of $4$ due to the different ways of rotating the table. Rather than $10\times 9\times 8\times 7$ it should have been $10\times 9\times 8\times 7 / 4$. The division by $4$ can be thought of as "forgetting which way was north" for that table. Note that the division is by $4$, not by $10$, because what we are dividing by was the number of times we had overcounted each occurrence. Similarly for the second table, we divide by $6$ because that was the number of different equivalent rotations of the table.
We get as a final result, the equivalent way of writing the final answer as:
$$\dfrac{10!}{4\times 6}$$
Best Answer
There are $9$ distinct spots in which the $4$ math majors can be seated consecutively around a table, if you treat the $4$ people as a clump. There $4!$ ways to arrange them within each clump. The other $5$ people can be seated in $5!$ different ways. Everyone can be seated in $9!$ different ways giving a probability of
$$\frac{9\cdot4!\cdot5!}{9!}\approx.0714$$
Note that this probability is lower if you seat the $9$ people in a line rather than around a circle, since there would only be $6$ ways to "clump" the $4$ math majors together.
To see why there are $9$ ways to seat a clump of $4$ people, consider a clock with numbers $1-9$ instead of $1-12$. Then the $4$ people can sit in spots $$1-4, 2-5, 3-6, 4-7, 5-8, 6-9,7-1, 8-2, \text{ or } 9-3$$.