In a row: There are $10!$ ways to arrange the people. We now count the bad arrangements, in which two A's are next to each other, or two B's, or two C's.
To do the counting, we use Inclusion/Exclusion. How many arrangements have the two A's together? Put them in a bag. We now have $8$ people and $1$ bag. These can be arranged in $9!$ ways. Then when we let the A's out of the bag, they can arrange themselves in $2$ ways, for a total of $2\cdot 9!$.
Do the same for the B's, the C's, and add up. We get $3\cdot 2\cdot 9!$.
However, we have double-counted the bad arrangements in which, for example, the A's and the B's are together. The same bag argument shows there are $2^2\cdot 8!$ such arrangements, for a total of $3\cdot 2^2\cdot 8!$.
Thus our next estimate for the number of bads is $3\cdot 2\cdot 9!-3\cdot 2^2\cdot 8!$.
However, we have subtracted one too many times the arrangements in which the A's are together, and the B's, and the C's. The now familiar technique shows there are $2^3\cdot 7!$ of these. So add back $2^3\cdot 7!$.
We end up with $7!(6\cdot 9\cdot 8-12\cdot 8+8)$, which is $8!\cdot 43$.
This is the number of bad arrangements. The number of good arrangements is $10!-8!\cdot 43$, which is $8!\cdot 47$.
Circular table: We use the convention that two orders are to be considered the same if they differ by a rotation. Assume that one of the "others" is a Canadian. He/she is probably too polite to complain, so one can put the Canadian in the worst chair.
The remaining chairs can now be thought of as a line. We have $9$ people, including $3$ "others" to put in a line, with the condition that the two A's, two B's, and two C's are separated. Same Inclusion/Exclusion technique.
This is the correct exponential generating function. To get the number of ways to distribute $r$ (distinguishable!) people into $6$ rooms as stated you want:
\begin{equation*}
\left[ \frac{x^r}{r!} \right]
\left( \frac{x^2}{2!} + \frac{x^3}{3!} + \frac{x^4}{4!} \right)^6
\end{equation*}
The algebra to get this for $r = 12$ to $r = 24$ is tedious, but not hard (use a computer algebra system!).
Best Answer
The count of $5+1$ needs to be updated. You have $4$ ways to choose the room that $5$ will be in, $3$ ways to choose the room for $1$, and $6$ ways to choose the lone traveler. So the number of ways to accommodate the guests is $4096-4-4\cdot 3 \cdot 6=4096-76=4020$