Added below: A solution using Burnside’s Lemma, which is the way to generalize and regularize what I did, as asked about in a comment. I didn't use Burnside's Lemma in my answer, because I wanted to give an answer that was elementary.
I know that they can be arranged in $\tfrac{10!}{5!5!}$ ways if they
are not placed in a circle. Each circular arrangement is then
overcounted 10 times, so the answer should be
$\tfrac{10!}{10\cdot5!5!}$.
Each arrangement is not overcounted by a factor of 10.
For example, while the ten linear arrangements $RRRRRDDDDD, DRRRRRDDDD,\dots RRRRDDDDDR$ all lead to the same seating arrangement (all party members together), $RDRDRDRDRD$ and $DRDRDRDRDR$ are the only two linear arrangements that lead to strictly alternating seating around the circle.
Each circular arrangement is counted a full 10 times only if its rotations by tenths of a circle yield different linear sequences when reading from the top clockwise. If an arrangement is unchanged by a rotation smaller than $2\pi$, there are fewer than 10 linear arrangements. The only possible smallest nonzero rotations by tenths that fix a circular pattern are $\tfrac{2\pi}{10}$ (pattern counted 1 times), $\tfrac{2\pi}{5}$ (pattern counted 2 times), and $\tfrac{2\pi}{2}$ (pattern counted 5 times).
For the question at hand, $\tfrac{2\pi}{5}$ is the only possibility. (If the pattern were fixed by a one-tenth rotation, all symbols would have to be the same. If it were fixed by a one-half rotation, the number of democrats and republicans would have to both be even.)
The only arrangement fixed by a $\tfrac{\pi}{5}$ rotation is the one I mentioned above, which is counted twice. Every other arrangement is counted ten times.
The number of arrangements is then $\frac{\tfrac{10!}{5!5!}-2}{10}+1$, or 26.
Added: Briefly, Burnside’s lemma (which is easy to google) says that given a collection of patterns, you can find the number of them that are distinguishable up to a group of motions (such as rotations here) by finding the average number of patterns fixed by one of the motions, which happens to be the same number. (This is not immediately obvious!)
So in this case (5 each of $R$s and $D$s at a round table), Look at the set of $\tfrac{10!}{5!5!}$ seating arrangements without considering two the same if one is a rotation of the other. Then consider the group of all rotations. There are 10 different rotations, by $0$, $\tfrac{\pi}{10}$, $2\tfrac{\pi}{10}$, ... $9\tfrac{\pi}{10}$. For each of these 10 rotations, count the number of seating arrangements “fixed” by the rotation. All $\tfrac{10!}{5!5!}$ are fixed by the $0$ rotation. None are fixed by the $k\tfrac{\pi}{10}$ for $k=1,3,5,7,9$, and $2$ patterns each are fixed by the $2\tfrac{\pi}{10},4\tfrac{\pi}{10},6\tfrac{\pi}{10}$ and $8\tfrac{\pi}{10}$ rotations. So the average number of arrangements fixed by a rotation is $\tfrac{1}{10}(\tfrac{10!}{5!5!}+0+2+0+2+0+2+0+2+0)$
Best Answer
In (a) you’ve counted each committee $6$ times, once for each of the $6$ possible orders in which you could have picked it: ABC, ACB, BAC, BCA, CAB, and CBA. As a check we can calculate it a bit differently: there are $\binom{50}3$ ways to pick the $3$ states from which the chosen senators come, and for each state we have a choice of $2$ senators, so there are $\binom{50}32^3$ ways to choose the committee. And indeed
$$\binom{50}32^3=\frac{50!2^3}{3!47!}=\frac{50\cdot49\cdot48\cdot2^3}{3\cdot2\cdot1}=\frac{100\cdot98\cdot96}6\;.$$
HINT for (b): First calculate the total number of possible $3$-person committees, and then subtract the number in which all $3$ senators are Republicans or all $3$ senators are Democrats. It’s also feasible to calculate directly the number of committees with $2$ Republicans and $1$ Democrat and the number with $2$ Democrats and $1$ Republican and add these two figures.