It makes perfect sense, and all implementations of random forests I've worked with (such as MATLAB's) provide probabilistic outputs as well to do just that.
I've not worked with the R implementation, but I'd be shocked if there wasn't a simple way to obtain soft outputs from the votes as well as the hard decision.
Edit: Just glanced at R, and predict.randomForest does output probabilities as well.
is there a good way to use the a priori knowledge that all non-c objects likely form two distinct clusters
If you are using a tree based method I don't think it matters as these classifiers partition the feature space then look at the proportion of samples in each class. So all that matters is the relative occurrence of class c in each terminal node.
If however you were using something like a mixture of normals, LDA, etc then combining two clusters would be a bad idea (assuming classes a and b form unique clusters). Here you need to preserve the class structure to accurately describe the feature space that maps to a,b and c. These models assume the features for each class have a different Normal distribution. If you combine a and b you will force a single Normal distribution to be fit to a mixture.
In summary for trees it shouldn't matter much if you:
I. Create three classifiers (1. a vs b, 2. a vs c and 3. b vs c) then predict with a voting based method.
II. Merge classes a and b to form a two-class problem.
III. Predict all three classes then map the prediction to a two class value (e.g. f(c) = c, f(a) = not c, f(b) = not c).
However if you use a method that is fitting a distribution to each class then avoid II. and test which of I. or III. works better for your problem
Best Answer
When getting up to speed on a topic, I find it helpful to start at the beginning and work forward chronologically. Breiman's original paper on random forests is where I would recommend starting.
Leo Breiman. "Random Forests." Machine Learning (2001). 45, 5-32.