MATLAB: Splitting an image by color (not RGB)

image analysisMATLAB

I'm trying to take an image (attached) and split it into five images, by paint color. I know how to split the RGB layers, but I need to split it by paint color instead. I could certainly feed in exemplars of small captures from the image to define the colors, but I'm not sure how to do the analysis with MATLAB.
Thanks for the help! jg