Because imregister works with volumetric input as well as with planar 2-D images, there is ambiguity with multi-channel 2-D images, like your RGB input image.
To avoid this ambiguity, you need to form a 2-D image from your RGB image prior to calling imregister. One way to do this is to use rgb2gray on your moving and fixed images prior to calling imregister:
fixed = rgb2gray(fixed);
moving = rgb2gray(moving);
You could also use a particular color channel, as ImageAnalyst suggested.
Otherwise, imregister is going to interpret your input imagery as a 3-plane volume. The error message you are getting is telling you that with the default number of PyramidLevels, there aren't enough pixels along the 3rd dimension to form a pyramid.
Best Answer