MATLAB: How to categorize the rectangle or circle

circlerectangular

What program is used for choosing rectangle or circle?

You probably want to look at regionprops() and the "Eccentricity" measurement. Circles have an eccentricity of 0, but rectangles never do.

In the line

biggestBlob=ismember(labeledImage,indexOfBiggest(1));

you are finding the biggest by looking at the first index in a sorted array. So to get the second biggest you do

secondBlob=ismember(labeledImage,indexOfBiggest(2));

Then you can combine them simply by doing

twoBiggest = biggestBlob | secondBlob;

This is because both blobs are binary - 1s and 0s - so by doing an OR operation it makes every white part in both images seen in both images.

No, you don't look at the eccentricity. You can look either at the orientation, or check the width vs. height of the bounding box.