MATLAB: How to complete one of these two rectangles? I want to use it as mask for RGB image.

Question

Answer 1

Use bwconvhull();

% Get convex hull of mask.
mask = bwconvhull(mask);
% Mask the image using bsxfun() function
maskedRgbImage = bsxfun(@times, rgbImage, cast(mask, class(rgbImage)));

You can use ~mask in the cast() if you want the inverse of the mask.