A12: Color Image Segmentation

In this activity, we're to extract a region of interest (ROI) through image segmentation. This technique makes use of properties unique to the ROI in separating the ROI from the rest of the scene or image.

In grayscale images, simple thresholding may be done to separate lighter colored objects from a dark background, for example. However, when the ROI or the object's pixel values (we're usually interested in extracting an object from a scene) do not differ much from the background, segmentation will not be successful. In colored images, we have three values (RGB) to consider, hence, more unique features may be extracted from the ROI. So, although most image processing done on 2D grayscale matrices are easier than those for 3D matrices of colored images, the amount of information present in colored images is an advantage in image processing.

We demonstrate two segmentation procedures: parametric and non-parametric segmentation. Both techiniques involve three steps essentially:

1. picking out a patch that defines the color histogram of the ROI;
2. converting RGB values of each pixel (of the image and of the chosen patch) to normalized chromaticity coordinates (NCC), which are just the RGB values normalized with respect to the sum I=R+G+B: r=R/I, g=G/I, b=B/I; and
3. tagging each pixel of the image as a member or a non-member of the ROI based on the patch.

The difference between the two methods lie in the third step. We note that, since r, g, and b are normalized, their values will be between 0 and 1 and, per pixel, r+g+b=1. Hence, we can just describe the chromaticity of pixels with just the first 2 coordinates, r and g, because we know that b is just 1-r-g. The code snippet below shows the Scilab implementation for calculating r, g, and b values for an image:

img = imread(ImageDirectory+Filename+'.jpg'); //import image
img = double(img); //for flexibility in matrix operations
imgR = img(:,:,1);
imgG = img(:,:,2);
imgB = img(:,:,3);
imgI = imgR + imgG + imgB;
imgI(find(imgI==0)) = 100000; //to avoid division by zero later
//normalized chromaticity coordinates
imgr = imgR./imgI;
imgg = imgG./imgI;
imgb = imgB./imgI;

We now demonstrate both methods by attempting to extract the each of the colored hearts individually from Figure 1. Figure 2 shows the different 5x5 pixel apparently monochromatic patches for extracting each heart.

Figure 1. Scene with three objects of interest with different colors.
(source: http://www.pepperspollywogs.com/news/feb09/visual_editor_preview_data/155.jpg)

Figure 2. "Monochromatic" patches for extracting (left to right) the red, green, and blue hearts individually.

Parametric Segmentation

Here we need to recalculate the probability distribution function (PDF) of the ROI based on the patch. We can assume a Gaussian distrubution:

where and are the mean and standard deviation of r values from the patch, respectively, and r values are from the image in Figure 1. The PDF for g is calculated the same way.

Figure 3 shows the results of extraction using the monochromatic patches in Figure 2. We note that only the red heart is recognizable. So, I tried using other patches (Figure 4), this time not necessarily monochromatic and I got significantly much better results for the blue heart (Figure 5, right). The green heart (Figure 5, left), however, did not improve much. I think it's because its surface was too shiny on the picture. We note that the reflection of the objects on the surface was also extracted.

Figure 3. Results of parametric segmentation using monochromatic patches in Figure 2 for the (left to right) red, green, and blue hearts.


Figure 4. Non-monochromatic patches for green and blue hearts.


Figure 5. Results of parametric segmentation using non-monochromatic patches in Figure 4 for the green and blue hearts.

Non-Parametric Segmentation

This involves getting the 2D histogram (r on the x-axis and g on the y-axis) of the patch. For ROI membership tagging, the image r and g values are matched with histogram x and y values (patch r and g) and the histogram value becomes the tag or the new pixel value of the image. We note that to be able to match pixel values, we bin r and g for both patch and image in Scilab:

BINS = 32;
patrint = round(patr*(BINS-1)+1);
patgint = round(patg*(BINS-1)+1);

To create the 2D histogram:

colors = patgint(:) + (patrint(:)-1)*BINS;
hist = zeros(BINS,BINS);
for row = 1:BINS,
for col = 1:(BINS-row+1),
hist(row,col) = length(find(colors==(((col+(row-1)*BINS)))));
end;
end;

Finally, to tag:

imgsize = size(img);
imgmod = zeros(imgsize(1),imgsize(2));
for row = 1:imgsize(1),
for col = 1:imgsize(2),
imgmod(row,col) = hist(imgrint(row,col),imggint(row,col));
end;
end;

Figure 6 shows the resulting images. The extracted red heart (Figure 6, leftmost) is similar to the one extracted through parametric segmentation (Figure 3, leftmost). The extracted green and blue hearts are better than their parametric counterparts. In an attempt again to improve the extracted images, we use the patches in Figure 4 and we get those in Figure 6. Images were indeed improved but not as improved as those from parametric segmentation.

Figure 6. Results of non-parametric segmentation using monochromatic patches in Figure 2 for the (left to right) red, green, and blue hearts.

Figure 7. Results of parametric segmentation using non-monochromatic patches in Figure 4 for the green and blue hearts.

We then can conclude that patch selection is important in segmentation. Monochromatic patches are good for extracting objects that are not shiny but non-monochromatic patches that accounts for most of the RGB values present in the object of interest. It has also been observed that non-parametric segmentation does better than parametric segmentation when the patches are monochromatic but the latter does better when the patches are non-monochromatic.

I give myself a grade of 9 because parametric and non-parametric segmentation was demonstrated.

I would like to thank Ms. Cherry May Palomero and Ms. Irene Crisologo for useful discussions.