Remove Unwanted Parts Of Mask Image

I'm successfully calculating the mask of an image using U2NET as seen below: However, as can be seen, the mask is not completely white inside and also some unwanted artifacts can

Solution 1:

You can dynamically threshold the image using Otsu's algorithm. After that you can fill in any hole by drawing over them with OpenCV contours. I'm not sure what value range U2Net can return when masking, but you also might just be alright manually setting a small threshold value ~50.

import cv2
import numpy as np

# load image
img = cv2.imread("mask.jpg", cv2.IMREAD_GRAYSCALE);

# otsu thresholding
_, mask = cv2.threshold(img,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU);

# show
cv2.imshow("Mask", mask);
cv2.waitKey(0);

# close everything inside
contour, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE);

# get the biggest contour # returns _, contours, _ if using OpenCV 3
biggest_area = -1;
biggest = None;
for con in contour:
    area = cv2.contourArea(con);
    if biggest_area < area:
        biggest_area = area;
        biggest = con;

# fill in the contour
cv2.drawContours(mask, [biggest], -1, 255, -1);

# show
cv2.imshow("Filled Mask", mask);
cv2.waitKey(0);

Solution 2:

One solution is to use binary thresholding to make parts not white but close to white to be white.

Another solution is to use morphological operators in order to fill or remove any gaps (Note that morphological operators like erode/dilate may need multiple passes on the image in order to get desired result).

Both can be used.

To automaticaly/adaptively compute the required threshold, the procedure I can think of is similar to performing morphological operations with suitable large structure elements. That is, scan regions of the image and use majority vote of that region's pixels to color all pixels of the region.

Solution 3:

1. use min filter (e.g MinFilter(8)) = result 1

2. then apply max filter on result 1 with same values of previous min (e.g MaxFilter(8)) = result 2

3. use result 2 as Mask for the original mask in the following way:

...

//assumptions: white=255, black=0
threshold = 128;// you can change it to give more weight for white or black
for(... loop all the the pixels)
for(...)
{
if((OrginalMask[x,y] > threshold) && (result2[x,y] < threshold))
    finalResult[x,y] = result2[x,y];//to remove unwanted artifacts outside the mask
else if ((OrginalMask[x,y] < threshold) && (result2[x,y] > threshold))
    finalResult[x,y] = result2[x,y];//to make the mask completely white inside
else
    finalResult[x,y] = OrginalMask[x,y];
}

Solution 4:

Although I think this can be solved with a well placed binary threshold (as stated in another answer), adding an addition level of basic morphology should make it more robust to significantly dirtier images. (sorry it is in C++)

I used an arbitrary binary threshold to demonstrate the concept, but would suggest using a statistics based threshold (such as otsu method) if available.

To explain the code a little: threshold converts the grayscale image to binary. Opening removes external noise (any little strips, pieces, or pixel noise) left by the threshold op. Closing infills any internal holes. "opening" and "closing" are just names for combinations of dilates and erodes in specific order to achieve a desired effect without changing the underlying objects size/shape.

#include <stdio.h>
#include <opencv2/opencv.hpp>
#include <Windows.h>
#include <string>

using namespace cv;

int main(int argc, char** argv)
{
    //C:/Local Software/voyDICOM/resources/images/oXsnC.jpg
    std::string fileName = "C:/Local Software/voyDICOM/resources/images/oXsnC.jpg";
    Mat tempImage = imread(fileName, cv::IMREAD_GRAYSCALE);

    Mat bwImg;
    //binary thresh (both of these work, otsu just gets a "smarter" threshold value rather than a hardcoded one)
    cv::threshold(tempImage, bwImg, 150, 255, cv::THRESH_BINARY);
    //cv::threshold(tempImage, bwImg, 0, 255, cv::THRESH_OTSU);
    
    Mat openedImage;
    //opening
    cv::erode(bwImg, openedImage, cv::getStructuringElement(cv::MORPH_CROSS, cv::Size(3, 3)), cv::Point(-1, -1), 2);
    cv::dilate(openedImage, openedImage, cv::getStructuringElement(cv::MORPH_CROSS, cv::Size(3, 3)), cv::Point(-1, -1), 2);

    Mat closedImg;
    //closing
    cv::dilate(openedImage, closedImg, cv::getStructuringElement(cv::MORPH_CROSS, cv::Size(3, 3)),cv::Point(-1,-1),5);
    cv::erode(closedImg, closedImg, cv::getStructuringElement(cv::MORPH_CROSS, cv::Size(3, 3)), cv::Point(-1, -1), 5);


    namedWindow("Original", WINDOW_AUTOSIZE);
    imshow("Original", tempImage);

    namedWindow("Thresh", WINDOW_AUTOSIZE);
    imshow("Thresh", bwImg);

    namedWindow("Opened", WINDOW_AUTOSIZE);
    imshow("Opened", openedImage);

    namedWindow("Closed", WINDOW_AUTOSIZE);
    imshow("Closed", closedImg);

    waitKey(0);
    system("pause");
    return 0;
}

Result: