We Keep Coding

Trying to fill color on square-like object using OpenCV and Java

I am trying to fill color to that square objects. It does not look like a perfect square. I have attached 4 sample images. Please have a look at it. Please help me with your ideas.
By using this code I can change the background and line colors. But I want to fill color that unshaped squares (squares only not triangle).
System.loadLibrary( Core.NATIVE_LIBRARY_NAME );
String img_url1 = "C:\\\\Users\\\\me\\\\Desktop\\\\cpt\\\\1.png";
Mat img = Imgcodecs.imread(img_url1, -1);
List<Mat> channels = new ArrayList<>();
Core.split(img, channels);
Mat black = new Mat();
Core.bitwise_not(channels.get(3), black);
String file2 = "C:\\\\\\\\Users\\\\\\\\me\\\\\\\\Desktop\\\\\\\\cpt\\\\\\\\1-cpy.png";
Imgcodecs.imwrite(file2, black);
Input Images :

How to set a Mat object into color mode? [OpenCV]

I am doing a tutorial on Image Segmentation using Watershed Algorithm and I want to know if I am getting the right result after segmenting the image into foreground and background.
The requirement is to view my finalMat object's foreground back to color mode using an imageView. I am using 'a leaf' photo as an example and I want to see the green back to please myself that I am getting the object out of its background.
Here is my code (Please feel free to comment on what am I doing wrong, or a better solution that I came up with. I am new to android development and OpenCV):
Mat finalMat = markersMat;
finalMat.convertTo(markersMat, CvType.CV_32S);
Imgproc.watershed(originalPicMat, finalMat);
//then return as CV_8U:
finalMat.convertTo(finalMat, CvType.CV_8U);
//test at final output:
Bitmap testBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(finalMat, testBmp);
imgView_finalOutput.setImageBitmap(testBmp);
Here is my code for the marker:
//Markers image:
Mat markersMat = new Mat(grayscaleMat.size(), CvType.CV_8U, new Scalar(0));
Core.add(foregroundMat, backgroundMat, markersMat);
//show markers:
Bitmap markersBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(markersMat, markersBmp);
imgView_markers.setImageBitmap(markersBmp);
Again, my question is how can I set the finalMat in color mode(return the color of the leaf)?
Here is my full code for the activity:
//Set Original pic:
String originalFilePathStr= Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES) +"/healthy1.jpg";
Mat originalPicMat = Imgcodecs.imread(originalFilePathStr);
//set imageview:
//File originalPicFilePath = new File(originalFilePathStr+"/healthy.jpg");
Uri originalPicUriPath = Uri.parse(originalFilePathStr);
imgView_original.setImageURI(originalPicUriPath);
/* IMAGE SEGMENTATION USING WATERSHED ALGORITHM */
//Create a Mat Object using originalPicture as is:
/* OTSU'S BINARIZATION */
Mat grayscaleMat = new Mat();
//check if i dont need to load_image as grayscale, if cvtColor does does work or vice versa:
Imgproc.cvtColor(originalPicMat, grayscaleMat, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold(grayscaleMat, grayscaleMat, 0, 255, Imgproc.THRESH_BINARY_INV + Imgproc.THRESH_OTSU);
//test grayscale: WORKS:
Bitmap grayscaleBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(grayscaleMat, grayscaleBmp);
imgView_grayscale.setImageBitmap(grayscaleBmp);
//Part 2: Create marker: foregroun dnd background
//Sure fg area:
Mat foregroundMat = new Mat();
Imgproc.erode(grayscaleMat, foregroundMat, new Mat(),new Point(-1,-1),2);
//Show fg area:
Bitmap foregroundBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(grayscaleMat, foregroundBmp);
imgView_foreground.setImageBitmap(foregroundBmp);
//Mat Bg:
Mat backgroundMat = new Mat();
Imgproc.dilate(grayscaleMat, backgroundMat, new Mat(), new Point(-1,-1), 3);
Imgproc.threshold(backgroundMat, backgroundMat, 1,128, Imgproc.THRESH_BINARY_INV);
Bitmap backgroundBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(backgroundMat, backgroundBmp);
imgView_background.setImageBitmap(backgroundBmp);
//Markers image:
Mat markersMat = new Mat(grayscaleMat.size(), CvType.CV_8U, new Scalar(0));
Core.add(foregroundMat, backgroundMat, markersMat);
//show markers:
Bitmap markersBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(markersMat, markersBmp);
imgView_markers.setImageBitmap(markersBmp);
/*Watershed: Finale */
//test watershed:
Mat finalMat = markersMat;
finalMat.convertTo(markersMat, CvType.CV_32S);
Imgproc.watershed(originalPicMat, finalMat);
//then return as CV_8U:
finalMat.convertTo(finalMat, CvType.CV_8U);
//test at final output:
Bitmap testBmp = Bitmap.createBitmap(grayscaleMat.cols(), grayscaleMat.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(finalMat, testBmp);
imgView_finalOutput.setImageBitmap(testBmp);
Here is a screenshot of my activity result (Not enough reputation points to post a screenshot).
UPDATE:
It's now working, I can mark now the leaf. The only problem that I have now is that it returns the leaf in a different color. Please refer to this photo.
The final photo returns a different shade of green and yellow turns to blue. The Problem is that I don't know where the bug came from. If you guys have an idea on where is the problem please tell me so.

Loading image from file as given "type" into an OpenCV Mat?

I recently started working with the Java bindings for OpenCV to make a quick and dirty project to do template matching. Basically I am trying to read a set of jpg images (saved in MS Paint) into Mats and then use template matching to find their locations from a screen shot taken with Java.Robot.
When it comes time to do the template matching this error is thrown
OpenCV Error: Assertion failed ((depth == CV_8U || depth == CV_32F)
&& type == _templ.type() && _img.dims() <= 2) in cv::matchTemplate
After searching it looks like the issue is that the two Mats I am trying to use do not have the same "type". What I am not sure of is what this refers to. I assume it is the Mats CvType, if I print out the CvType of the image and template I get a type() of 4 == CvType.CV_32SC1 for my template I get a type() of 20 == CvType.CV_32SC3.
But I feel like this is not the correct type() I am trying to compare, I have feeling it refers to the data type of how the data is stored in the Mat? But I have no good links to back this up just remembrances from many SO searches.
Here is my code for loading in my jpg images into a Mat
Mat pic_ = Imgcodecs.imread("MyPath\\image.jpg");
pic_.convertTo(pic_, CvType.CV_32SC1);
Here the second line turns my type() from 20 to 16, though as per my last comment I don't think this is the proper way to alter the Mat to match the image?... Because convertTo'ing this Mat to match the type of the screen shot `(below) does not fix the error?
Here is how I am creating the image Mat
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage screenShot = rob.createScreenCapture(screenRect);
Mat screenImage = bufferedImageToMat(screenShot);
So I first take a screenshot with Java.Robot.createScreenCapture I then convert it to a Mat with
private Mat bufferedImageToMat(BufferedImage inBuffImg)
{
BufferedImage image = new BufferedImage(inBuffImg.getWidth(), inBuffImg.getHeight(), BufferedImage.TYPE_INT_RGB);
Graphics2D g2d= image.createGraphics();
g2d.drawImage(inBuffImg, 0, 0, null);
g2d.dispose();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_32SC1);
int[] data = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
mat.put(0, 0, data);
return mat;
}
From what I could tell the BufferedImage created by Robot is of type BufferedImage.TYPE_3BYTE_BGR which gives me an error "DataBufferInt cannot be cast to DataBufferByte" when trying to get the pixel data. So per the linked question I redraw the BufferedImage as type BufferedImage.TYPE_INT_RGB and pull out the data as a DataBufferInt.
So in all, should I be trying to match the Mat.type() or does my problem lie elsewhere? If not elsewhere how can I alter either of the Mats so that they can be used with Imgproc.matchTemplate properly?
I feel like the easiest solution would be to convert the image loaded from file to match the screenshot Mat?
EDIT: The exact section of code that gives the error is below
// Mat imageTemplate is a function argument; the loaded jpg image
// Take a picture of the screen
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage screenShot = rob.createScreenCapture(screenRect);
Mat screenImage = bufferedImageToMat(screenShot);
// Create the result matrix
int result_cols = screenImage.cols() - imageTemplate.cols() + 1;
int result_rows = screenImage.rows() - imageTemplate.rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32SC1);
newStatus("ScreenType: " + screenImage.type());
newStatus("TemplaType: " + imageTemplate.type());
// Choose a matching method
int matchMethod = Imgproc.TM_SQDIFF_NORMED;
// Do the Matching and Normalize
Imgproc.matchTemplate(screenImage, imageTemplate, result, matchMethod);
// Error occurs on previous line

As #Miki pointed out in the comments the answer was getting the channe type to match for the image and template. I ended up changing my bufferedImageToMat function.
private Mat bufferedImageToMat(BufferedImage inBuffImg)
{
BufferedImage image = new BufferedImage(inBuffImg.getWidth(), inBuffImg.getHeight(), BufferedImage.TYPE_4BYTE_ABGR);
Graphics2D g2d= image.createGraphics();
g2d.drawImage(inBuffImg, 0, 0, null);
g2d.dispose();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC3);
byte[] data = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
mat.put(0, 0, data);
return mat;
}
My templates are read in as CvType.CV_8UC3, so it was just a matter of creating a Mat from the screen image with this type!

Quick & fast template matching on screen. Coordinates needed too. Java

I need a way to find an image on the screen. I've searched for ways to do this on SO but some take extremely long. I need it to be fast and efficient, does not need to be accurate. Basically i'm planning to compare or search for a small pixelated image, say 11x10 pixels for example, on the screen.
I also need a way to know the x and y coordinates of the small image on the screen.
Although I've looked through many tools out there like JavaCV and OpenCV, I just wanted to see if there are any other ways to do this.
TL;DR
I need a fast way to search for a small (11x10 example.) image on the screen and know its x,y coordinates.

I think you many find this answer relevant! But it is for Windows & in c++. But i'm sure that you can convert it very easily to any language.

This question is very old, But im trying to acheive the exact same thing here. Ive found that combining these answers would do the trick:
Convert BufferedImage TYPE_INT_RGB to OpenCV Mat Object
OpenCV Template Matching example in Android
The reason you need to do a conversion is because when u grab a screenshot with awt.Robot class its in the INT_RGB format. The matching template example expects bytes and you cannot grab byte data from this type of image directly.
Heres my implementation of these two answers, but it is incomplete. The output is all screwed up and i think it may have something to do with the IntBuffer/ByteBuffers.
-Edit-
I've added a new helper method that converts a INT_RGB to a BYTE_BGR. I can now grab the coordinates of template on the image using matchLoc.This seems to work pretty well, I was able to use this with a robot that clicks the start menu for me based on the template.
private BufferedImage FindTemplate() {
System.out.println("\nRunning Template Matching");
int match_method = Imgproc.TM_SQDIFF;
BufferedImage screenShot = null;
try {
Robot rob = new Robot();
screenShot = rob.createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
} catch (AWTException ex) {
Logger.getLogger(MainGUI.class.getName()).log(Level.SEVERE, null, ex);
}
if(screenShot == null) return;
Mat img = BufferedImageToMat(convertIntRGBTo3ByteBGR(screenShot));
String templateFile = "C:\\Temp\\template1.JPG";
Mat templ = Highgui.imread(templateFile);
// / Create the result matrix
int result_cols = img.cols() - templ.cols() + 1;
int result_rows = img.rows() - templ.rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32FC1);
// / Do the Matching and Normalize
Imgproc.matchTemplate(img, templ, result, match_method);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
Highgui.imwrite("out2.png", result);
// / Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (match_method == Imgproc.TM_SQDIFF
|| match_method == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
Graphics2D graphics = screenShot.createGraphics();
graphics.setColor(Color.red);
graphics.setStroke(new BasicStroke(3));
graphics.drawRect(matchLoc.x, matchLoc.y, templ.width(), templ.height());
graphics.dispose();
return screenShot;
}
private Mat BufferedImageToMat(BufferedImage img){
int[] data = ((DataBufferInt) img.getRaster().getDataBuffer()).getData();
ByteBuffer byteBuffer = ByteBuffer.allocate(data.length * 4);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(data);
Mat mat = new Mat(img.getHeight(), img.getWidth(), CvType.CV_8UC3);
mat.put(0, 0, byteBuffer.array());
return mat;
}`
private BufferedImage convertIntRGBTo3ByteBGR(BufferedImage img){
BufferedImage convertedImage = new BufferedImage(img.getWidth(), img.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
Graphics2D graphics = convertedImage.createGraphics();
graphics.drawImage(img, 0, 0, null);
graphics.dispose();
return convertedImage;
}
Results:
Template:

OpenCV on Android converting to grayscale not working

I am trying to convert some OpenCV Mat to grayscale for Contours detection algorithms. For some reason the image after convertion is all black. Mine code (b is Android Bitmap):
Mat tmp = new Mat (b.getWidth(), b.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(b, tmp);
Imgproc.cvtColor(tmp, tmp, Imgproc.COLOR_BGR2GRAY);
//there could be some processing
Imgproc.cvtColor(tmp, tmp, Imgproc.COLOR_GRAY2BGRA, 4);
Utils.matToBitmap(tmp, b);
And now I am drawing this bitmap and it is all black. When I applied contour detection to this bitmap (in place of comment) there were no match, so I assume problem is with convertion. After removing convertion (simply call bitmapToMat and matToBitmap) then bitmap is not all black so convertion to Mat are also working. Bitmap is in ARGB_8888 and there are no errors, just the output bitmap is all black.
Edit: Just to be sure I tried to save a bitmap with ocv imwrite - it's still all black so the problem is for 100% at cvtColor...

If the bitmap b is from the android device, then try using COLOR_RGB2GRAY instead of COLOR_BGR2GRAY because BGR is the default pixel format for OpenCV images, not all images.
Try this code:
Mat tmp = new Mat (b.getWidth(), b.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(b, tmp);
Imgproc.cvtColor(tmp, tmp, Imgproc.COLOR_RGB2GRAY);
//there could be some processing
Imgproc.cvtColor(tmp, tmp, Imgproc.COLOR_GRAY2RGB, 4);
Utils.matToBitmap(tmp, b);

I tried this method and it works perfectly, first you must convert it into grey scale, then to Canny, and finally to Bitmap.
this function returns a black and white image.
public static Bitmap edgesim(Mat img1) {
Bitmap image1;
//mat gray img1 holder
Mat imageGray1 = new Mat();
//mat canny image
Mat imageCny1 = new Mat();
//mat canny image
Mat imageCny2 = new Mat();
/////////////////////////////////////////////////////////////////
//Convert img1 into gray image
Imgproc.cvtColor(img1, imageGray1, Imgproc.COLOR_BGR2GRAY);
//Canny Edge Detection
Imgproc.Canny(imageGray1, imageCny1, 10, 100, 3, true);
///////////////////////////////////////////////////////////////////
//////////////////Transform Canny to Bitmap/////////////////////////////////////////
image1= Bitmap.createBitmap(imageCny1.cols(), imageCny1.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(imageCny1, image1);
return image1;
}