I need a way to find an image on the screen. I've searched for ways to do this on SO but some take extremely long. I need it to be fast and efficient, does not need to be accurate. Basically i'm planning to compare or search for a small pixelated image, say 11x10 pixels for example, on the screen.
I also need a way to know the x and y coordinates of the small image on the screen.
Although I've looked through many tools out there like JavaCV and OpenCV, I just wanted to see if there are any other ways to do this.
TL;DR
I need a fast way to search for a small (11x10 example.) image on the screen and know its x,y coordinates.
I think you many find this answer relevant! But it is for Windows & in c++. But i'm sure that you can convert it very easily to any language.
This question is very old, But im trying to acheive the exact same thing here. Ive found that combining these answers would do the trick:
Convert BufferedImage TYPE_INT_RGB to OpenCV Mat Object
OpenCV Template Matching example in Android
The reason you need to do a conversion is because when u grab a screenshot with awt.Robot class its in the INT_RGB format. The matching template example expects bytes and you cannot grab byte data from this type of image directly.
Heres my implementation of these two answers, but it is incomplete. The output is all screwed up and i think it may have something to do with the IntBuffer/ByteBuffers.
-Edit-
I've added a new helper method that converts a INT_RGB to a BYTE_BGR. I can now grab the coordinates of template on the image using matchLoc.This seems to work pretty well, I was able to use this with a robot that clicks the start menu for me based on the template.
private BufferedImage FindTemplate() {
System.out.println("\nRunning Template Matching");
int match_method = Imgproc.TM_SQDIFF;
BufferedImage screenShot = null;
try {
Robot rob = new Robot();
screenShot = rob.createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
} catch (AWTException ex) {
Logger.getLogger(MainGUI.class.getName()).log(Level.SEVERE, null, ex);
}
if(screenShot == null) return;
Mat img = BufferedImageToMat(convertIntRGBTo3ByteBGR(screenShot));
String templateFile = "C:\\Temp\\template1.JPG";
Mat templ = Highgui.imread(templateFile);
// / Create the result matrix
int result_cols = img.cols() - templ.cols() + 1;
int result_rows = img.rows() - templ.rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32FC1);
// / Do the Matching and Normalize
Imgproc.matchTemplate(img, templ, result, match_method);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
Highgui.imwrite("out2.png", result);
// / Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (match_method == Imgproc.TM_SQDIFF
|| match_method == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
Graphics2D graphics = screenShot.createGraphics();
graphics.setColor(Color.red);
graphics.setStroke(new BasicStroke(3));
graphics.drawRect(matchLoc.x, matchLoc.y, templ.width(), templ.height());
graphics.dispose();
return screenShot;
}
private Mat BufferedImageToMat(BufferedImage img){
int[] data = ((DataBufferInt) img.getRaster().getDataBuffer()).getData();
ByteBuffer byteBuffer = ByteBuffer.allocate(data.length * 4);
IntBuffer intBuffer = byteBuffer.asIntBuffer();
intBuffer.put(data);
Mat mat = new Mat(img.getHeight(), img.getWidth(), CvType.CV_8UC3);
mat.put(0, 0, byteBuffer.array());
return mat;
}`
private BufferedImage convertIntRGBTo3ByteBGR(BufferedImage img){
BufferedImage convertedImage = new BufferedImage(img.getWidth(), img.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
Graphics2D graphics = convertedImage.createGraphics();
graphics.drawImage(img, 0, 0, null);
graphics.dispose();
return convertedImage;
}
Results:
Template:
Related
please help me with this as I am new to image processing.
I have the following images, there is glitterings/powder on the surface of the hand. How do I go about detecting these things on the hand?
I have tried with detecting by getting 70% of the max Intensity in the image. However, only image one works the rest does not. Could anyone how to suggest any methods that I can use to perform the detection please.(any available codes to try with will be good) Thank you?
Input Image 1: The only image that works with the above code
Input Image 2
Input Image 3
Desired Outcome
//convert from bitmap to mat
Mat mat = new Mat(bitmap1.getWidth(), bitmap1.getHeight(), CvType.CV_8UC3);
Utils.bitmapToMat(bitmap1, mat);
Mat grayMat = new Mat();
Imgproc.cvtColor(mat, grayMat, Imgproc.COLOR_BGR2GRAY);
//Log.d("TAGG","intensity" + mat.dump());
int rows = mat.rows();
int cols = mat.cols();
double maxIntensity = Core.minMaxLoc(grayMat).maxVal;
double minIntensity = 0.7 * maxIntensity;
Log.d("TAGG", "intensity" + maxIntensity);
Mat thresholdMat = new Mat();
Imgproc.threshold(grayMat, thresholdMat, minIntensity, maxIntensity, Imgproc.THRESH_BINARY_INV);
Bitmap outputBitmap = Bitmap.createBitmap(thresholdMat.cols(), thresholdMat.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(thresholdMat, outputBitmap);
I want to get a region of interest (ROI) from an SWT Image object.
I am unable to find a method that will do this for me. Is it possible to avoid writing one by myself?
I need something like (similar to opencv implementations):
Image img = new Image(display, path);
Rectangle roi = new Rectangle(0, 0, 10, 10);
Image imgRoi = img.getROI(roi);
I'm not aware of a base SWT method to create images from a region, but I found a snippet that should do that here https://gist.github.com/azhawkes/4347761 (credits goes to the author Andy Hawkes).
Here I slightly modified the method to get in input the original image. I also implemented an optimization suggested by greg-449 (minimized the calls to Image.getImageData).
public static Image loadImageFromRegion(Image original, Rectangle region) {
ImageData originalImageData = original.getImageData();
ImageData data = new ImageData(region.width, region.height, originalImageData.depth, originalImageData.palette);
int[] pixels = new int[region.width];
byte[] alphas = new byte[region.width];
for (int y = 0; y < region.height; y++) {
originalImageData.getAlphas(region.x, region.y + y, region.width, alphas, 0);
originalImageData.getPixels(region.x, region.y + y, region.width, pixels, 0);
data.setPixels(0, y, region.width, pixels, 0);
data.setAlphas(0, y, region.width, alphas, 0);
}
return new Image(Display.getCurrent(), data);
}
I recently started working with the Java bindings for OpenCV to make a quick and dirty project to do template matching. Basically I am trying to read a set of jpg images (saved in MS Paint) into Mats and then use template matching to find their locations from a screen shot taken with Java.Robot.
When it comes time to do the template matching this error is thrown
OpenCV Error: Assertion failed ((depth == CV_8U || depth == CV_32F)
&& type == _templ.type() && _img.dims() <= 2) in cv::matchTemplate
After searching it looks like the issue is that the two Mats I am trying to use do not have the same "type". What I am not sure of is what this refers to. I assume it is the Mats CvType, if I print out the CvType of the image and template I get a type() of 4 == CvType.CV_32SC1 for my template I get a type() of 20 == CvType.CV_32SC3.
But I feel like this is not the correct type() I am trying to compare, I have feeling it refers to the data type of how the data is stored in the Mat? But I have no good links to back this up just remembrances from many SO searches.
Here is my code for loading in my jpg images into a Mat
Mat pic_ = Imgcodecs.imread("MyPath\\image.jpg");
pic_.convertTo(pic_, CvType.CV_32SC1);
Here the second line turns my type() from 20 to 16, though as per my last comment I don't think this is the proper way to alter the Mat to match the image?... Because convertTo'ing this Mat to match the type of the screen shot `(below) does not fix the error?
Here is how I am creating the image Mat
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage screenShot = rob.createScreenCapture(screenRect);
Mat screenImage = bufferedImageToMat(screenShot);
So I first take a screenshot with Java.Robot.createScreenCapture I then convert it to a Mat with
private Mat bufferedImageToMat(BufferedImage inBuffImg)
{
BufferedImage image = new BufferedImage(inBuffImg.getWidth(), inBuffImg.getHeight(), BufferedImage.TYPE_INT_RGB);
Graphics2D g2d= image.createGraphics();
g2d.drawImage(inBuffImg, 0, 0, null);
g2d.dispose();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_32SC1);
int[] data = ((DataBufferInt) image.getRaster().getDataBuffer()).getData();
mat.put(0, 0, data);
return mat;
}
From what I could tell the BufferedImage created by Robot is of type BufferedImage.TYPE_3BYTE_BGR which gives me an error "DataBufferInt cannot be cast to DataBufferByte" when trying to get the pixel data. So per the linked question I redraw the BufferedImage as type BufferedImage.TYPE_INT_RGB and pull out the data as a DataBufferInt.
So in all, should I be trying to match the Mat.type() or does my problem lie elsewhere? If not elsewhere how can I alter either of the Mats so that they can be used with Imgproc.matchTemplate properly?
I feel like the easiest solution would be to convert the image loaded from file to match the screenshot Mat?
EDIT: The exact section of code that gives the error is below
// Mat imageTemplate is a function argument; the loaded jpg image
// Take a picture of the screen
Rectangle screenRect = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
BufferedImage screenShot = rob.createScreenCapture(screenRect);
Mat screenImage = bufferedImageToMat(screenShot);
// Create the result matrix
int result_cols = screenImage.cols() - imageTemplate.cols() + 1;
int result_rows = screenImage.rows() - imageTemplate.rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32SC1);
newStatus("ScreenType: " + screenImage.type());
newStatus("TemplaType: " + imageTemplate.type());
// Choose a matching method
int matchMethod = Imgproc.TM_SQDIFF_NORMED;
// Do the Matching and Normalize
Imgproc.matchTemplate(screenImage, imageTemplate, result, matchMethod);
// Error occurs on previous line
As #Miki pointed out in the comments the answer was getting the channe type to match for the image and template. I ended up changing my bufferedImageToMat function.
private Mat bufferedImageToMat(BufferedImage inBuffImg)
{
BufferedImage image = new BufferedImage(inBuffImg.getWidth(), inBuffImg.getHeight(), BufferedImage.TYPE_4BYTE_ABGR);
Graphics2D g2d= image.createGraphics();
g2d.drawImage(inBuffImg, 0, 0, null);
g2d.dispose();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC3);
byte[] data = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
mat.put(0, 0, data);
return mat;
}
My templates are read in as CvType.CV_8UC3, so it was just a matter of creating a Mat from the screen image with this type!
As the title suggests, I am interested in getting the HSV value of a specific pixel using java CV. This sounds easy enough, and it seems to be straight forward in C++ or Python, but I simply cant figure out how to do it in Java. I am pretty new to OpenCV, and if I decide to do more projects using this library I will definitely write them in C++ or Python.
For reference, my goal is to do a color analysis of an object that has varying levels of lighting. The end goal is to be able to take an image of something like a t-shirt and be able to say "this t shirt is x% red".
Here is some of the code I was using. Surprisingly inRange() takes much longer than just looping through every pixel and getting RGB one by one. I want to be able to do exactly this, just with the HSV color space. If you know of a better way to accomplish this goal, please let me know as this has destroyed my entire Saturday. Thanks!
Scalar min = new Scalar(22,11,3);
Scalar max = new Scalar(103,87,74);
int sum = 0;
System.loadLibrary( Core.NATIVE_LIBRARY_NAME );
File input = new File("bluesample.jpg");
BufferedImage image = ImageIO.read(input);
byte[] data = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC3);
Mat mat1 = new Mat(image.getHeight(),image.getWidth(),CvType.CV_8UC3);
mat.put(0, 0, data);
Core.inRange(mat, min, max, mat1);
System.out.println(mat1.total());
System.out.println(mat1.total());
for (int i=0;i<mat1.rows(); i++ ){
for (int j=0;i<mat1.cols();j++){
sum += mat1.get(j, i, data);
}
}
System.out.println(sum/mat1.total());
EDIT:
try { System.loadLibrary( Core.NATIVE_LIBRARY_NAME );
File input = new File("singlehsvpix.jpg");
BufferedImage image = ImageIO.read(input);
byte[] data = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
Mat mat = new Mat(image.getHeight(), image.getWidth(), CvType.CV_8UC3);
mat.put(0, 0, data);
Mat mat1 = new Mat(image.getHeight(),image.getWidth(),CvType.CV_8UC1);
Imgproc.cvtColor(mat, mat1, Imgproc.COLOR_RGB2HSV);
System.out.println(mat1.dump());
byte[] data1 = new byte[mat1.rows() * mat1.cols() * (int)(mat1.elemSize())];
mat1.get(0, 0, data1);
//BufferedImage image1 = new BufferedImage(mat1.cols(),mat1.rows(), BufferedImage.TYPE_BYTE_GRAY);
BufferedImage image1 = new BufferedImage(mat1.cols(),mat1.rows(), 5);
image1.getRaster().setDataElements(0, 0, mat1.cols(), mat1.rows(), data1);
File output = new File("PLS!.jpg");
ImageIO.write(image1, "jpg", output);
System.out.println(mat1.get(0, 0, data1)); // RELEVANT LINE
System.out.println("Done");
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
}
Is printing:
[ 54, 213, 193]
3
Done
For this pic, 54, 213, 193 are the BGR values... I guess I don't understand enough about OpenCV to know why my mat1.get is printing 3
So, you want to convert rgb to hsv.
Imgproc.cvtColor(im_rgb, im_hsv, Imgproc.COLOR_RGB2HSV);
Then, process as you like
Edit: in your code, change mat to mat1
for (int i=0;i<mat1.rows(); i++ ){
for (int j=0;i<mat1.cols();j++){
sum += mat.get(j, i, data); //this line
}
}
System.out.println(sum/mat1.total());
You are adding the value in original matrix.
I'd like to convert gif images to jpeg using Java. It works great for most images, but I have a simple transparent gif image:
Input gif image http://img292.imageshack.us/img292/2103/indexedtestal7.gif
[In case the image is missing: it's a blue circle with transparent pixels around it]
When I convert this image using the following code:
File file = new File("indexed_test.gif");
BufferedImage image = ImageIO.read(file);
File f = new File("indexed_test.jpg");
ImageIO.write(image, "jpg", f);
This code works without throwing an Exception, but results an invalid jpeg image:
[In case the image is missing: IE cannot show the jpeg, Firefox shows the image with invalid colors.]
I'm using Java 1.5.
I also tried converting the sample gif to png with gimp and using the png as an input for the Java code. The result is the same.
Is it a bug in the JDK? How can I convert images correctly preferably without 3rd party libraries?
UPDATE:
Answers indicate that jpeg conversion cannot handle transparency correctly (I still think that this is a bug) and suggest a workaround for replacing transparent pixels with predefined color. Both of the suggested methods are quite complex, so I've implemented a simpler one (will post as an answer). I accept the first published answer with this workaround (by Markus). I don't know which implementation is the better. I go for the simplest one still I found a gif where it's not working.
For Java 6 (and 5 too, I think):
BufferedImage bufferedImage = new BufferedImage(image.getWidth(null), image.getHeight(null), BufferedImage.TYPE_INT_RGB);
g = bufferedImage.createGraphics();
//Color.WHITE estes the background to white. You can use any other color
g.drawImage(image, 0, 0, bufferedImage.getWidth(), bufferedImage.getHeight(), Color.WHITE, null);
As already mentioned in the UPDATE of the question I've implemented a simpler way of replacing transparent pixels with predefined color:
public static BufferedImage fillTransparentPixels( BufferedImage image,
Color fillColor ) {
int w = image.getWidth();
int h = image.getHeight();
BufferedImage image2 = new BufferedImage(w, h,
BufferedImage.TYPE_INT_RGB);
Graphics2D g = image2.createGraphics();
g.setColor(fillColor);
g.fillRect(0,0,w,h);
g.drawRenderedImage(image, null);
g.dispose();
return image2;
}
and I call this method before jpeg conversion in this way:
if( inputImage.getColorModel().getTransparency() != Transparency.OPAQUE) {
inputImage = fillTransparentPixels(inputImage, Color.WHITE);
}
The problem (at least with png to jpg conversion) is that the color scheme isn't the same, because jpg doesn't support transparency.
What we've done successfully is something along these lines (this is pulled from various bits of code - so please forgive the crudeness of the formatting):
File file = new File("indexed_test.gif");
BufferedImage image = ImageIO.read(file);
int width = image.getWidth();
int height = image.getHeight();
BufferedImage jpgImage;
//you can probably do this without the headless check if you just use the first block
if (GraphicsEnvironment.isHeadless()) {
if (image.getType() == BufferedImage.TYPE_CUSTOM) {
//coerce it to TYPE_INT_ARGB and cross fingers -- PNGs give a TYPE_CUSTOM and that doesn't work with
//trying to create a new BufferedImage
jpgImage = new BufferedImage(width,height,BufferedImage.TYPE_INT_ARGB);
} else {
jpgImage = new BufferedImage(width, height, image.getType());
}
} else {
jgpImage = GraphicsEnvironment.getLocalGraphicsEnvironment().
getDefaultScreenDevice().getDefaultConfiguration().
createCompatibleImage(width, height, image.getTransparency());
}
//copy the original to the new image
Graphics2D g2 = null;
try {
g2 = jpg.createGraphics();
g2.setRenderingHint(RenderingHints.KEY_INTERPOLATION,
RenderingHints.VALUE_INTERPOLATION_BICUBIC);
g2.drawImage(image, 0, 0, width, height, null);
}
finally {
if (g2 != null) {
g2.dispose();
}
}
File f = new File("indexed_test.jpg");
ImageIO.write(jpgImage, "jpg", f);
This works for png to jpg and gif to jpg. And you will have a white background where the transparent bits were. You can change this by having g2 fill the image with another color before the drawImage call.
3 months late, but I am having a very similar problem (although not even loading a gif, but simply generating a transparent image - say, no background, a colored shape - where when saving to jpeg, all colors are messed up, not only the background)
Found this bit of code in this rather old thread of the java2d-interest list, thought I'd share, because after a quick test, it is much more performant than your solution:
final WritableRaster raster = img.getRaster();
final WritableRaster newRaster = raster.createWritableChild(0, 0, img.getWidth(), img.getHeight(), 0, 0, new int[]{0, 1, 2});
// create a ColorModel that represents the one of the ARGB except the alpha channel
final DirectColorModel cm = (DirectColorModel) img.getColorModel();
final DirectColorModel newCM = new DirectColorModel(cm.getPixelSize(), cm.getRedMask(), cm.getGreenMask(), cm.getBlueMask());
// now create the new buffer that we'll use to write the image
return new BufferedImage(newCM, newRaster, false, null);
Unfortunately, I can't say I understand exactly what it does ;)
If you create a BufferedImage of type BufferedImage.TYPE_INT_ARGB and save to JPEG weird things will result. In my case the colors are scewed into orange. In other cases the produced image might be invalid and other readers will refuse loading it.
But if you create an image of type BufferedImage.TYPE_INT_RGB then saving it to JPEG works fine.
I think this is therefore a bug in Java JPEG image writer - it should write only what it can without transparency (like what .NET GDI+ does). Or in the worst case thrown an exception with a meaningful message e.g. "cannot write an image that has transparency".
JPEG has no support for transparency. So even when you get the circle color correctly you will still have a black or white background, depending on your encoder and/or renderer.
BufferedImage originalImage = ImageIO.read(getContent());
BufferedImage newImage = new BufferedImage(originalImage.getWidth(), originalImage.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
for (int x = 0; x < originalImage.getWidth(); x++) {
for (int y = 0; y < originalImage.getHeight(); y++) {
newImage.setRGB(x, y, originalImage.getRGB(x, y));
}
}
ImageIO.write(newImage, "jpg", f);
7/9/2020 Edit: added imageIO.write