In OpenCV, I am getting an image and crop some ROIs out (using mat.submat(...)). I would like to avoid cloning the submats to save time and memory, but I am afraid that the image might be released before the rois are. Given that submat returns a Mat that uses the original Mat as its backing storage, my question is: What happens to sub-mats after their parent mat has been released? Is it safe to use the submats afterwards?
Here's the code to explain the question:
// Some big image that I get from somewhere
Mat image = Mat.zeros(1080, 1920, CvType.CV_8UC3);
Mat roi = image.submat(10, 20, 10, 20);
image.release();
// Still safe to use roi?
Mat blurredRoi = new Mat();
Imgproc.blur(roi, blurredRoi, new Size(5, 5));
OpenCV uses reference counting.
submat adds another reference to the data memory.
.release() does not deallocate the memory, unless the last reference was removed/decremented.
Related
I am using Otsu, which is a global thresholding technique, using OpenCV in java.
But I read that local thresholding techniques (e.g. Local Otsu, Sauvola, Niblack, etc...) are more effective in leaving out text from images (I am preprocessing images for OCR).
This is what I am doing:
Mat src = Imgcodecs.imread(imageFilePath, Imgcodecs.IMREAD_GRAYSCALE);
//Creating an empty matrices to store the destination image.
Mat dst = new Mat(src.rows(), src.cols(), src.type());
//Applying simple threshold
Imgproc.threshold(src, dst, 50, 255, Imgproc.THRESH_OTSU);
Bitmap bitmap = BitmapFactory.decodeFile(imageFilePath);
Utils.matToBitmap(dst, bitmap);
Should I use otsu? If so, how can be applied locally?
Or should I use another binarization algorithm?
A global method cannot work everywhere on an image if the illumination is uneven, or if the background color changes.
A local method copes with that but requires a scale parameter, which depends on the size of the features to detect. (This is a problem for general-purpose solutions where this size is unknown.)
Applying the filter in a sliding window is best because it results in a continuous threshold function, but is more time-consuming than with tiled windows.
I need to convert near white pixels to white and near black pixels to black.
I found a code snippet in python on how to do it.
hsv=cv.cvtColor(image,cv.COLOR_BGR2HSV)
# Define lower and upper limits of what we call "brown"
brown_lo=np.array([10,0,0])
brown_hi=np.array([20,255,255])
# Mask image to only select browns
mask=cv.inRange(hsv,brown_lo,brown_hi)
# Change image to red where we found brown
image[mask>0]=(0,0,255)
I have converted it java as below.
Mat temp= new Mat();
Imgproc.cvtColor(src,temp,COLOR_BGR2HSV);
Scalar low= new Scalar(10,0,0);
Scalar high= new Scalar(20,255,255);
Mat mask = new Mat();
inRange(temp,low,high,mask);
But I am facing problem converting below statement to java and there is no good opencv documentation in java with samples.
image[mask>0]=(0,0,255)
Could somebody help on how to convert above statement to java...?
I have tried setTo but it is not giving desired behaviour(attached screenshot below). Refer https://stackoverflow.com/a/50215020/12643143 for the expected result.
src.setTo(new Scalar(0,0,255),mask);
I recommend to use setTo(). This method can set all he pixels in a Mat. If an optionally mask argument is specified, then all the pixels who have a corresponding pixel with a non-zero value in the mask will be set.
Thus the python statement
image[mask>0]=(0,0,255)
can be substituted in Java by:
image.setTo(new Scalar(0, 0, 255), mask);
where image has to be a Mat object.
Answer to the question
As #Rabbid76 mentioned setTo is the correct way to do this. However if you want specific logic like image[mask>127]=(0,0,255), then do threshold (Imgproc.threshold(grey,grey, 127, 255, THRESH_BINARY);) and then use setTo.
Solution to my problem
Actually my problem was not due to setTo. Its the logic mismatch between how I read/write the Mat in my code Vs the post I referred.
I am posting the solution to the problem that I have faced so that it might help new bees like me.
Problem in reading Image
The post use Imgcodecs.imread() to read image to Mat in BGR format.
Whereas I am loading bitmap using bitmapToMat in CV_8UC4 type as below which reads the image to Mat in RGBA format.
Mat src = new Mat(bitmap.getHeight(), bitmap.getWidth(), CV_8UC4);
org.opencv.android.Utils.bitmapToMat(bitmap, src);
Fix is to convert the format properly.
Mat src = new Mat(bitmap.getHeight(), bitmap.getWidth(), CV_8UC3); //notice 3 channel
org.opencv.android.Utils.bitmapToMat(bitmap, src);
Imgproc.cvtColor(src,hsv,COLOR_RGB2HSV); //Convert RGB to HSV. COLOR_RGBA2HSV not exist, hence we load it in CV_8UC3(3 channel R,G,B).
Problem in writing the Image
Similarly as we have differences in reading between bitmapToMat and imread, the same are applicable for writing. Imgcodecs.imwrite() will write the BGR image to bitmap, where as I have to convert it back to RGB format for matToBitmap to work like Imgproc.cvtColor(rgb, rgb, Imgproc.COLOR_BGR2RGB);
I would like to preprocess a given picture by thresholding it in order to then hand it over to Tesseract. I first did that using Gimp (2.8.16) and a fixed range of 130 – 255. When I then implemented it in OpenCV (3.1) using Java, I first forgot to call cvtColor resulting in a picture that still had some colors in it (these areas were white in Gimp). Besides that, the picture was as expected. However, when I implemented the corresponding call, I got a picture that was different to the one I would have expected. It seems that the areas that were colored previously are now black while the remaining picture is similar to the one I created with Gimp.
Is there anything that I am missing to create a more similar output?
The reason I am asking this question is that, unfortunately, Tesseract (with psm 6) creates quite different results for the two images:
for the one created in Gimp: "2011 1 L 0006"
for the second one created with OpenCV: "2011ÔÇö] L 0 0006 1"
Here is the code that I used:
Mat thres = new Mat();
Mat tmp = new Mat();
Imgproc.cvtColor(src, tmp, Imgproc.COLOR_BGR2GRAY); // tmp = src.clone(); in my first attempt
Imgproc.threshold(tmp, thres, 130, 255, Imgproc.THRESH_BINARY);
Imgcodecs.imwrite("output.jpg", thres);
Here are the pictures:
Given picture:
Picture created with Gimp:
First result using OpenCV:
Second result using OpenCV:
In the first case, You are doing thresholding on color image (tmp = src.clone() creates another copy of src which is a color image). So you are getting result like that and in the second case, you are first converting to grayscale and then thresholding which gives a better result. Thresholding is good on gray scale images.
Right now, our application is running Snapdragon SDK successfully. We are trying to implement FaceDetector from Vision 8.3.0 on our project, in order to increase the number of compatible devices. We can't use CameraSource, as we rely on a custom camera + surface to provide certain functionality. We want to reuse as much code as possible, and Snapdragon SDK is doing amazingly with our current implementation.
Workflow is as follows:
1) Retrieve camera preview
2) Transform incoming byte array to bitmap (for some reason, we haven't managed to work with ByteBuffers. Image size, rotation and NV21 image format are provided and verified, but no faces are found). Bitmap is a global variable already initialized inside of processing thread, in order to avoid slowdowns from allocations.
3) Feed detector via receiveFrame
Results so far aren't good enough. Detection is way too slow (2-3 seconds) and inaccurate, even though we have disabled landmarks and classifications.
The question is: Is it possible to replicate CameraSource + Detector performance without using the former? Is is mandatory to use CameraSource to make it work with live input?
Thanks in advance!
EDIT
Following pm0733464 recommendations below, I'm trying to use ByteBuffer instead of Bitmap. This are the steps I follow:
// Initialize variables
// Mat is part of opencvSDK
Mat currentFrame = new Mat(cameraPreviewHeight + cameraPreviewHeight / 2, cameraPreviewWidth, CvType.CV_8UC1);
Mat yuvMat = new Mat(cameraPreviewHeight + cameraPreviewHeight / 2, cameraPreviewWidth, CvType.CV_8UC1);
// Load current frame
yuvMat.put(0, 0, data);
// Convert the frame to gray for better processing
Imgproc.cvtColor(yuvMat, currentFrame, Imgproc.COLOR_YUV420sp2RGB);
Imgproc.cvtColor(currentFrame, currentFrame, Imgproc.COLOR_BGR2GRAY);
From here, the byte array creation:
// Initialize grayscale byte array
byte[] grayscaleBytes = new byte[data.length];
// Extract grayscale data
currentFrame.get(0, 0, grayscaleBytes);
// Allocate ByteBuffer
ByteBuffer buffer = ByteBuffer.allocateDirect(grayscaleBytes.length);
// Wrap grayscale byte array
buffer.wrap(grayscaleBytes);
// Create frame
// rotation is calculated before
Frame currentGoogleFrame = new Frame.Builder().setImageData(buffer, currentFrame.cols(), currentFrame.rows(), ImageFormat.NV21).setRotation(rotation).build();
Constructing frames this way results in no faces found. However, using bitmaps it works as expected:
if(bitmap == null) {
// Bitmap allocation
bitmap = Bitmap.createBitmap(currentFrame.cols(), currentFrame.rows(), Bitmap.Config.ARGB_8888);
}
// Copy grayscale contents
org.opencv.android.Utils.matToBitmap(currentFrame, bitmap);
// Scale down to improve performance
Matrix scaleMatrix = new Matrix();
scaleMatrix.postScale(scaleFactor, scaleFactor);
// Recycle before creating scaleBitmap
if(scaledBitmap != null) {
scaledBitmap.recycle();
}
// Generate scaled bitmap
scaledBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight(), rotationMatrix, true);
// Create frame
// The same rotation as before is still used
if(scaledBitmap != null) {
Frame currentGoogleFrame = new Frame.Builder().setBitmap(scaledBitmap).setRotation(rotation).build();
}
Having detection take 2-3 seconds isn't typical. Using CameraSource isn't necessary to get the best performance What hardware are you using? Can you provide more specifics?
Some aspects of face detection are speed vs. accuracy trade-offs.
Speed:
Try using lower resolution images, if possible. Face detection should work fine at 640x480, for example. The face detector code does downsample large images before running detection, although this take additional time in comparison to receiving a lower resolution original.
Using ByteBuffers rather than Bitmaps will be a bit faster. The first portion of this should be just a grayscale image (no color info).
As you noted above, disabling landmarks and classification will make it faster.
In a future release, there will be a "min face size" option. Setting the min size higher makes the face detection faster (at the accuracy trade-off of not detecting smaller faces).
Setting the mode to "fast" will make it faster (at the accuracy trade-off of not detecting non-frontal faces).
Using the "prominent face only" option will be faster, but it only detects a single large face (at least 35% the width of the image).
Accuracy:
Enabling landmarks will allow the pose angles to be computed more accurately.
Setting the mode to "accurate" will detect faces at a wider range of angles (e.g., faces in profile). However, this takes more time.
Lacking the "min face size" option mentioned above, only faces larger than 10% the width of the image are detected by default. Smaller faces will not be detected. Changing this setting in the future will help to detect smaller faces. However, note that detecting smaller faces takes longer.
Using a higher resolution image will be more accurate than a lower resolution image. For example, some faces in a 320x240 image might be missed that would have been detected if the image were 640x480. The lower the "min face size" you set, the higher the resolution you need to detect faces of that size.
Make sure that you have the rotation right. The face won't be detected if it is upside down, for example. You should call the face detector again with a rotated image if you want to detect upside down faces.
Also, garbage collection time can be a factor if you're creating a lot of Bitmaps. An advantage of using ByteBuffer is that you can reuse the same buffer repeatedly without incurring per-image GC overhead that you would have encountered if you had used a Bitmap per image. CameraSource has this advantage, since it uses only a few buffers.
i Have a rectangle drawn by opencv+java+android. now i need to crop and display in my imagviwe how to crop it. submat methord is allow only int values. but rect.tl().x values are in double. casting is not a good solution. it make errors.
Core.rectangle(ImageMatin, rect.tl(), rect.br(), new Scalar(255, 0, 0),1); // works
Mat cropped = ImageMatin.submat(rect.tl().x, rect.height, rect.tl().x, rect.width); //error
i recommend this library from github: https://github.com/edmodo/cropper , it even has a sample so if you get stuck you can always refer to it. Also this library is great for cropping purposes and lightweight as well, i myself have used it in one or two apps
You can crop Mat images by appling this command:
Mat croppedImg = new Mat (sourceMat, ROI);