Right now, our application is running Snapdragon SDK successfully. We are trying to implement FaceDetector from Vision 8.3.0 on our project, in order to increase the number of compatible devices. We can't use CameraSource, as we rely on a custom camera + surface to provide certain functionality. We want to reuse as much code as possible, and Snapdragon SDK is doing amazingly with our current implementation.
Workflow is as follows:
1) Retrieve camera preview
2) Transform incoming byte array to bitmap (for some reason, we haven't managed to work with ByteBuffers. Image size, rotation and NV21 image format are provided and verified, but no faces are found). Bitmap is a global variable already initialized inside of processing thread, in order to avoid slowdowns from allocations.
3) Feed detector via receiveFrame
Results so far aren't good enough. Detection is way too slow (2-3 seconds) and inaccurate, even though we have disabled landmarks and classifications.
The question is: Is it possible to replicate CameraSource + Detector performance without using the former? Is is mandatory to use CameraSource to make it work with live input?
Thanks in advance!
EDIT
Following pm0733464 recommendations below, I'm trying to use ByteBuffer instead of Bitmap. This are the steps I follow:
// Initialize variables
// Mat is part of opencvSDK
Mat currentFrame = new Mat(cameraPreviewHeight + cameraPreviewHeight / 2, cameraPreviewWidth, CvType.CV_8UC1);
Mat yuvMat = new Mat(cameraPreviewHeight + cameraPreviewHeight / 2, cameraPreviewWidth, CvType.CV_8UC1);
// Load current frame
yuvMat.put(0, 0, data);
// Convert the frame to gray for better processing
Imgproc.cvtColor(yuvMat, currentFrame, Imgproc.COLOR_YUV420sp2RGB);
Imgproc.cvtColor(currentFrame, currentFrame, Imgproc.COLOR_BGR2GRAY);
From here, the byte array creation:
// Initialize grayscale byte array
byte[] grayscaleBytes = new byte[data.length];
// Extract grayscale data
currentFrame.get(0, 0, grayscaleBytes);
// Allocate ByteBuffer
ByteBuffer buffer = ByteBuffer.allocateDirect(grayscaleBytes.length);
// Wrap grayscale byte array
buffer.wrap(grayscaleBytes);
// Create frame
// rotation is calculated before
Frame currentGoogleFrame = new Frame.Builder().setImageData(buffer, currentFrame.cols(), currentFrame.rows(), ImageFormat.NV21).setRotation(rotation).build();
Constructing frames this way results in no faces found. However, using bitmaps it works as expected:
if(bitmap == null) {
// Bitmap allocation
bitmap = Bitmap.createBitmap(currentFrame.cols(), currentFrame.rows(), Bitmap.Config.ARGB_8888);
}
// Copy grayscale contents
org.opencv.android.Utils.matToBitmap(currentFrame, bitmap);
// Scale down to improve performance
Matrix scaleMatrix = new Matrix();
scaleMatrix.postScale(scaleFactor, scaleFactor);
// Recycle before creating scaleBitmap
if(scaledBitmap != null) {
scaledBitmap.recycle();
}
// Generate scaled bitmap
scaledBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight(), rotationMatrix, true);
// Create frame
// The same rotation as before is still used
if(scaledBitmap != null) {
Frame currentGoogleFrame = new Frame.Builder().setBitmap(scaledBitmap).setRotation(rotation).build();
}
Having detection take 2-3 seconds isn't typical. Using CameraSource isn't necessary to get the best performance What hardware are you using? Can you provide more specifics?
Some aspects of face detection are speed vs. accuracy trade-offs.
Speed:
Try using lower resolution images, if possible. Face detection should work fine at 640x480, for example. The face detector code does downsample large images before running detection, although this take additional time in comparison to receiving a lower resolution original.
Using ByteBuffers rather than Bitmaps will be a bit faster. The first portion of this should be just a grayscale image (no color info).
As you noted above, disabling landmarks and classification will make it faster.
In a future release, there will be a "min face size" option. Setting the min size higher makes the face detection faster (at the accuracy trade-off of not detecting smaller faces).
Setting the mode to "fast" will make it faster (at the accuracy trade-off of not detecting non-frontal faces).
Using the "prominent face only" option will be faster, but it only detects a single large face (at least 35% the width of the image).
Accuracy:
Enabling landmarks will allow the pose angles to be computed more accurately.
Setting the mode to "accurate" will detect faces at a wider range of angles (e.g., faces in profile). However, this takes more time.
Lacking the "min face size" option mentioned above, only faces larger than 10% the width of the image are detected by default. Smaller faces will not be detected. Changing this setting in the future will help to detect smaller faces. However, note that detecting smaller faces takes longer.
Using a higher resolution image will be more accurate than a lower resolution image. For example, some faces in a 320x240 image might be missed that would have been detected if the image were 640x480. The lower the "min face size" you set, the higher the resolution you need to detect faces of that size.
Make sure that you have the rotation right. The face won't be detected if it is upside down, for example. You should call the face detector again with a rotated image if you want to detect upside down faces.
Also, garbage collection time can be a factor if you're creating a lot of Bitmaps. An advantage of using ByteBuffer is that you can reuse the same buffer repeatedly without incurring per-image GC overhead that you would have encountered if you had used a Bitmap per image. CameraSource has this advantage, since it uses only a few buffers.
Related
I am using Otsu, which is a global thresholding technique, using OpenCV in java.
But I read that local thresholding techniques (e.g. Local Otsu, Sauvola, Niblack, etc...) are more effective in leaving out text from images (I am preprocessing images for OCR).
This is what I am doing:
Mat src = Imgcodecs.imread(imageFilePath, Imgcodecs.IMREAD_GRAYSCALE);
//Creating an empty matrices to store the destination image.
Mat dst = new Mat(src.rows(), src.cols(), src.type());
//Applying simple threshold
Imgproc.threshold(src, dst, 50, 255, Imgproc.THRESH_OTSU);
Bitmap bitmap = BitmapFactory.decodeFile(imageFilePath);
Utils.matToBitmap(dst, bitmap);
Should I use otsu? If so, how can be applied locally?
Or should I use another binarization algorithm?
A global method cannot work everywhere on an image if the illumination is uneven, or if the background color changes.
A local method copes with that but requires a scale parameter, which depends on the size of the features to detect. (This is a problem for general-purpose solutions where this size is unknown.)
Applying the filter in a sliding window is best because it results in a continuous threshold function, but is more time-consuming than with tiled windows.
I am trying to write a SWT component, that is able to take and draw an instance of java.awt.BufferedImage. My problem is that SWT's Image and AWT's BufferedImage are incompatible: SWT components can't draw java.awt.Image, AWT/Swing components can't draw org.eclipse.swt.graphics.Image.
There are several approaches that try to solve this problem in other ways (which also may have some variations, but basically there are these two):
Convert between SWT Image and AWT BufferedImage
Swing/SWT Integration
They all have shortcomings and didn't satisfy my expectations:
The first approach, to convert an SWT Image to a BufferedImage, results in poor performance for large images due to the creation of a new RGB instance for every Pixel.
The second approach has several shortcomings in usability. See the "workarounds" at the end of the linked article.
This lead to the conclusion that I'd try my best to write a component (based on org.eclipse.swt.widgets.Canvas or org.eclipse.swt.widgets.Composite) which allows to draw a BufferedImage directly without any conversion of images.
My approach was to draw it pixel by pixel. Therefore I simply had to get an instance of GC, walk the source BufferedImage line by line, left-to-right and drawing the corresponding Color using GC.setForeground(Color color) and GC.drawPoint(int x, int y).
First, I created a new instance of Color for every pixel, which uses quite a lot of memory and adds an additional delay, since new Color acquires system resources and creating a new object for every pixel also takes its time.
Then I tried to pre-load all possible (24 bit) Colors into an array before drawing the image. This lead to an explosion of memory usage (>= 600 MB), which was clear before I was trying it, but I had to verify it.
Caching only the used Colors also lead to more memory consumption than would have been required.
I think there has to be a more low-level approach that doesn't require that much memory, since SWT is able to draw whole (SWT) Images without consuming that much memory.
I would appreciate any ideas or solutions.
I found out there's a way to "convert" an BufferedImage to an Image by using the original image's data buffer if it is 24 bit RGB. This is possible, since the image formats are compatible.
final BufferedImage original = ImageIO.read(new File("some-image.jpg");
final PaletteData palette =
new PaletteData(0x0000FF, 0x00FF00, 0xFF0000);
// the last argument contains the byte[] with the image data
final ImageData data = new ImageData(original.getWidth(), original.getHeight(),
24, palette, 4,
((DataBufferByte) original.getData().getDataBuffer()).getData());
final Image converted = new Image(getDevice(), data);
This way, one doesn't have to create thousands of new objects. This approach comes with the disadvantage that one needs to ensure that the original image is of type RGB 24 bit. Otherwise the image has to be converted to this format.
After that, an image can be drawn with the following code:
// get the GC of your component
gc.drawImage(image, 0, 0);
Probably other bit depths can be converted in a similar way, but this is all I need for the moment.
I am trying to rotate a bufferedImage of a missile turret so that it looks like it's following a target. Basically, I can do it easily with the AffineTransform/ affinetransform
my current code in a nutshell is:
public BufferedImage tower = null;
try
{
tower = ImageIO.read(SpriteSheet.class.getResource("/spriteSheet/testTower.png"));
}
catch(IOException e)
{
AffineTransform tx = AffineTransform.getRotateInstance(rotationRequired, locationX, locationY);
AffineTransformOp = op = new AffineTransformOp(tx, AffineTransformOp.TYPE_BILINEAR);
//then I draw it using
g.drawImage(op.filter(tower, null), towerLocationX, towerLocationY, null);
this works, but what I want to do is transform(rotate) the bufferedImage, then copy the newly rotated pixel data into a pixel array and then draw it onto the screen because I believe this is how most games draw rotating images as opposed to drawing a png directly to the screen.
But what do I know. How exactly do 2D games draw rotating images? Am I doing it correctly, or is there a better/ more memory efficient way of doing this?
There are a lot of ways to tackle image manipulation in 2D games. Before optimizing though, you should ask yourself if there's a real need for it to begin with. Moreover, memory optimization usually comes at the cost of CPU performance and vice verse.
If CPU time is the problem, a common approach is to keep an array of images already rotated to certain angles (precalculated).
If memory is the problem, keep a single image and calculate the rotated form each time it's displayed. An even more memory efficient yet CPU consuming approach, is to draw vector shapes rather than images. This also leads to better looking results than the interpolation of the smoothing algorithm used for images when transformed. Java supports SVG, and there are several good packages available (e.g. http://xmlgraphics.apache.org/batik/).
Finally, Java can be connected to graphic libraries in order to perform the rendering, thus improving performance. Such libraries (OpenGL, etc.) use the memory of the graphic cards to store images in order to improve CPU usage (http://jogamp.org/jogl/www/).
I am trying to run a sketch that is supposed to show images (png´s, between 100kb and 1,5mb in size, 55.4mb total) in a coverflow animation. it works with about 10 images, but using more I get a out of memory error. I am loading the images file names into an string array like so:
String[] names = {"00.jpg", "01.jpg", "02.jpg"};
and then they get loaded into the sketch like so:
covers = new Cover[names.length];
for (int i = 0; i < covers.length; i++ ) {
covers[i] = new Cover(names[i]);
}
initCovers();
covers class:
class Cover {
PImage img;
Cover( String name ) {
img = loadImage(name);
public void drawCover() {
beginShape();
textureMode(NORMALIZED);
texture(img);
vertex(-300, -300, 0, 0, 0);
vertex( 300, -300, 0, 1, 0);
vertex( 300, 300, 0, 1, 1);
vertex(-300, 300, 0, 0, 1);
endShape();
when I run the sketch, my ram (8gb) gets filled within seconds, and the sketch doesn´t even load, it just crashes. when I start the sketch with about 10 images, everything works fine ( bout 1,5gb of ram usage).
my question is: why is it using so much memory? is it normal? is there a way to make it run more memory efficient (e.g. freeup memory of images that are not currently displayed because we can only see about 3 images at once on screen).
EDIT: I think the problem is that in the cover class, each time it gets called a new PImage is created. could that be possible?
image size in memory: width * height * (color depth/8), so for my images (1575y1969, 24bit) that woul be 8,9mb. times 91 images: about 807mb of memory usage just for the images.
Now that I understand the use-case better, I recommend to change the entire approach.
The types of applications (e.g. seen above) that you are trying to emulate do not load the entire slew of images as soon as the app. is presented. Instead, they read the small group of images that they are going to present to the user first. As the user approaches the end of that group, the software flushes some of the images at the start of the sequence (if memory is tight) and loads some more at the end.
Try increasing the JVM heap space
java -Xmx1024m
(Yes, I know, 1gig is a 'little' excessive, but after some experimentation, this value can be trimmed down)
As #millimoose states, the images loaded by Java are uncompressed into memory when they are loaded, so even a small image of 100kb on disk can suddenly occupy mb's of RAM when uncompressed. It becomes even more complicated when you start dealing with the alpha channel as well.
The size of compressed images isn't a good guide to the memory requirements.
The size in pixels is better. For example a modern camera photo with 8 megapixel resolution requires at least 32mb of memory to represent. If you are manipulating images this size with swing, double or triple that, at least. It's easy to gobble up a lot of memory.
Also, Java's internal memory management isn't very good at dealing with chunks this size.
I would say try painting them to a component using a single temp Cover as a handle as opposed to having N-many objects hanging around? And if you need interactions with images after draw time just hold back some simple meta data about their draw positions and dimensions, then use click event x,y etc to look up a a given image to allow you to work with it.
I'm trying to load a background image for a game as well as some smaller images, placing them on a Canvas, and letting people interact with the smaller overlayed images (like move, rotate)
In order to maintain aspect ratio (e.g. 5:3) I tried loading in the images as a bitmap and resizing them myself. The idea was to do cropping/letter-boxing for the background according to the canvas's width/height, and maintain the correct ratio of size for the smaller images.
Bitmap originalBitmap = BitmapFactory.decodeResource(getResources(), resourceImg);
Bitmap resizedBitmap = Bitmap.createBitmap(bitmapOrg2, 0, 0, width, height, matrix, true);
In order to cater for tablets/phones i have a background PNG background image at 1600x1000 and 200kb.
However I am now struggling with out of memory issues due to the bitmap being 1600x1000x4byte=6.4 mb of ram and more when it tries to resize.
(I am using the emulator at the moment when these issues occur)
I decided to change it to use canvas.setBackgroundResource
SceneCanvas sceneCanvas = (SceneCanvas) findViewById(R.id.the_canvas);
sceneCanvas.setBackgroundResource(R.drawable.my_scene_1600x900);
This works well, except it fills the screen and does not maintain aspect ratio.
Is there a way to set the background maintaining aspect ratio? Or have I just gone down the wrong route completely and should use ImageViews and render to the canvas somehow to avoid OutOfMemory issues
Given that Java code is only allowed a heap size of around 20MB or so, you’re always going to have trouble with large bitmaps (unless you resort to native code in C/C++/etc).
One option is to use a BitmapFactory to load your image, and in the Options you can specify an inSampleSize to downsample the image as it’s being read in. This should avoid chewing up memory by trying to load the entire original image. There is even an inJustDecodeBounds option, so your code can check the dimensions of the image, instead of having them hard-wired into the code.
It seems that the memory limit on Android is somewhere between 16 - 24 MB memory (depending on device). This is regardless of whether the device has a lot more memory. Also, the memory used by Bitmaps is included in the limit, resulting in lang.OutOfMemoryError: bitmap size exceeds VM budget. After some searching, there are 3 options I could find:
Allocate memory from native code using the NDK (native development kit) and JNI
For images one can also use OpenGL textures, the texture memory is not counted towards the limit.
take advantage of certain bitmap options to reduce size; http://developer.android.com/reference/android/graphics/BitmapFactory.Options.html
To see how much memory your app has allocated one can use, android.os.Debug.getNativeHeapAllocatedSize().