Recently I have been given a project, where I have to extract face (face+hair) from a given image.
I am solving this problem in the following ways.
I am extracting face locations from given image. [I am getting a rectangle]
I am extracting that rectangle and placing it in another image of same dimensions as input image.[face_image]
I am applying grabCut algorithm on the face_image of step 2.
When the face_image contains smooth background then the algorithm grabCut it working well but when the background of face_image is complex then the algorithm grabCut extracts some part of background too in the processed image.
Here is a snapshot of the results that I am getting.
Here is my code of grabCut:
public void extractFace(Mat image, String fileNameWithCompletePath,
int xOne, int xTwo, int yOne, int yTwo) throws CvException {
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Rect rectangle = new Rect(xOne, yOne, xTwo, yTwo);
Mat result = new Mat();
Mat bgdModel = new Mat();
Mat fgdModel = new Mat();
Mat source = new Mat(1, 1, CvType.CV_8U, new Scalar(3));
Imgproc.grabCut(image, result, rectangle, bgdModel, fgdModel, 8, Imgproc.GC_INIT_WITH_RECT);
Core.compare(result, source, result, Core.CMP_EQ);
Mat foreground = new Mat(image.size(), CvType.CV_8UC3, new Scalar(255, 255, 255));
image.copyTo(foreground, result);
Imgcodecs.imwrite(fileNameWithCompletePath, foreground);
}
How can I improve performance of grabCut algorithm so that it will detect only face and hair from given image?
You should be able to do this by "helping" grabCut know a little about the foreground and background. There is a python tutorial that shows how this is done manually by selecting the foreground and background.
To do this automatically, you will need to find programmatic ways to detect the foreground and background. The foreground consists mostly of hair and skin so you will need to detect them.
Skin - There are several papers and blogs on how to do this. Some of them are pretty simple and this OpenCV tutorial may also help. I've found plain hue/saturation to get me pretty far.
Hair - This is trickier but is definitely still doable. You may be able to hair and just use skin and background if this turns out to be too much work.
Background - You should be able to use range() to find things in the image that are purple, green, and blue. You know for sure that these things are not skin or hair and are therefore part of the background.
Use thresholding to create a mask of the areas that are most likely skin, hair, and background. You can then use them as bgdModel and fgdModel (or the skin and hair masks) instead of Mat().
Sorry this is so high-level. I hope it helps!
Another approach, since you have already detected the face, is to simply choose a better initial mask for initialising GrabCut - e.g. by using an oval instead of a rectangle.
Detect face rectangle (as you are already doing)
Create a mask:
a) Create a new black image of the same size as your input image
b) Draw a white-filled ellipse with the same height, width, top and left positions as the face rectangle
Call GrabCut with GC_INIT_WITH_MASK instead of GC_INIT_WITH_RECT:
Imgproc.grabCut(image, mask, rectangle, bgdModel, fgdModel, 8, Imgproc.GC_INIT_WITH_MASK);
This initializes the foreground with a better model because faces are more oval-shaped than rectangle-shaped, so it should include less of the background to begin with.
I would suggest to "play" with the rectangle coordinates (int xOne, int xTwo, int yOne, int yTwo). Using your code and these coordinates 1, 400, 30, 400 I was able to avoid the background. (I tried to post the images I successfully cropped but I need at least 10 reputation to do so)
The best optimization that can be done to any Java routine is conversion to a native language.
Related
I'm developing a little graphic engine using Canvas in JavaFX. In some point I had to render an off screen image, and then print it on my primary canvas using its GraphicContext.
I'm using this code right now:
private Canvas offScreenCanvas;
private GraphicsContext offScreenGraphic;
private SnapshotParameters parameters;
private WritableImage offScreenImage;
[...]
offScreenCanvas = new Canvas(WIDTH, HEIGHT);
offScreenGraphic = offScreenCanvas.getGraphicsContext2D();
parameters = new SnapshotParameters();
parameters.setFill(Color.TRANSPARENT);
[...]
offScreenImage = offScreenCanvas.snapshot(parameters, offScreenImage);
graphic.setGlobalBlendMode(BlendMode.HARD_LIGHT);
graphic.drawImage(offScreenImage, 0, 0);
graphic.setGlobalBlendMode(BlendMode.SRC_OVER);
My problem is the method snaphot() takes too much time, ~14ms, in each execution. I need to update the canvas at least at 60fps, so this consumes practically all the time I have to draw.
Is there another way to get an Image or WritableImage from a canvas? Maybe another different process?
This is another method to obtain a visual equivalent result, without reduce performance.
I have used java.awt clases, instead of JavaFX clases. The creation of a java.awt.image.BufferedImage offers the possibility to get a java.awt.Graphics2D where you can draw using other methods. The main problem here is that draw big images consumes a lot of time using this libraries, but you can create a scaled image. In my case, I have created a quarter-size BufferedImage, and I have drawn all the objects using that scale factor.
At the end of the draw process, just convert the BufferedImage, to a javafx.scene.image.Image, using:
SwingFXUtils.WritableImage toFXImage(BufferedImage bimg, WritableImage wimg);
Then print it on the main canvas using:
graphic.drawImage(Image image, 0, 0, WIDTH, HEIGHT);
To fill all the canvas with the image.
Finally, the scale factor is configurable, so if you need a detailed image, just use a higher value. For me, a 25-percent-size image is enough because I am drawing gradients. Now, it takes 1-3ms to draw the image, this is much better than before.
Hope it helps someone.
I have a quadrilateral drawn in Path2D, and I would like for there to be an image on it. More specifically, I am trying to draw an image of my choice to 4 different points on a quadrilateral. In my case, it is a parallelogram. I do not want the image to go over the paralellogram. A better way to see what I am trying to say is to see the screenshot below.
I would like the image to be transformed to fit the green area. Not clipped.
I want the image to be pinned over the green paralellogram. However. I do not want the image to go over into the blue paralellogram, or the white space foe that matter.
So far I have tried
Researching for a way to place images directly onto Path2D.Double() objects. No answer
Rotating the image to fit the paralellogram. Didnt work.
Using AffineTransform in java. Dont get it ;-;
Thanks. I am new to java so do try to be lenient?
One way is to:
create a separate BufferedImage.
Apply a transform to the new image.
Draw your image to that new image.
Use the Shape object for the green area as a clip on the main drawing area
Draw the transformed image onto the main drawing area.
It's been a while since I have done transformations. You may have to set the transformation first and then draw the image after. Transformation has to come first.
public void paintComponent(Graphics g) {
Graphics2D g2 = (Graphics2D)g;
g2.transform(AffineTransform.getShearInstance(1.0, 0));
g2.drawImage(image, 0, 0, this);
}
Here is a simple example of how transforms work. You will have to spend some time on figuring out what values you need to make it work or if you might need to manually create a transformation matrix yourself.
I'm using Java Graphics2D to generate this map with some sort of tinted red overlay over it. As you can see, the overlay gets cut off along the image boundary on the left side:-
After demo'ing this to my project stakeholders, what they want is for this overlay to clip along the map boundary with some consistent padding around it. The simple reason for this is to give users the idea that the overlay extends outside the map.
So, my initial thought was to perform a "zoom and shift", by creating another larger map that serves as a "cookie cutter", here's my simplified code:-
// polygon of the map
Polygon minnesotaPolygon = ...;
// convert polygon to area
Area minnesotaArea = new Area();
minnesotaArea.add(new Area(minnesotaPolygon));
// this represents the whole image
Area wholeImageArea = new Area(new Rectangle(mapWidth, mapHeight));
// zoom in by 8%
double zoom = 1.08;
// performing "zoom and shift"
Rectangle bound = minnesotaArea.getBounds();
AffineTransform affineTransform = new AffineTransform(g.getTransform());
affineTransform.translate(-((bound.getWidth() * zoom) - bound.getWidth()) / 2,
-((bound.getHeight() * zoom) - bound.getHeight()) / 2);
affineTransform.scale(zoom, zoom);
minnesotaArea.transform(affineTransform);
// using it as a cookie cutter
wholeImageArea.subtract(minnesotaArea);
g.setColor(Color.GREEN);
g.fill(wholeImageArea);
The reason I'm filling the outside part with green is to allow me to see if the cookie cutter is implemented properly. Here's the result:-
As you can see, "zoom and shift" doesn't work in this case. There is absolutely no padding at the bottom right. Then, I realized that this technique will not work for irregular shape, like the map... and it only works on simpler shapes like square, circle, etc.
What I want is to create consistent padding/margin around the map before clipping the rest off. To make sure you understand what I'm saying here, I photoshopped this image below (albeit, poorly done) to explain what I'm trying to accomplish here:-
I'm not sure how to proceed from here, and I hope you guys can give me some guidance on this.
Thanks.
I'll just explain the logic, as I don't have time to write the code myself. The short answer is that you should step through each pixel of the map image and if any pixels in the surrounding area (i.e. a certain distance away) are considered "land" then you register the current pixel as part of the padding area.
For the long answer, here are 9 steps to achieve your goal.
1. Decide on the size of the padding. Let's say 6 pixels.
2. Create an image of the map in monochrome (black is "water", white is "land"). Leave a margin of at least 6 pixels around the edge. This is the input image: (it isn't to scale)
3. Create an image of a circle which is 11 pixels in diameter (11 = 6*2-1). Again, black is empty/transparent, white is solid. This is the hit-area image:
4. Create a third picture which is all black (to start with). Make it the same size as the input image. It will be used as the output image.
5. Iterate each pixel of the input image.
6. At that pixel overlay the hit-area image (only do this virtually, via calculation), so that the center of the hit-area (the white circle) is over the current input image pixel.
7. Now iterate each pixel of the hit-area image.
8. If the any white pixel of the hit-area image intersects a white pixel of the input image then draw a white pixel (where the center of the circle is) into the output image.
9. Go to step 5.
Admittedly, from step 6 onward it isn't so simple, but it should be fairly easy to implement. Hopefully you understand the logic. If my explanation is too confusing (sorry) then I could spend some time and write the full solution (in Javascript, C# or Haskell).
I'm currently exploring opengl through the use of the JOGL library (the java wrappers for openGL) which I'm using to create 2d/3d graphs. At the minute I'm having a little issue with text I've rendered through the "glutBitmapString" method, it isn't resizing in respect of the window as shown in the screenshot below. Unfortunately the job spec I've been given is that this must be done in Java, so I can't jump to any other language that has a better supported version of openGL.
Everything else in the window resizes correctly so I'm assuming the issue is in the code I've posted below, if not then I'll be happy to post code you feel is relevant to the issue.
Here is a snippet of my code I'm using to render the text
GL gl = drawable.getGL();
GLUT glut = new GLUT();
float textPosx = -0.4f;
float textPosy = -2.1f;
gl.glColor3f(1.0f, 0.0f, 0.0f);
// Move to rastering position
gl.glRasterPos2f(textPosx, textPosy);
// convert text to bitmap and tell what string to put
glut.glutBitmapString(GLUT.BITMAP_HELVETICA_12, "0");
textPosx = 1.75f;
textPosy = -2.15f;
// Move to rastering position
gl.glRasterPos2f(textPosx, textPosy);
// convert text to bitmap and tell what string to put
glut.glutBitmapString(GLUT.BITMAP_HELVETICA_18, "TIME");
textPosx = -1.0f;
textPosy = 1.0f;
gl.glColor3f(0.0f, 1.0f, 0.0f);
// Move to rastering position
gl.glRasterPos2f(textPosx, textPosy);
// convert text to bitmap and tell what string to put
glut.glutBitmapString(GLUT.BITMAP_HELVETICA_18, "ERRORS");
glutBitmapString draws text in 2D. 2D text size is based on font size. So, if you set the font size to 18, as you have in this example, then it will be standard 18 pt font size on the screen, no matter how large you make the window or how close you zoom in. This is not a Java issue. Java is not actually drawing anything. Everything is being drawn by the OpenGL native libraries, which are written in C++, so it will be exactly the same in C++ as it is in Java.
There are two ways you could work around this. One would be to change the font size of the text as you zoom in or out. This would be kind of a awkward, and may be difficult to get right. A better option, imo, would be to simply use 3D text. In JOGL you use the TextRenderer object to draw 3D text.
In your init method create a global variable like so:
textr = new TextRenderer(new Font("SansSerif", Font.PLAIN, 18));
Obviously, change the font settings to whatever you prefer. Then in your display loop:
textr.setColor(Color.GREEN);
textr.begin3DRendering();
textr.draw3D("ERRORS", xLocation, yLocation, zLocation, scale);
textr.end3DRendering();
Personally, I prefer to use a large font size and then scale it down some, that way, when you zoom in, it doesn't get pixelated.
Also, unlike 2D text, 3D text will not always face the screen. You have to do that manually. It depends on how your camera is set up, but if you're using basic rotations to move the camera around, usually you can just negate those rotations on the 3D text object to make it face the camera.
For the x, y, and z locations, those are the locations within the current object (local coordinates). Think of the beginRendering() to endRendering() as one object with its own local coordinate system. Usually, I prefer to draw my text at 0, 0, 0 local coordinates, then move the entire object to the proper location. That way rotations are easier to understand.
You could try using TextRenderer. Works fine.
This is a really hard question to explain in words (well it is for me anyway). I need to be able to take an image (bitmap) and crop the image down to a certain size in the centre of the screen but keeping the size of the image the same. Hopefully the picture below can explain what I mean:
So the image as a whole is cropped down to the square in the middle but is not stretched across the screen and remains in the centre, so basically removing the pointless part of the image but keeping to co-ordinates of the pixels the same.
So let's say you have done your face detection, and have found one face in your image. Your image is 320 x 240, and the face is bound by the rectangle with location 100,40 and width 20 x 30. Now what would you like to do with that information? I'll do my best to help, but you'll probably need to clear up any poor assumptions on my part.
First, you can grab the face and store it into a new bitmap with something like Bitmap.createBitmap():
Bitmap face = Bitmap.createBitmap(largeSource, 100, 40, 20, 30);
This should be done outside of the draw loop, like in onCreate or some other initialization step.
It sounds like you've got some container (ImageView? Custom View with overridden onDraw?) which is housing your large image. And now you want to just draw the face in that container, at its original position? If you've got a custom view, that's as simple as the following in your onDraw:
canvas.drawBitmap(face, 100, 40, facePaint);
If you're using an ImageView instead, I'd suggest going to a custom-drawn view instead, since it sounds like you need some fine-grained drawing control.
Finally, if you've got a bunch of these faces, create a new FaceObj POJO object, which just has a bitmap, x, and y coordinate. As you detect faces, add them to an ArrayList, and then iterate over this in in your onDraw to draw all your faces:
faces.add(new FaceObj(Bitmap.createBitmap(largeSource, 100, 40, 20, 30), 100, 40);
...
foreach(FaceObj f : faces)
canvas.drawBitmap(f.bitmap, f.x, f.y, facePaint);
If I understand you don't really want to crop your image but "hide" any pixels around the square.
There are many ways to do this depending on what you are trying to do. For example you can fill the uninteresting part of the picture with black or make it transparent.
This way the coordinates of your "cropped" picture will remain the same on the screen.
If all you are interested in is the center of the image, then one really easy way to do this is to just add the android:scaleType="center" attribute to your ImageView, and set the ImageView to the crop size you want. If you're wanting it to be positionable, though, that's a different story.
Using Canvas.drawBitmap() you might be able to work to copy a part of the image to a different bitmap, and discard the other. With this particular version of the method, you can send in the array of colors you get with getPixels(), and set an offset, and the width and height that you want to copy. The stride parameter is important though, as it needs to be set to the width of the original image, even if your final image will be smaller, as it's pulling pixels from the original image.