[edit] Reformatting into question and answer format following fadden# suggestion.
In ExtractMpegFramesTest_egl14.java.txt, method saveFrame(), there is a loop for reordering RGBA into ARGB for Bitmap png compression (see below quotes from that file), how can this be optimised?
// glReadPixels gives us a ByteBuffer filled with what is essentially big-endian RGBA
// data (i.e. a byte of red, followed by a byte of green...). We need an int[] filled
// with little-endian ARGB data to feed to Bitmap.
//
...
// So... we set the ByteBuffer to little-endian, which should turn the bulk IntBuffer
// get() into a straight memcpy on most Android devices. Our ints will hold ABGR data.
// Swapping B and R gives us ARGB. We need about 30ms for the bulk get(), and another
// 270ms for the color swap.
...
for (int i = 0; i < pixelCount; i++) {
int c = colors[i];
colors[i] = (c & 0xff00ff00) | ((c & 0x00ff0000) >> 16) | ((c & 0x000000ff) << 16);
}
It turns out there's an even faster approach.
Using the suggestion in #elmiguelao's answer, I modified the fragment shader to do the pixel swap. This allowed me to remove the swap code from saveFrame(). Since I no longer needed a temporary copy of the pixels in memory, I eliminated the int[] buffer entirely, switching from this:
int[] colors = [... copy from mPixelBuf, swap ...]
Bitmap.createBitmap(colors, mWidth, mHeight, Bitmap.Config.ARGB_8888);
to this:
Bitmap bmp = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
bmp.copyPixelsFromBuffer(mPixelBuf);
As soon as I did that, all of my colors were wrong.
It turns out that Bitmap#copyPixelsFromBuffer() wants the pixels in RGBA order, not ARGB order. The values coming out of glReadPixels() are already in the right format. So by doing it this way I avoid the swap, avoid an unnecessary copy, and don't need to tweak the fragment shader at all.
[edit] Reformatting into question and answer format following fadden# suggestion
I wanted to suggest that this conversion can happen in the FragmentShader by changing the line
gl_FragColor = texture2D(sTexture, vTextureCoord);
into
gl_FragColor = texture2D(sTexture, vTextureCoord).argb;
which is an efficient shortcut to reorder in GPU the shader's output channels, that works in other ways too: .abgr or even .bggr etc.
Related
I know this is an expensive operation and I already tried to use the robot.getPixelColor() function but it works slow, can only calculate like 5 times in a second.
What I'm saying is that 5 times is too small for what I actually want to do, but 20 should be enough. So I ask you if you can suggest me some optimisations to make to my actual code in order to get this result.
My code is:
while (true) {
color = robot.getPixelColor(x, y);
red = color.getRed();
green = color.getGreen();
blue = color.getBlue();
// do a few other operations in constant time
}
I don't know if this would help, but x and y don't change inside the while loop. So it's all the time the same pixel coordinates.
Thanks in advance!
EDIT: The pixel color will be taken from a game which will run at the same time with the java program, it will keep changing. The only thing is that are always the same coordinates.
I'm assuming the color is represented as a 32-bit int encoded as ARGB. In that case, instead of calling a method, you could just do bit masking to extract the colors, and that may end up being faster because you don't waste the overhead of calling a method. I'd recommend doing something like this:
int color = robot.getPixelColor(x,y);
int redBitMask = 0x00FF0000;
int greenBitMask = 0x0000FF00;
int blueBitMask = 0x000000FF;
int extractedRed = (color & redBitMask) >> 16;
int extractedGreen = (color & greenBitMask) >> 8;
int extractedBlue = (color & blueBitMask);
Bit shifting and bitwise operations tend to be very fast.
I can sucessfully read and write values to an image file which accurately shows an image created.
I simply read the values using getRGB(), and then bit shift them into red, green and blue arrays respectively. Then I simply set them back into another BufferedImage object using the setRGB() method.
Now, I am trying to alter the pixel values, say the very first pixel of the red array. I then print out the first 5 pixels of the red array and the first value is changed as expected before invoking the setRGB() method, but when I read in that image again the first value is now back to its original value?
Does anybody know that using the setRGB() only changes the values in memory, but does not actually write that altered values?
EDITS - THIS IS A SAMPLE REPRESENTATION OF MY CODE (this works perfectly due to getting back an image)
//READ IN IMAGE
BufferedImage imgBuf =null;
imgBuf = ImageIO.read(new File("test.jpg"));
int w = imgBuf.getWidth();
int h = imgBuf.getHeight();
int[] RGBarray = imgBuf.getRGB(0,0,w,h,null,0,w);
//BIT SHIFT VALUES INTO ARRAY
for(int row=0; row<h; row++)
{
for(int col=0; col<w; col++)
{
alphaPixels[row][col] = ((RGBarray[g]>>24)&0xff);
redPixels[row][col] = ((RGBarray[g]>>16)&0xff);
greenPixels[row][col] = ((RGBarray[g]>>8)&0xff);
bluePixels[row][col] = (RGBarray[g]&0xff);
g++;
}
}
//BIT SHIFT VALUES BACK TO INTEGERS
for(int row=0; row<h; row++)
{
for(int col=0; col<w; col++)
{
int rgb = (alphaPixels[row][col] & 0xff) << 24 | (redPixels[row][col] & 0xff) << 16 | (greenPixels[row][col] & 0xff) << 8 | (bluePixels[row][col] & 0xff);
imgBuf.setRGB(col, row, rgb);
}
}
//WRITE IMAGE BACK OUT
ImageIO.write(imgBuf, "jpeg", new File("new-test.jpg"));
Write where? If you change the RGB value of the BufferedImage's raster, then yes the memory value is written to and changed. If you mean does it change it to disk? No, not unless you write the image yourself to disk, often with ImageIO.write(...). Changes to the memory representation of disk data will not mathemagically change the disk representation on it's own; instead you have to explicitly do this with your code. I think that you may be missing this last important step.
Edit
You state in comment:
Currently I can write to an image created on disk with a new name. So if that works, then surely changing a few values should be the same effect? (Using setRBG() )
I'm still not clear on this. Say for instance:
If you have an image on disk, say imageA.jpg,
and say you read this into a BufferedImage via ImageIO.read(...), say into the bufferedImageA variable,
and then you change the data raster via setRGB(...)
and then write your changed BufferedImage to disk with ImageIO.write(...), say to a new file, imageB.jpg,
Then if you read in imageB.jpg, it should show the changes made.
But if you re-read in the unchanged imageA.jpg file, it will remain unchanged.
I have a method converting BufferedImages who's type is TYPE_CUSTOM to TYPE_INT_RGB. I am using the following code, however I would really like to find a faster way of doing this.
BufferedImage newImg = new BufferedImage(
src.getWidth(),
src.getHeight(),
BufferedImage.TYPE_INT_RGB);
ColorConvertOp op = new ColorConvertOp(null);
op.filter(src, newImg);
It works fine, however it's quite slow and I am wondering if there is a faster way to do this conversion.
ColorModel Before Conversion:
ColorModel: #pixelBits = 24 numComponents = 3 color space = java.awt.color.ICC_ColorSpace#1c92586f transparency = 1 has alpha = false isAlphaPre = false
ColorModel After Conversion:
DirectColorModel: rmask=ff0000 gmask=ff00 bmask=ff amask=0
Thanks!
Update:
Turns out working with the raw pixel data was the best way. Since the TYPE_CUSTOM was actually RGB converting it manually is simple and is about 95% faster than ColorConvertOp.
public static BufferedImage makeCompatible(BufferedImage img) throws IOException {
// Allocate the new image
BufferedImage dstImage = new BufferedImage(img.getWidth(), img.getHeight(), BufferedImage.TYPE_INT_RGB);
// Check if the ColorSpace is RGB and the TransferType is BYTE.
// Otherwise this fast method does not work as expected
ColorModel cm = img.getColorModel();
if ( cm.getColorSpace().getType() == ColorSpace.TYPE_RGB && img.getRaster().getTransferType() == DataBuffer.TYPE_BYTE ) {
//Allocate arrays
int len = img.getWidth()*img.getHeight();
byte[] src = new byte[len*3];
int[] dst = new int[len];
// Read the src image data into the array
img.getRaster().getDataElements(0, 0, img.getWidth(), img.getHeight(), src);
// Convert to INT_RGB
int j = 0;
for ( int i=0; i<len; i++ ) {
dst[i] = (((int)src[j++] & 0xFF) << 16) |
(((int)src[j++] & 0xFF) << 8) |
(((int)src[j++] & 0xFF));
}
// Set the dst image data
dstImage.getRaster().setDataElements(0, 0, img.getWidth(), img.getHeight(), dst);
return dstImage;
}
ColorConvertOp op = new ColorConvertOp(null);
op.filter(img, dstImage);
return dstImage;
}
BufferedImages are painfully slow. I got a solution but I'm not sure you will like it. The fastest way to process and convert buffered images is to extract the raw data array from inside the BufferedImage. You do that by calling buffImg.getRaster() and converting it into the specific raster. Then call raster.getDataStorage(). Once you have access to the raw data it is possible to write fast image processing code without all the abstraction in BufferedImages slowing it down. This technique also requires an in depth understanding of image formats and some reverse engineering on your part. This is the only way I have been able to get image processing code to run fast enough for my applications.
Example:
ByteInterleavedRaster srcRaster = (ByteInterleavedRaster)src.getRaster();
byte srcData[] = srcRaster.getDataStorage();
IntegerInterleavedRaster dstRaster = (IntegerInterleavedRaster)dst.getRaster();
int dstData[] = dstRaster.getDataStorage();
dstData[0] = srcData[0] << 16 | srcData[1] << 8 | srcData[2];
or something like that. Expect compiler errors warning you not to access low level rasters like that. The only place I have had issues with this technique is inside of applets where an access violation will occur.
I've found rendering using Graphics.drawImage() instead of ColorConvertOp 50 times faster. I can only assume that drawImage() is GPU accelerated.
i.e this is really slow, like 50ms a go for 100x200 rectangles
public void BufferdImage convert(BufferedImage input) {
BufferedImage output= new BufferedImage(input.getWidht(), input.getHeight(), BufferedImage.TYPE_BYTE_BINARY, CUSTOM_PALETTE);
ColorConvertOp op = new ColorConvertOp(input.getColorModel().getColorSpace(),
output.getColorModel().getColorSpace());
op.filter(input, output);
return output;
}
i.e however this registers < 1ms for same inputs
public void BufferdImage convert(BufferedImage input) {
BufferedImage output= new BufferedImage(input.getWidht(), input.getHeight(), BufferedImage.TYPE_BYTE_BINARY, CUSTOM_PALETTE);
Graphics graphics = output.getGraphics();
graphics.drawImage(input, 0, 0, null);
graphics.dispose();
return output;
}
Have you tried supplying any RenderingHints? No guarantees, but using
ColorConvertOp op = new ColorConvertOp(new RenderingHints(
RenderingHints.KEY_COLOR_RENDERING,
RenderingHints.VALUE_COLOR_RENDER_SPEED));
rather than the null in your code snippet might speed it up somewhat.
I suspect the problem might be that ColorConvertOp() works pixel-by-pixel (guaranteed to be "slow").
Q: Is it possible for you to use gc.createCompatibleImage()?
Q: Is your original bitmap true color, or does it use a colormap?
Q: Failing all else, would you be agreeable to writing a JNI interface? Either to your own, custom C code, or to an external library such as ImageMagick?
If you have JAI installed then you might try uninstalling it, if you can, or otherwise look for some way to disable codecLib when loading JPEG. In a past life I had similar issues (http://www.java.net/node/660804) and ColorConvertOp was the fastest at the time.
As I recall the fundamental problem is that Java2D is not at all optimized for TYPE_CUSTOM images in general. When you install JAI it comes with codecLib which has a decoder that returns TYPE_CUSTOM and gets used instead of the default. The JAI list may be able to provide more help, it's been several years.
maybe try this:
Bitmap source = Bitmap.create(width, height, RGB_565);//don't remember exactly...
Canvas c = new Canvas(source);
// then
c.draw(bitmap, 0, 0);
Then the source bitmap will be modified.
Later you can do:
onDraw(Canvas canvas){
canvas.draw(source, rectSrs,rectDestination, op);
}
if you can manage always reuse the bitmap so you be able to get better performance. As well you can use other canvas functions to draw your bitmap
I want to do a simple color to grayscale conversion using java.awt.image.BufferedImage. I'm a beginner in the field of image processing, so please forgive if I confused something.
My input image is an RGB 24-bit image (no alpha), I'd like to obtain a 8-bit grayscale BufferedImage on the output, which means I have a class like this (details omitted for clarity):
public class GrayscaleFilter {
private BufferedImage colorFrame;
private BufferedImage grayFrame =
new BufferedImage(width, height, BufferedImage.TYPE_BYTE_GRAY);
I've succesfully tried out 2 conversion methods until now, first being:
private BufferedImageOp grayscaleConv =
new ColorConvertOp(ColorSpace.getInstance(ColorSpace.CS_GRAY), null);
protected void filter() {
grayscaleConv.filter(colorFrame, grayFrame);
}
And the second being:
protected void filter() {
WritableRaster raster = grayFrame.getRaster();
for(int x = 0; x < raster.getWidth(); x++) {
for(int y = 0; y < raster.getHeight(); y++){
int argb = colorFrame.getRGB(x,y);
int r = (argb >> 16) & 0xff;
int g = (argb >> 8) & 0xff;
int b = (argb ) & 0xff;
int l = (int) (.299 * r + .587 * g + .114 * b);
raster.setSample(x, y, 0, l);
}
}
}
The first method works much faster but the image produced is very dark, which means I'm losing bandwidth which is unacceptable (there is some color conversion mapping used between grayscale and sRGB ColorModel called tosRGB8LUT which doesn't work well for me, as far as I can tell but I'm not sure, I just suppose those values are used). The second method works slower, but the effect is very nice.
Is there a method of combining those two, eg. using a custom indexed ColorSpace for ColorConvertOp? If yes, could you please give me an example?
Thanks in advance.
public BufferedImage getGrayScale(BufferedImage inputImage){
BufferedImage img = new BufferedImage(inputImage.getWidth(), inputImage.getHeight(), BufferedImage.TYPE_BYTE_GRAY);
Graphics g = img.getGraphics();
g.drawImage(inputImage, 0, 0, null);
g.dispose();
return img;
}
There's an example here which differs from your first example in one small aspect, the parameters to ColorConvertOp. Try this:
protected void filter() {
BufferedImageOp grayscaleConv =
new ColorConvertOp(colorFrame.getColorModel().getColorSpace(),
grayFrame.getColorModel().getColorSpace(), null);
grayscaleConv.filter(colorFrame, grayFrame);
}
Try modifying your second approach. Instead of working on a single pixel, retrieve an array of argb int values, convert that and set it back.
The second method is based on pixel's luminance therefore it obtains more favorable visual results. It could be sped a little bit by optimizing the expensive floating point arithmetic operation when calculate l using lookup array or hash table.
Here is a solution that has worked for me in some situations.
Take image height y, image width x, the image color depth m, and the integer bit size n. Only works if (2^m)/(x*y*2^n) >= 1.
Keep a n bit integer total for each color channel as you process the initial gray scale values. Divide each total by the (x*y) for the average value avr[channel] of each channel. Add (192 - avr[channel]) to each pixel for each channel.
Keep in mind that this approach probably won't have the same level of quality as standard luminance approaches, but if you're looking for a compromise between speed and quality, and don't want to deal with expensive floating point operations, it may work for you.
Anyone know of a simple way of converting the RGBint value returned from <BufferedImage> getRGB(i,j) into a greyscale value?
I was going to simply average the RGB values by breaking them up using this;
int alpha = (pixel >> 24) & 0xff;
int red = (pixel >> 16) & 0xff;
int green = (pixel >> 8) & 0xff;
int blue = (pixel) & 0xff;
and then average red,green,blue.
But i feel like for such a simple operation I must be missing something...
After a great answer to a different question, I should clear up what i want.
I want to take the RGB value returned from getRGB(i,j), and turn that into a white-value in the range 0-255 representing the "Darkness" of that pixel.
This can be accomplished by averaging and such but I am looking for an OTS implementation to save me a few lines.
This tutorial shows 3 ways to do it:
By changing ColorSpace
ColorSpace cs = ColorSpace.getInstance(ColorSpace.CS_GRAY);
ColorConvertOp op = new ColorConvertOp(cs, null);
BufferedImage image = op.filter(bufferedImage, null);
By drawing to a grayscale BufferedImage
BufferedImage image = new BufferedImage(width, height,
BufferedImage.TYPE_BYTE_GRAY);
Graphics g = image.getGraphics();
g.drawImage(colorImage, 0, 0, null);
g.dispose();
By using GrayFilter
ImageFilter filter = new GrayFilter(true, 50);
ImageProducer producer = new FilteredImageSource(colorImage.getSource(), filter);
Image image = this.createImage(producer);
this isn't as simple as it sounds because there is no 100% correct answer for how to map a colour to greyscale.
the method i'd use is to convert RGB to HSL then 0 the S portion (and optionally convert back to RGB) but this might not be exactly what you want. (it is equivalent to the average of the highest and lowest rgb value so a little different to the average of all 3)
Averaging sounds good, although Matlab rgb2gray uses weighted sum.
Check Matlab rgb2gray
UPDATE
I tried implementing Matlab method in Java, maybe i did it wrong, but the averaging gave better results.