I have some C code that decodes a video frame by frame. I get to a point where i have an AVFrame in BGR32 and would like to send it back to Java for editing.
I have a ByteBuffer object in my C code that was created in Java using AllocateDirect but i struggle to write the content of the AVFrame->data[0] (of uint8_t type) to it and read it back. I have tried memcpy with no luck. Does anyone have an idea how to achieve this?
UPDATE
Followed Will's comment below and wrote this in C
char *buf = (*pEnv)->GetDirectBufferAddress(pEnv, byteBuffer);
memcpy(buf, rgb_frame->data[0], output_width*output_height*4);
The buffer does contain some data in Java but doing the following returns a null bitmap
BufferedImage frame = ImageIO.read(bitmapStream);
Where bitmapStream is a ByteBufferInputStream defined here:
https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/ByteBufferInputStream.java?r=205
Not sure if I am not writing things correctly in this buffer
UPDATE 2
Got pretty close now thanks to the latest snippet. I am using BGR32 format in my C code ie 4 bytes per pixel. So I modified things a bit in Java:
final byte[] dataArray = new byte[width*height*4];
imageData.get(dataArray);
final BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
final DataBuffer buffer = new DataBufferByte(dataArray, dataArray.length);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);
I get the image correctly but there seems to be an issue with color channels
Tried different formats with no luck
From Oracle's JNI Functions Documentation at
https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetDirectBufferAddress
GetDirectBufferAddress
void* GetDirectBufferAddress(JNIEnv* env, jobject buf);
Fetches and returns the starting address of the memory region
referenced by the given direct java.nio.Buffer.
This function allows native code to access the same memory region that
is accessible to Java code via the buffer object. LINKAGE:
Index 230 in the JNIEnv interface function table. PARAMETERS:
env: the JNIEnv interface pointer
buf: a direct java.nio.Buffer object (must not be NULL) RETURNS:
Returns the starting address of the memory region referenced by the
buffer. Returns NULL if the memory region is undefined, if the given
object is not a direct java.nio.Buffer, or if JNI access to direct
buffers is not supported by this virtual machine. SINCE:
JDK/JRE 1.4
I tested with this C++ code:
static char framebuf[100];
JNIEXPORT void JNICALL Java_javaapplication45_UseByteBuffer_readBuf
(JNIEnv *env, jobject usebb, jobject bb) {
void *addr = env->GetDirectBufferAddress(bb);
framebuf[0] = 77;
memcpy(addr,framebuf,100);
}
and this Java Code:
public class UseByteBuffer {
public native void readBuf(ByteBuffer bb);
}
...
public static void main(String[] args) {
System.load("/home/shackle/NetBeansProjects/usebb/dist/Debug/GNU-Linux-x86/libusebb.so");
ByteBuffer bb = ByteBuffer.allocateDirect(100);
new UseByteBuffer().readBuf(bb);
byte first_byte = bb.get(0);
System.out.println("first_byte = " + first_byte);
}
And it printed the first_byte=77 indicating it got the data copied correctly.
Update
ImageIO.read() will not accept just any set of bytes it has to be in a format that one of the installed ImageReader's can recognize such as JPEG or PNG.
Here is an example getting raw for (3 byte r,g,b )bytes into an image
int width = 256;
int height = 256;
ByteBuffer bb = ByteBuffer.allocateDirect(height*width*3);
byte[] raw = new byte[width * height * 3];
bb.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(DataBuffer.TYPE_BYTE, width, height, 3, width * 3, new int[]{0,1,2});
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);
Update 2
For BGR32 I believe this would be closer:
ByteBuffer imageData = ByteBuffer.allocateDirect(height * width * 4);
byte[] raw = new byte[width * height * 4];
imageData.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(
DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
new int[]{2,1,0} // Try {1,2,3}, {3,2,1}, {0,1,2}
);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);
Notice where I have commented, where I suspect you may need to experiment with the array of bandOffsets in the third argument of the ComponentSampleModel constructor to fix the color model.
Update 3
One can reuse the sampleModel to get data out of the image by using BufferedImage.copyData() to a WritableRaster instead of using getRaster().
SampleModel sampleModel = new ComponentSampleModel(
DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
new int[]{2, 1, 0}
);
...
BufferedImage newImage = ImageIO.read(new File("test.png"));
byte newRaw[] = new byte[height*width*4];
DataBuffer newBuffer = new DataBufferByte(newRaw, newRaw.length);
WritableRaster newRaster = Raster.createWritableRaster(sampleModel, newBuffer, null);
newImage.copyData(newRaster);
Related
Some operations on BufferedImages with 16 bit per channel result in images with random colored pixels. Is it possible to avoid this problem?
I see the problem at least with
ConvolveOp
AffineTransformOp with INTERPOLATION_BICUBIC on images with alpha channel
Sample code:
Kernel kernel = new Kernel(2, 2, new float[] { 0.25f, 0.25f, 0.25f, 0.25f });
ConvolveOp blurOp = new ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null);
img = blurOp.filter(img, null);
Input: output image:
The operations work fine when the image is 8 bit per channel.
I tried to convert the image from 16 to 8 bit per channel while keeping the color profile using the following code but this also results in a garbled image.
private static BufferedImage changeTo8BitDepth(BufferedImage bi) {
ColorModel cm = bi.getColorModel();
boolean hasAlpha = cm.hasAlpha();
boolean isAlphaPre = cm.isAlphaPremultiplied();
int transferType = DataBuffer.TYPE_BYTE;
int transparency = cm.getTransparency();
ColorSpace cs = cm.getColorSpace();
ColorModel newCm = new ComponentColorModel(cs, hasAlpha, isAlphaPre, transparency, transferType);
WritableRaster newRaster = newCm.createCompatibleWritableRaster(bi.getWidth(), bi.getHeight());
BufferedImage newBi = new BufferedImage(newCm, newRaster, isAlphaPre, null);
// convert using setData
newBi.setData(bi.getRaster());
return newBi;
}
(It is possible to use ColorConvertOp to convert to an 8-bit sRGB image but I need the non-sRGB color profile.)
I tested on Java 8, 11, and 17 on macOS and Linux. For full source code and images for tests see https://github.com/robcast/java-imaging-test (class Test16BitColor)
After som testing and research, I think the fact that ConvolveOp and AffineTransformOp doesn't work with 16 bits/sample (TYPE_USHORT data type) images out of the box, is a JDK bug. It might be that the underlying native code only works with 8 bits/sample images, but in that case "Op"s should throw an exception (or perhaps add a slower, but correct Java fallback code path). You might want to report that to the OpenJDK community.
For the 16 to 8 bits/sample conversion, the problem is you can't set 16 bit values into an 8 bit buffer, as there's no normalization done on the samples. I guess you'll just end up with the lower 8 bits of the 16 bits sample, which will typically look like static/noise. This can be fixed, however.
Here's a version that will convert the values correctly to 8 bit, but otherwise keep the color space/color profile unchanged:
private static BufferedImage changeTo8BitDepth(BufferedImage original) {
ColorModel cm = original.getColorModel();
// Create 8 bit color model
ColorModel newCM = new ComponentColorModel(cm.getColorSpace(), cm.hasAlpha(), cm.isAlphaPremultiplied(), cm.getTransparency(), DataBuffer.TYPE_BYTE);
WritableRaster newRaster = newCM.createCompatibleWritableRaster(original.getWidth(), original.getHeight());
BufferedImage newImage = new BufferedImage(newCM, newRaster, newCM.isAlphaPremultiplied(), null);
// convert using createGraphics/dawImage
Graphics2D graphics = newImage.createGraphics();
try {
graphics.drawImage(original, 0, 0, null);
}
finally {
graphics.dispose();
}
return newImage;
}
If you prefer conversion using rasters only, it's also possible with some hacks:
private static BufferedImage changeTo8BitDepth(BufferedImage original) {
ColorModel cm = original.getColorModel();
// Create 8 bit color model
ColorModel newCM = new ComponentColorModel(cm.getColorSpace(), cm.hasAlpha(), cm.isAlphaPremultiplied(), cm.getTransparency(), DataBuffer.TYPE_BYTE);
WritableRaster newRaster = newCM.createCompatibleWritableRaster(original.getWidth(), original.getHeight());
BufferedImage newImage = new BufferedImage(newCM, newRaster, newCM.isAlphaPremultiplied(), null);
// convert using setData
// newImage.setData(as8BitRaster(original.getRaster())); // Works
newRaster.setDataElements(0, 0, as8BitRaster(original.getRaster())); // Faster, requires less conversion
return newImage;
}
private static Raster as8BitRaster(WritableRaster raster) {
// Assumption: Raster is TYPE_USHORT (16 bit) and has PixelInterleavedSampleModel
PixelInterleavedSampleModel sampleModel = (PixelInterleavedSampleModel) raster.getSampleModel();
// We'll create a custom data buffer, that delegates to the original 16 bit buffer
final DataBuffer buffer = raster.getDataBuffer();
return Raster.createInterleavedRaster(new DataBuffer(DataBuffer.TYPE_BYTE, buffer.getSize()) {
#Override public int getElem(int bank, int i) {
return buffer.getElem(bank, i) >>> 8; // We only need the upper 8 bits of the 16 bit sample
}
#Override public void setElem(int bank, int i, int val) {
throw new UnsupportedOperationException("Raster is read only!");
}
}, raster.getWidth(), raster.getHeight(), sampleModel.getScanlineStride(), sampleModel.getPixelStride(), sampleModel.getBandOffsets(), new Point());
}
Hello I have a problem with converting my 4-bit data buffer to WritableRaster.
Image resolution: 1024x768 (786432)
Here is description what I'm doing.
1) Create 4-bit BufferedImage
bit4Image = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_BINARY, MY_BIT_4_COLOR_MODEL);
Graphics graphics = bit4Image.getGraphics();
graphics.drawImage(originalImage, 0, 0, null);
graphics.dispose();
//4-bit BufferedImage created. 4-bit BufferedImage is properly made cause it can be saved to hdd and looks good
2) Get byte array from DataBuffer from 4-bit
byte[] pixelData = ((DataBufferByte) bit4Image.getRaster().getDataBuffer()).getData();
// pixelData length is 393216
3) Now I want to create BufferedImage from this byte array pixelData
BufferedImage dest = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_BINARY, MY_BIT_4_COLOR_MODEL);
DataBufferByte buffer = new DataBufferByte(pixelData, pixelData.length);
WritableRaster raster = Raster.createInterleavedRaster(buffer, width, height, width, 1, new int[]{0}, new Point(0, 0));
dest.setData(raster);
Problem is when I call Raster.createInterleavedRaster. Exception:
java.awt.image.RasterFormatException: Data array too small (should be > 786431 )
I also tried something like this
BufferedImage dest = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_BINARY, MY_BIT_4_COLOR_MODEL);
dest.getRaster().setDataElements(0, 0, width, height, pixelData);
But this one gives me similar failure:
java.lang.ArrayIndexOutOfBoundsException: 393216
Could someone give me a hint or show the proper way of setting this 4-bit pixelData to WritableRaster?
Resolved.
I just had to create WritableRaster with giving it SampleModel:
SampleModel sm = MY_BIT_4_COLOR_MODEL.createCompatibleSampleModel(width, height);
WritableRaster wr = Raster.createWritableRaster(sm, buffer, new Point(0,0));
The initial code you had almost worked, the only problem is you tried to create an "interleaved" raster. For palette images (IndexColorModel) you typically have only one sample (the palette index) per pixel, so there's no samples to interleave.
Instead, your pixel data is 4 bit/pixel, stored as two pixels per byte. Storing more samples per storage unit, is often referred to as "packed". This means you need to create a "packed" raster, using one of the Raster.createPackedRaster methods.
Here's a full, runnable sample:
public static void main(String[] args) {
int width = 100;
int height = 100;
// Create initial 4 bit image
IndexColorModel icm = new IndexColorModel(4, 16, new int[16], 0, false, -1, DataBuffer.TYPE_BYTE);
BufferedImage bit4Image = new BufferedImage(width, height, BufferedImage.TYPE_BYTE_BINARY, icm);
// ...you probably do something in between here
// Get the pixel data
byte[] pixelData = ((DataBufferByte) bit4Image.getRaster().getDataBuffer()).getData();
// ...you probably do something in between here
// Create a data buffer around the array, wrap in raster
DataBufferByte buffer = new DataBufferByte(pixelData, pixelData.length);
WritableRaster raster = Raster.createPackedRaster(buffer, width, height, 4, null);
// Finally create a copy of the image, sharing pixel data
BufferedImage copy = new BufferedImage(icm, raster, icm.isAlphaPremultiplied(), null);
System.out.println("copy: " + copy);
}
What I'm trying to do is create a BufferedImage from a byte array. Here is what I'm doing now:
try {
ByteArrayInputStream in = new ByteArrayInputStream(bytearray);
BufferedImage bImageFromConvert = ImageIO.read(in);
Color color = new Color(bImageFromConvert.getRGB((int) local_car.x, (int) local_car.z));
System.out.println("R :: "+color.getRed() + " B :: "+color.getBlue() + " G :: "+color.getGreen());
} catch (IOException e) {
e.printStackTrace();
}
}
The documentation of ImageIO.read says:
Returns a BufferedImage as the result of decoding a supplied URL with an ImageReader chosen automatically from among those currently registered. An InputStream is obtained from the URL, which is wrapped in an ImageInputStream. If no registered ImageReader claims to be able to read the resulting stream, null is returned.
I'm receiving a null pointer exception from ImageIO.read() returning null. I am sending my bytearray in the form of RGBA. Why is ImageIO.read returning null?
The ImageIO functions are for reading files and expect the input stream to be in one of the file formats such as PNG or JPG, not for reading simple arrays of rgba. To read in a simple array try something like:
int width = 256;
int height = 256;
final int bytes_per_pixel = 4;
byte[] raw = new byte[width * height * bytes_per_pixel];
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
IntBuffer intBuf
= ByteBuffer.wrap(raw)
.order(ByteOrder.LITTLE_ENDIAN)
.asIntBuffer();
int[] array = new int[intBuf.remaining()];
intBuf.get(array);
image.setRGB(0, 0, width, height, array, 0, width);
How to convert mat(OpenCV) to image(JavaFX)?
I think this isn't not best method:
MatOfByte byteMat = new MatOfByte();
Highgui.imencode(".bmp", mat, byteMat);
return new Image(new ByteArrayInputStream(byteMat.toArray()));
P.S.
Image - import javafx.scene.image.Image;
One way to do it would be this.I do not remember the source from where I got this:-
private Image mat2Image(Mat frame)
{
int type = BufferedImage.TYPE_BYTE_GRAY;
if ( frame.channels() > 1 ) {
type = BufferedImage.TYPE_3BYTE_BGR;
}
int bufferSize = frame.channels()*frame.cols()*frame.rows();
byte [] b = new byte[bufferSize];
frame.get(0,0,b); // get all the pixels
BufferedImage image = new BufferedImage(frame.cols(),frame.rows(), type);
final byte[] targetPixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
System.arraycopy(b, 0, targetPixels, 0, b.length);
return SwingFXUtils.toFXImage(image,null);
}
There may be a way to make it neater using the Converters class from Opencv along with JDK8. I will update this if I find any such thing.
Paritosh, the issue with your method is that it only applies to the Mats with the type of CvType.CV_8U or CvType.CV_8S, since those Mats can be contained in a byte array. If the type of the Mat is, lets say, CvType.CV_32F, you would need a float array to hold the data. The float array cannot be System.arraycopied into a byte[] targetPixels array.
I have very little experience with Java IO and images, and I've been unsuccessful at converting an aerial image saved as a byte array into a BufferedImage.
Here's my code:
int width = scaledImage.getWidth();
int height = scaledImage.getHeight();
DataBuffer buffer = new DataBufferByte(scaledImage.getImage(), scaledImage.getImage().length, 0);
SampleModel sampleModel = new SinglePixelPackedSampleModel(DataBuffer.TYPE_BYTE, width, height, new int[]{(byte)0xf});
WritableRaster raster = Raster.createWritableRaster(sampleModel, buffer, null);
ColorModel colorModel = imageManager.generateColorModel();
BufferedImage image = new BufferedImage(colorModel, raster, false, null);
Most of this code is borrowed from http://www.exampledepot.com/egs/java.awt.image/Mandelbrot2.html.
This code throws the following exception
java.awt.image.RasterFormatException: Data array too small (should be 122499 )
the actual length of the data array is 52341.
The dimensions are 350px X 350px
Here is the line that is killing you:
DataBuffer buffer = new DataBufferByte(scaledImage.getImage(), scaledImage.getImage().length, 0);
The example you show does width * height instead of scaledImage.getImage().length. In the model you've chosen, you need a byte per pixel, which is 350x350 or 122500.