I'm trying to use Google Mobile Vision API with the camera2 module and I'm having a lot of trouble.
I'm using Google's android-Camera2Video example code as a base. I've modified it to include the following callback:
Camera2VideoFragment.java
OnCameraImageAvailable mCameraImageCallback;
public interface OnCameraImageAvailable {
void onCameraImageAvailable(Image image);
}
ImageReader.OnImageAvailableListener mImageAvailable = new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
Image image = reader.acquireLatestImage();
if (image == null)
return;
mCameraImageCallback.onCameraImageAvailable(image);
image.close();
}
};
That way any fragment including Camera2VideoFragment.java can get access to its images.
Now, The Barcode API only accepts Bitmap images, but I'm unable to convert YUV_420_888 to Bitmap. Instead, I changed the imageReader's file format to JPEG and ran the following conversion code:
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
buffer.rewind();
byte[] data = new byte[buffer.capacity()];
buffer.get(data);
Bitmap bitmap = BitmapFactory.decodeByteArray(data, 0, data.length);
This worked but the framerate drop of feeding JPEG data to the imageReader was significant. I'm wondering if anyone has worked around this issue before.
A late answer but hopefully still helpful.
As Ezequiel Adrian on his Example has explained the conversion of YUV_420_888 into one of the supported formats (In his case NV21), you can do the similar thing to get your Bitmap output:
private byte[] convertYUV420888ToNV21(Image imgYUV420) {
// Converting YUV_420_888 data to YUV_420_SP (NV21).
byte[] data;
ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();
int buffer0_size = buffer0.remaining();
int buffer2_size = buffer2.remaining();
data = new byte[buffer0_size + buffer2_size];
buffer0.get(data, 0, buffer0_size);
buffer2.get(data, buffer0_size, buffer2_size);
return data;}
Then you can convert the result into Bitmap:
Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
Related
I am trying to use this new feature of CameraX Image Analysis (version 1.1.0-alpha08): using setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888), images sent to the analyzer will have RGBA format.
See this for reference: https://developer.android.com/reference/androidx/camera/core/ImageAnalysis#OUTPUT_IMAGE_FORMAT_RGBA_8888
I need to turn the image sent to the analyzer into a Bitmap so that I can input it to a TensorFlow classifier.
Without this new feature I would receive the image in the standard YUV_420_888 format then I would have to use one of the several solutions that can be googled in order to turn YUV_420_888 to RGBA then to Bitmap. Like this: https://blog.minhazav.dev/how-to-convert-yuv-420-sp-android.media.Image-to-Bitmap-or-jpeg/.
I assume getting the Media Image directly in RGBA format should help me avoid implementing those painfull solutions (that I have actually tried and do not seem to work very well for me so far).
Problem is I don't know how to turn this RGBA Media Image into a Bitmap. I have noticed that calling mediaImage.getFormat() returns 1 which is not an ImageFormat value but a PixelFormat one, the one logically corresponding to RGBA_8888 format, which is in line with the documentation: "All ImageProxy sent to ImageAnalysis.Analyzer.analyze(ImageProxy) will have format PixelFormat.RGBA_8888".
I have tried this:
private Bitmap toBitmapRGBA(Image image) {
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
buffer.rewind();
int size = buffer.remaining();
byte[] bytes = new byte[size];
buffer.get(bytes);
Bitmap bitmapImage = BitmapFactory.decodeByteArray(bytes, 0, bytes.length, null);
return bitmapImage;
}
This returns null indicating the decodeByteArray does not work. (I notice the image has got only one plane).
private Bitmap toBitmapRGBA2(Image image) {
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888);
buffer.rewind();
bitmap.copyPixelsFromBuffer(buffer);
return bitmap;
}
This returns a Bitmap that looks noting but noise.
Please help!
Kind regards
Mickael
I actually found a solution myself, so I post it here if anyone is interested:
private Bitmap toBitmap(Image image) {
Image.Plane[] planes = image.getPlanes();
ByteBuffer buffer = planes[0].getBuffer();
int pixelStride = planes[0].getPixelStride();
int rowStride = planes[0].getRowStride();
int rowPadding = rowStride - pixelStride * image.getWidth();
Bitmap bitmap = Bitmap.createBitmap(image.getWidth()+rowPadding/pixelStride,
image.getHeight(), Bitmap.Config.ARGB_8888);
bitmap.copyPixelsFromBuffer(buffer);
return bitmap;
}
if you want to process the pixel array further on without creating a bitmap object you can do something like this:
val data = imageProxy.planes[0].buffer.toByteArray()
val pixels = IntArray(data.size / imageProxy.planes[0].pixelStride) {
var index = it * imageProxy.planes[0].pixelStride
(data[index++].toInt() and 0xff.shl(16)) or
(data[index++].toInt() and 0xff).shl(8) or
(data[index++].toInt() and 0xff).shl(0) or
(data[index].toInt() and 0xff).shl(24)
}
And then you can create bitmap this way:
Bitmap.createBitmap(
pixels,
0,
imageProxy.planes[0].rowStride / imageProxy.planes[0].pixelStride,
imageProxy.width,
imageProxy.height,
Bitmap.Config.ARGB_8888
)
So i'm using the legacy Camera API (as far as I can tell) to get previewFrame call backs to then run a few machine learning models I have. I have confirmed that the machine learning models work when given a bitmap decoded when I take a picture via the onPictureTaken callback. Right now in the samples below, I am just simply testing on ML Kit's barcode scanner as a base case, but my custom models seemed to work fine with the onPictureTaken callback as well.
From what i've gathered, using onPreviewFrame isn't necessarily the best way to do this, but for the sake of having a quick sample play-around (and learning experience) I decided to just go this route. Based on everything i've tried from others having solutions online, I can't seem to get anything to work properly. The below code returns null:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
// Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
// final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
// Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
Log.d("onPreviewFrame - width", String.valueOf(width));
Log.d("onPreviewFrame - height", String.valueOf(height));
Log.d("onPreviewFrame - parameters.getPreviewFormat()", String.valueOf(parameters.getPreviewFormat()));
YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);
//
// byte[] bytes = out.toByteArray();
// final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
byte[] bytes = yuv.getYuvData();
final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}
Here's something else I tried:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
// Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
// final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
// Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
Log.d("onPreviewFrame - width", String.valueOf(width));
Log.d("onPreviewFrame - height", String.valueOf(height));
YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);
byte[] bytes = out.toByteArray();
final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}
Unfortunately I got this error:
ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.
with parameters.getPreviewFormat() returning 17 which is NV21. I also tried simply by changing that to ImageFormat.YUV_420_888 but that resulted in the below illegal argument exception:
only support ImageFormat.NV21 and ImageFormat.YUY2 for now
Instead of using the Camera API, try using CameraX. It's easy to use and you can execute your code whenever a frame is received from the camera. While trying to integrate an ML model with the camera, I faced a similar error and then turned to CameraX.
Basically, we'll create an ImageAnalysis.Analyser class through which we would get the Image object ( frames ). Using an extension function, we will convert this Image object to a YuvImage.
You can follow this codelab to use CameraX to analyze frames. You will create a class that extends ImageAnalysis.Analyser class.
class FrameAnalyser() : ImageAnalysis.Analyzer {
override fun analyze(image: ImageProxy?, rotationDegrees: Int) {
val yuvImage = image?.image?.toYuv() // The extension function
}
}
Create the extension function which transforms the Image to a YuvImage.
private fun Image.toYuv(): YuvImage {
val yBuffer = planes[0].buffer
val uBuffer = planes[1].buffer
val vBuffer = planes[2].buffer
val ySize = yBuffer.remaining()
val uSize = uBuffer.remaining()
val vSize = vBuffer.remaining()
val nv21 = ByteArray(ySize + uSize + vSize)
yBuffer.get(nv21, 0, ySize)
vBuffer.get(nv21, ySize, vSize)
uBuffer.get(nv21, ySize + vSize, uSize)
val yuvImage = YuvImage(nv21, ImageFormat.NV21, this.width, this.height, null)
return yuvImage
}
You can change the YUV Image format as required. Refer to these docs.
Insted of directly passing FirebaseVisionImage
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
you can do it like this
var bitmap = toARGBBitmap(ocrBitmap)
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
private fun toARGBBitmap(img: Bitmap): Bitmap {
return img.copy(Bitmap.Config.ARGB_8888, true)
}
You can try this:)
as the title says, how to convert bitmap RGB back to YUV byte[] using ScriptIntrinsicColorMatrix? below is the sample code (cannot be decoded by zxing):
public byte[] getYUVBytes(Bitmap src, boolean initOutAllocOnce){
if(!initOutAllocOnce){
outYUV = null;
}
if(outYUV == null){
outYUV = Allocation.createSized(rs, Element.U8(rs), src.getByteCount());
}
byte[] yuvData = new byte[src.getByteCount()];
Allocation in;
in = Allocation.createFromBitmap(rs, src,
Allocation.MipmapControl.MIPMAP_NONE,
Allocation.USAGE_SCRIPT);
scriptColor.setRGBtoYUV();
scriptColor.forEach(in, outYUV);
outYUV.copyTo(yuvData);
return yuvData;
}
one thing i notice is from original camera YUV byte is 3110400 bytes but after using the ScriptIntrinsicColorMatrix convertion in becomes 8294400, which i think is wrong.
Reason for YUV -> BW -> YUV is I want to convert the image into black and white (not grayscale) and back to YUV for zxing to decode at the same time show the black and white in surfaceview (like a custom camera filter).
tried below code but its a bit slow (can be decoded by zxing).
int[] intArray = null;
intArray = new int[bmp.getWidth() * bmp.getHeight()];
bmp.getPixels(intArray, 0, bmp.getWidth(), 0, 0, bmp.getWidth(), bmp.getHeight());
LuminanceSource source = new RGBLuminanceSource(cameraResolution.x, cameraResolution.y, intArray);
data = source.getMatrix();
any other alternative for RGB to YUV that is fast? if ScriptIntrinsicColorMatrix class cannot be done?
please and thank you
This question already has answers here:
OpenCV Mat object serialization in java
(4 answers)
Closed 5 years ago.
I have a Mat image for my system and I want to be able to store it in my sqlite database. So I am thinking I need to try convert it to a byte array to be able to store it. But then Im not sure that is right because I'm unsure how to use the value I get if I was to access it from the db to be able to turn it back into its original Mat image.
Below is what I have come up with so far:
static byte[] matToByte(Mat mat) throws SQLException {
int length = (int) (mat.total()*mat.elemSize());
byte buffer[] = new byte[length];
int converted = mat.get(0, 0, buffer);
IrisInitialDatabase.addFeatures(converted);
return buffer;
}
static Mat byteToMat(byte[] value) {
Mat m = new Mat();
m.put(0, 0, value);
return m;
}
thanks :)
Save it to the database in Base64 format.
Mat to bitmap
Bitmap image = Bitmap.createBitmap(rgba.cols(),
rgba.rows(), Bitmap.Config.RGB_565);
Utils.matToBitmap(rgba, image);
Bitmap bitmap = (Bitmap) image;
bitmap = Bitmap.createScaledBitmap(bitmap, 600, 450, false);
bitmap to byte array
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, byteArrayOutputStream);
byte[] byteArray = byteArrayOutputStream .toByteArray();
---save to database---
get mat back
m = Highgui.imdecode(new MatOfByte(Base64.decode(base64ImageFromDB,0)),Highgui.IMREAD_UNCHANGED);
Android's Google Maps api v2 provides a TileOverlay to which TileProviders can be added. A TileProvider will generate a Tile object given the lat, long, and depth. In order to make a tile, one must give it width (easy), height (easy), and an image represented as a byte array (confusing me). If I wanted to 'draw' a simple object and then turn it into a byte array, how would I do this?
For instance I am looking for something that looks like
Canvas canvas = new canvas();
...
canvas.drawRect(); //Or something like this (just an example)
...
byte[] bytes = canvas.SomeConversionFunctionOrProcessThatIDontKnow();
return new Tile(1,1,bytes);
public byte[] getByteArray (String image) throws IOException {
File yourImg = new File(image);
BufferedImage bufferedImage = ImageIO.read(yourImg);
WritableRaster wRaster = bufferedImage .getRaster();
DataBufferByte data = (DataBufferByte) wRaster.getDataBuffer();
return data.getData();
}
This should do the trick ; )
Tile#data should be compressed image data in one of the supported image formats. In other words, the raw contents of an image file. If you already have a decoded Bitmap, use Bitmap#compress(...) to write it to a ByteArrayOutputStream, then get the byte[] from that.
Bitmap bitmap = Bitmap.createBitmap(TILE_DIMENSION, TILE_DIMENSION,Bitmap.Config.ARGB_8888);
Canvas canvas = new Canvas(bitmap);
......
//draw something
......
Tile tile = convertBitmap(bitmap);
bitmap.recycle();
return tile;
}
private Tile convertBitmap(Bitmap bitmap){
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] bitmapData = stream.toByteArray();
return new Tile(TILE_DIMENSION, TILE_DIMENSION, bitmapData);
}