We Keep Coding

Using onPreviewFrame To run ML models

So i'm using the legacy Camera API (as far as I can tell) to get previewFrame call backs to then run a few machine learning models I have. I have confirmed that the machine learning models work when given a bitmap decoded when I take a picture via the onPictureTaken callback. Right now in the samples below, I am just simply testing on ML Kit's barcode scanner as a base case, but my custom models seemed to work fine with the onPictureTaken callback as well.
From what i've gathered, using onPreviewFrame isn't necessarily the best way to do this, but for the sake of having a quick sample play-around (and learning experience) I decided to just go this route. Based on everything i've tried from others having solutions online, I can't seem to get anything to work properly. The below code returns null:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
// Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
// final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
// Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
Log.d("onPreviewFrame - width", String.valueOf(width));
Log.d("onPreviewFrame - height", String.valueOf(height));
Log.d("onPreviewFrame - parameters.getPreviewFormat()", String.valueOf(parameters.getPreviewFormat()));
YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);
//
// byte[] bytes = out.toByteArray();
// final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
byte[] bytes = yuv.getYuvData();
final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}
Here's something else I tried:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
// Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
// final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
// Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
Log.d("onPreviewFrame - width", String.valueOf(width));
Log.d("onPreviewFrame - height", String.valueOf(height));
YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);
byte[] bytes = out.toByteArray();
final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}
Unfortunately I got this error:
ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.
with parameters.getPreviewFormat() returning 17 which is NV21. I also tried simply by changing that to ImageFormat.YUV_420_888 but that resulted in the below illegal argument exception:
only support ImageFormat.NV21 and ImageFormat.YUY2 for now

Instead of using the Camera API, try using CameraX. It's easy to use and you can execute your code whenever a frame is received from the camera. While trying to integrate an ML model with the camera, I faced a similar error and then turned to CameraX.
Basically, we'll create an ImageAnalysis.Analyser class through which we would get the Image object ( frames ). Using an extension function, we will convert this Image object to a YuvImage.
You can follow this codelab to use CameraX to analyze frames. You will create a class that extends ImageAnalysis.Analyser class.
class FrameAnalyser() : ImageAnalysis.Analyzer {
override fun analyze(image: ImageProxy?, rotationDegrees: Int) {
val yuvImage = image?.image?.toYuv() // The extension function
}
}
Create the extension function which transforms the Image to a YuvImage.
private fun Image.toYuv(): YuvImage {
val yBuffer = planes[0].buffer
val uBuffer = planes[1].buffer
val vBuffer = planes[2].buffer
val ySize = yBuffer.remaining()
val uSize = uBuffer.remaining()
val vSize = vBuffer.remaining()
val nv21 = ByteArray(ySize + uSize + vSize)
yBuffer.get(nv21, 0, ySize)
vBuffer.get(nv21, ySize, vSize)
uBuffer.get(nv21, ySize + vSize, uSize)
val yuvImage = YuvImage(nv21, ImageFormat.NV21, this.width, this.height, null)
return yuvImage
}
You can change the YUV Image format as required. Refer to these docs.

Insted of directly passing FirebaseVisionImage
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
you can do it like this
var bitmap = toARGBBitmap(ocrBitmap)
extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
private fun toARGBBitmap(img: Bitmap): Bitmap {
return img.copy(Bitmap.Config.ARGB_8888, true)
}
You can try this:)

¿How to manipulate an image from an array of bytes?

I tried this but when I show the bitmap I can not see anything, what I want is to paint a white outline on the original image
Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.dog);
public byte[] myImagetoBytes(Bitmap bitmap){
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
return stream.toByteArray();
}
byte[] bytes = myImagetoBytes();
bytes[0] = 1 & 0xff;
Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
this.imageView.setImageBitmap(bitmap);

Not sure what your code is supposed to do, but if you want to draw over a bitmap, you should use Canvas. You create it with its constructor:
Canvas canvas = new Canvas(bitmap);
And then you can use it like this:
Paint paint = new Paint();
paint.setColor(Color.GRAY);
RectF ovalRect = new RectF(0, 0, 10, 10);
canvas.drawOval(ovalRect, paint);

android renderscript ScriptIntrinsicColorMatrix setRGBtoYUV example usage

as the title says, how to convert bitmap RGB back to YUV byte[] using ScriptIntrinsicColorMatrix? below is the sample code (cannot be decoded by zxing):
public byte[] getYUVBytes(Bitmap src, boolean initOutAllocOnce){
if(!initOutAllocOnce){
outYUV = null;
}
if(outYUV == null){
outYUV = Allocation.createSized(rs, Element.U8(rs), src.getByteCount());
}
byte[] yuvData = new byte[src.getByteCount()];
Allocation in;
in = Allocation.createFromBitmap(rs, src,
Allocation.MipmapControl.MIPMAP_NONE,
Allocation.USAGE_SCRIPT);
scriptColor.setRGBtoYUV();
scriptColor.forEach(in, outYUV);
outYUV.copyTo(yuvData);
return yuvData;
}
one thing i notice is from original camera YUV byte is 3110400 bytes but after using the ScriptIntrinsicColorMatrix convertion in becomes 8294400, which i think is wrong.
Reason for YUV -> BW -> YUV is I want to convert the image into black and white (not grayscale) and back to YUV for zxing to decode at the same time show the black and white in surfaceview (like a custom camera filter).
tried below code but its a bit slow (can be decoded by zxing).
int[] intArray = null;
intArray = new int[bmp.getWidth() * bmp.getHeight()];
bmp.getPixels(intArray, 0, bmp.getWidth(), 0, 0, bmp.getWidth(), bmp.getHeight());
LuminanceSource source = new RGBLuminanceSource(cameraResolution.x, cameraResolution.y, intArray);
data = source.getMatrix();
any other alternative for RGB to YUV that is fast? if ScriptIntrinsicColorMatrix class cannot be done?
please and thank you

How to convert Mat to byte array, store the value and then convert it back again? java opencv [duplicate]

This question already has answers here:
OpenCV Mat object serialization in java
(4 answers)
Closed 5 years ago.
I have a Mat image for my system and I want to be able to store it in my sqlite database. So I am thinking I need to try convert it to a byte array to be able to store it. But then Im not sure that is right because I'm unsure how to use the value I get if I was to access it from the db to be able to turn it back into its original Mat image.
Below is what I have come up with so far:
static byte[] matToByte(Mat mat) throws SQLException {
int length = (int) (mat.total()*mat.elemSize());
byte buffer[] = new byte[length];
int converted = mat.get(0, 0, buffer);
IrisInitialDatabase.addFeatures(converted);
return buffer;
}
static Mat byteToMat(byte[] value) {
Mat m = new Mat();
m.put(0, 0, value);
return m;
}
thanks :)

Save it to the database in Base64 format.
Mat to bitmap
Bitmap image = Bitmap.createBitmap(rgba.cols(),
rgba.rows(), Bitmap.Config.RGB_565);
Utils.matToBitmap(rgba, image);
Bitmap bitmap = (Bitmap) image;
bitmap = Bitmap.createScaledBitmap(bitmap, 600, 450, false);
bitmap to byte array
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, byteArrayOutputStream);
byte[] byteArray = byteArrayOutputStream .toByteArray();
---save to database---
get mat back
m = Highgui.imdecode(new MatOfByte(Base64.decode(base64ImageFromDB,0)),Highgui.IMREAD_UNCHANGED);

Out of memory happened on showing sticker on imageview

a list of images and stickers(webp format) must be shown on a recycleview.
to show sticker on imageView, this [repository] (https://github.com/EverythingMe/webp-android) is used. this repository was one of suggested
solution on this post(WebP for Android)
sticker file is readed from external storage, convert to byte array, by using library of the repository, byte array convert to bitmap, and finally bitmap is shown on imageView. below code convert sticker file to bitmap
private void ShowStickerOnImageView(String stickerPath){
File file = new File(stickerPath);
int size = (int) file.length();
byte[] bytes = new byte[size];
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file));
buf.read(bytes, 0, bytes.length);
buf.close();
Bitmap bitmap = null;
boolean NATIVE_WEB_P_SUPPORT = Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN_MR2;
if (!NATIVE_WEB_P_SUPPORT) {
bitmap = WebPDecoder.getInstance().decodeWebP(bytes);
} else {
bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
}
holder.imageView.setImageBitmap(bitmap);
}
.....
public Bitmap decodeWebP(byte[] encoded, int w, int h) {
int[] width = new int[]{w};
int[] height = new int[]{h};
byte[] decoded = decodeRGBAnative(encoded, encoded.length, width, height);
if (decoded.length == 0) return null;
int[] pixels = new int[decoded.length / 4];
ByteBuffer.wrap(decoded).asIntBuffer().get(pixels);
return Bitmap.createBitmap(pixels, width[0], height[0], Bitmap.Config.ARGB_8888);
}
when 'NATIVE_WEB_P_SUPPORT' is false, 'decodeWebP' method is called, this method work fine in most of the time, but sometimes 'out of memory' error is happened on this method. most of the time, this error is happened on these lines
int[] pixels = new int[decoded.length / 4];
ByteBuffer.wrap(decoded).asIntBuffer().get(pixels);
return Bitmap.createBitmap(pixels, width[0], height[0], Bitmap.Config.ARGB_8888);
i found that byte array length of sticker file is big , can i decrease sticker file size programmatically? i want to find solution, to decrease byte array size.

You are creating a Bitmap that is being used as native size, but applied to an ImageView. Decrease the Bitmap to the size of the View:
Bitmap yourThumbnail= Bitmap.createScaledBitmap(
theOriginalBitmap,
desiredWidth,
desiredHeight,
false
);
Do note that:
public static Bitmap createBitmap(int colors[], int width, int height, Config config) {
return createBitmap(null, colors, 0, width, width, height, config);
}
Will call
public static Bitmap createBitmap(DisplayMetrics display, int colors[],
int offset, int stride, int width, int height, Config config)
And that will lead to:
Bitmap bm = nativeCreate(colors, offset, stride, width, height,
config.nativeInt, false);
Basically, you cannot create a huge Bitmap in memory, for no reason. If this is for phones, assume a 20 MB size for application.
An 800*600*4 image, yelds 1920000 bytes. Lower Image quality, such as using RGB_565 (half byte ammount per pixel, compared with RGB_8888), or pre-re-scale your source Bit map.