I would like to detect a barcode within a captured image. I capture an image using android's camera2. Following this, the image's metadata is retrieved and the image is saved to the device. The metadata is all passed along to the next activity, which is where the application attempts to detect a barcode.
This next activity creates a byte[] from the File saved previously. Next, the relevant FirebaseVision objects are created using the data passed with the intent. Finally, the application attempts to call the detectInImage() method, where an error is thrown:
"java.lang.IllegalArgumentException: Invalid image data size."
I suspect this is from the captured image being too large, however I cannot seem to figure out how to capture a smaller image, and I also cannot find anything in the reference documentation regarding the maximum size allowed. Information regarding this error and how to solve it would be very much appreciated. Below is what I believe to be the relevant code.
private final ImageReader.OnImageAvailableListener onImageAvailableListener
= new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader imageReader) {
try{
// Semaphore ensures date is recorded before starting next activity
storeData.acquire();
Image resultImg = imageReader.acquireNextImage(); // Image from camera
imgWidth = resultImg.getWidth();
imgHeight = resultImg.getHeight();
ByteBuffer buffer = resultImg.getPlanes()[0].getBuffer();
data = new byte[buffer.remaining()]; // Byte array with the images data
buffer.get(data);
String timeStamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date());
// Note: mediaFile directs to Pictures/"ThisProject" folder
File media = new File(mediaFile.getPath() +
File.separator + "IMG_" + timeStamp + ".jpg");
// Saving the image
FileOutputStream fos = null;
try {
fos = new FileOutputStream(media);
fos.write(data);
uri = Uri.fromFile(media);
} catch (IOException e) {
Log.e(TAG, e.getMessage());
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
Log.e(TAG, e.getMessage());
}
}
}
resultImg.close();
} catch (InterruptedException e) {
Log.e(TAG, e.getMessage());
}
storeData.release();
}
};
This essentially retrieves the image height & width, then writes it to a file.
The data sent to the next activity consists of the: Image width, Image height, Image rotation, and the Uri directing to the file.
Using this, I try to detect a barcode using Firebase ML Kit:
// uri is the uri referencing the saved image
File f = new File(uri.getPath());
data = new byte[(int) f.length()];
try{
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
DataInputStream dis = new DataInputStream(bis);
dis.readFully(data);
} catch (IOException e) {
Log.e(TAG, e.getMessage());
}
FirebaseVisionBarcodeDetectorOptions options = new FirebaseVisionBarcodeDetectorOptions.Builder().setBarcodeFormats(
FirebaseVisionBarcode.FORMAT_QR_CODE,
FirebaseVisionBarcode.FORMAT_DATA_MATRIX
).build();
FirebaseVisionBarcodeDetector detector = FirebaseVision.getInstance().getVisionBarcodeDetector(options);
FirebaseVisionImage image;
int rotationResult;
switch (imgRotation) {
case 0: {
rotationResult = FirebaseVisionImageMetadata.ROTATION_0;
break;
}
case 90: {
rotationResult = FirebaseVisionImageMetadata.ROTATION_90;
break;
}
case 180: {
rotationResult = FirebaseVisionImageMetadata.ROTATION_180;
break;
}
case 270: {
rotationResult = FirebaseVisionImageMetadata.ROTATION_270;
break;
}
default: {
rotationResult = FirebaseVisionImageMetadata.ROTATION_0;
break;
}
}
FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
.setWidth(imgWidth)
.setHeight(imgHeight)
.setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
.setRotation(rotationResult)
.build();
image = FirebaseVisionImage.fromByteArray(data, metadata);
Task<List<FirebaseVisionBarcode>> result = detector.detectInImage(image)
A few things.
Your image format should not be NV21 if you use camera2. See here for all camera2 supported image formats:
https://developer.android.com/reference/android/media/Image#getFormat()
Your byte[] is not NV21 and you specified IMAGE_FORMAT_NV21 and led to the error
Most intuitive integration with camera2 is like below:
Specify JPEG format when you instantiate the ImageReader.
onImageAvailable will give you back an android.media.Image and you can directly use FirebaseVisionImage.fromMediaImage(...) to create a FirebaseVisionImage. (You can find how to compute the rotation info from official doc here)
If you must do two Activities, then you need to work around the fact that android.media.Image is not Parcelable. I'd suggest you convert it to Bitmap first which is Parcelable and you can directly set it as an Intent extra (Up to you. Just thinking from end user's perspective, it's non-common to see the barcode being saved to my image gallery.
So you might want to consider skipping the step of saving it to file). Later, in your 2nd Activity, you can use FirebaseVisionImage.fromBitmap(...).
Related
Currently, I'm building a camera app using camera2 api. I record the video and that file is sent to another activity to make sure we can correct orientation and watch the video. Then the altered video or picture is saved to the device.
When I use a still image it works, because I can pull the bitmap image and then resave the image like this:
public String saveImage() {
//Getting a new file name and file path
//Should we delete these images after were done with them?
File newImageFile = getExternalFilesDir(Environment.DIRECTORY_PICTURES);
File newImageFolder = new File(newImageFile, "camera2VideoImage");
if (newImageFolder.exists())
{
newImageFolder.mkdirs();
}
String timestamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date());
String prepend = "Image_" + timestamp + "_";
File imageFile = null;
try {
imageFile = File.createTempFile(prepend, ".jpg", newImageFolder);
} catch (IOException e) {
e.printStackTrace();
}
String newFileName = imageFile.getAbsolutePath();
Bitmap bitmap = ((BitmapDrawable) imageView.getDrawable()).getBitmap();
try {
FileOutputStream fos = new FileOutputStream(newFileName);
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, fos);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
//returning the new file path.
return newFileName;
}
This works, they view the image and have the option to change orientation on the image in case it is messed up and then we resave.
I'm trying to do the same with a video, but not sure how to pull the video from the textureView so I can save again with the corrected orientation as the front facing camera is upside down sometimes depending on the phone.
The recording save method:
public String saveVideo() {
//Getting a new file name and file path
//Should we delete these images after were done with them?
File newVideoFile = getExternalFilesDir(Environment.DIRECTORY_MOVIES);
File newVideoFolder = new File(newVideoFile, "camera2VideoImage");
if (newVideoFolder.exists())
{
newVideoFolder.mkdirs();
}
String timestamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date());
String prepend = "Video_" + timestamp + "_";
File videoFile = null;
try {
videoFile = File.createTempFile(prepend, ".jpg", newVideoFolder);
} catch (IOException e) {
e.printStackTrace();
}
String newFileName = videoFile.getAbsolutePath();
textureView.getBitmap();
FileOutputStream fos = new FileOutputStream(newFileName);
//returning the new file path.
return newFileName;
}
How do I get the corrected video from the texture view and then save it, like the Image method above?
This is not a recommended way; the cost of getBitmap on TextureView is high, and not likely suitable for 30fps video recording.
But if you really want to try, you need to feed the Bitmap to a MediaRecorder; you may be able to use MediaRecorder.getSurface() for that, then lock the Surface Canvas and draw your Bitmap into it.
However, I would not be surprised if the performance is poor, or if the MediaRecorder Surface won't accept RGB Bitmaps.
In general, you want to connect the camera API directly to the MediaRecorder or MediaCodec Surface. If you really need to edit frames in the middle, using the GPU is generally the most performant option, though it's a lot of code to write to do that.
I ended up changing the configuration on the Media Recorder, looks like I was trying to correct other orientation problems and caused this. So under my set up mediaRecorder I did this:
private void setupMediaRecorder() throws IOException {
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);
mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
mMediaRecorder.setOutputFile(mVideoFileName);
mMediaRecorder.setVideoEncodingBitRate(100000000);
mMediaRecorder.setVideoFrameRate(30);
mMediaRecorder.setVideoSize(mVideoSize.getWidth(),mVideoSize.getHeight());
mMediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);
mMediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
if (cameraCheck.contains("1") && mTotalRotation == 180){
//This corrects problems for the front facing camera when recording, the default settings work, so we do nothing here.
}else {
mMediaRecorder.setOrientationHint(mTotalRotation);
}
mMediaRecorder.prepare();
}
This checks if it is the front-facing camera and the phone is positioned at 180. If it is do nothing for correction, else use the correction.
I am making a camera app.
There has been a host of issues getting orientation right because some phones don't write EXIF orientation data. Because of this, I get the bitmap, save it (since I don't think I should read EXIF data from the byte[]), then rotate the bitmap, then save over the original file.
It works, and the the orientation issue is fixed. The problem is its taking me 25 seconds or longer on some of the top of the line phones. Can you advise why my code is so slow or advise me on how I can find the problem?
Note: If I only save the image once (i.e. with the wrong orientation) it only takes a couple seconds.
Here is my image capture callback:
private Camera.PictureCallback pictureCallback = new Camera.PictureCallback()
{
#Override
public void onPictureTaken(byte[] data, Camera camera)
{
File pictureFile = getOutputMediaFile(MEDIA_TYPE_IMAGE);
if (pictureFile == null){
Log.d("EditPhotoFragment", "Error creating media file, check storage permissions");
return;
}
try
{
FileOutputStream fos = new FileOutputStream(pictureFile);
fos.write(data);
fos.flush();
fos.close();
orientPicture(pictureFile);
//TODO async
galleryAddPic(pictureFile);
} catch (FileNotFoundException e) {
Log.d("EditPhotoFragment", "File not found: " + e.getMessage());
} catch (IOException e) {
Log.d("EditPhotoFragment", "Error accessing file: " + e.getMessage());
}
}
};
And here is where I orient and resave the image:
private Bitmap orientPicture(File pictureFile)
{
Bitmap bitmap = BitmapFactory.decodeFile(pictureFile.getAbsolutePath());
Uri uri = Uri.parse(pictureFile.toString());
ExifInterface exif = null;
try{
exif = new ExifInterface(uri.getPath());
}catch (Exception e)
{
e.printStackTrace();
}
int exifOrientation = exif.getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL);
Matrix matrix = new Matrix();
int rotationInDegrees = 0;
//If the orientation tag is missing need to manually rotate it by the 'default' camera
//orientation and if its front facing need to do 360 - the camera rotation value
if(exifOrientation == ExifInterface.ORIENTATION_UNDEFINED)//All phones in this bucket can go fuck themselves
{
Camera.CameraInfo info = new Camera.CameraInfo();
if(_cameraPreview.isBackFacing())
{
Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_BACK, info);
}else
{
Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_FRONT, info);
}
rotationInDegrees = info.orientation; //set it to the default camera orientation
}else
{
rotationInDegrees = exifToDegrees(exifOrientation);
if(!_cameraPreview.isBackFacing())//handle mirroring of front camera
{
Camera.CameraInfo info = new Camera.CameraInfo();
Camera.getCameraInfo(Camera.CameraInfo.CAMERA_FACING_BACK, info);
rotationInDegrees = 360 - rotationInDegrees; //For the front camera doing 360 - gets the right orientation
}
}
matrix.preRotate(rotationInDegrees);
if(!_cameraPreview.isBackFacing())//mirror it
{
matrix.preScale(1,-1);
}
Bitmap adjustedBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight(), matrix, true);
//This saves the proper image over top if it
try
{
FileOutputStream fos = new FileOutputStream(pictureFile);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
adjustedBitmap.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] byteArray = stream.toByteArray();
fos.write(byteArray);
fos.flush();
fos.close();
}catch(Exception e)
{
e.printStackTrace();
}
return adjustedBitmap;
}
SOLUTION
As advised I SHOULD read the exif data which I was able to do without needing an external library thanks to this:
https://stackoverflow.com/a/13581324/3324388
Can you advise why my code is so slow
Perhaps among other reasons, you are writing the image to a file, re-reading the same image from the file, doing the transform, then writing the image back out to a file. That is going to take a lot of time.
Note: If I only save the image once (i.e. with the wrong orientation) it only takes a couple seconds.
That's because you are doing a lot less work, including only ~33% of the disk I/O, and disk I/O is going to be slow.
since I don't think I should read EXIF data from the byte[]
My apologies if you were viciously attacked by a byte[] as a young child or something. However, if you want better performance, you are going to have to read the EXIF data out of the existing in-memory copy of the image.
It seems to me there are two ways to store an attachment in a NotesDocument.
Either as a RichTextField or as a "MIME Part".
If they are stored as RichText you can do stuff like:
document.getAttachment(fileName)
That does not seem to work for an attachment stored as a MIME Part. See screenshot
I have thousands of documents like this in the backend. This is NOT a UI issue where I need to use the file Download control of XPages.
Each document as only 1 attachment. An Image. A JPG file. I have 3 databases for different sizes. Original, Large, and Small. Originally I created everything from documents that had the attachment stored as RichText. But my code saved them as MIME Part. that's just what it did. Not really my intent.
What happened is I lost some of my "Small" pictures so I need to rebuild them from the Original pictures that are now stored as MIME Part. So my ultimate goal is to get it from the NotesDocument into a Java Buffered Image.
I think I have the code to do what I want but I just "simply" can't figure out how to get the attachment off the document and then into a Java Buffered Image.
Below is some rough code I'm working with. My goal is to pass in the document with the original picture. I already have the fileName because I stored that out in metaData. But I don't know how to get that from the document itself. And I'm passing in "Small" to create the Small image.
I think I just don't know how to work with attachments stored in this manner.
Any ideas/advice would be appreciated! Thanks!!!
public Document processImage(Document inputDoc, String fileName, String size) throws IOException {
// fileName is the name of the attachment on the document
// The goal is to return a NEW BLANK document with the image on it
// The Calling code can then deal with keys and meta data.
// size is "Original", "Large" or "Small"
System.out.println("Processing Image, Size = " + size);
//System.out.println("Filename = " + fileName);
boolean result = false;
Session session = Factory.getSession();
Database db = session.getCurrentDatabase();
session.setConvertMime(true);
BufferedImage img;
BufferedImage convertedImage = null; // the output image
EmbeddedObject image = null;
InputStream imageStream = null;
int currentSize = 0;
int newWidth = 0;
String currentName = "";
try {
// Get the Embedded Object
image = inputDoc.getAttachment(fileName);
System.out.println("Input Form : " + inputDoc.getItemValueString("form"));
if (null == image) {
System.out.println("ALERT - IMAGE IS NULL");
}
currentSize = image.getFileSize();
currentName = image.getName();
// Get a Stream of the Imahe
imageStream = image.getInputStream();
img = ImageIO.read(imageStream); // this is the buffered image we'll work with
imageStream.close();
Document newDoc = db.createDocument();
// Remember this is a BLANK document. The calling code needs to set the form
if ("original".equalsIgnoreCase(size)) {
this.attachImage(newDoc, img, fileName, "JPG");
return newDoc;
}
if ("Large".equalsIgnoreCase(size)) {
// Now we need to convert the LARGE image
// We're assuming FIXED HEIGHT of 600px
newWidth = this.getNewWidth(img.getHeight(), img.getWidth(), 600);
convertedImage = this.getScaledInstance(img, newWidth, 600, false);
this.attachImage(newDoc, img, fileName, "JPG");
return newDoc;
}
if ("Small".equalsIgnoreCase(size)) {
System.out.println("converting Small");
newWidth = this.getNewWidth(img.getHeight(), img.getWidth(), 240);
convertedImage = this.getScaledInstance(img, newWidth, 240, false);
this.attachImage(newDoc, img, fileName, "JPG");
System.out.println("End Converting Small");
return newDoc;
}
return newDoc;
} catch (Exception e) {
// HANDLE EXCEPTION HERE
// SAMLPLE WRITE TO LOG.NSF
System.out.println("****************");
System.out.println("EXCEPTION IN processImage()");
System.out.println("****************");
System.out.println("picName: " + fileName);
e.printStackTrace();
return null;
} finally {
if (null != imageStream) {
imageStream.close();
}
if (null != image) {
LibraryUtils.incinerate(image);
}
}
}
I believe it will be some variation of the following code snippet. You might have to change which mimeentity has the content so it might be in the parent or another child depending.
Stream stream = session.createStream();
doc.getMIMEEntity().getFirstChildEntity().getContentAsBytes(stream);
ByteArrayInputStream bais = new ByteArrayInputStream(stream.read());
return ImageIO.read(bais);
EDIT:
session.setConvertMime(false);
Stream stream = session.createStream();
Item itm = doc.getFirstItem("ParentEntity");
MIMEEntity me = itm.getMIMEEntity();
MIMEEntity childEntity = me.getFirstChildEntity();
childEntity.getContentAsBytes(stream);
ByteArrayOutputStream bo = new ByteArrayOutputStream();
stream.getContents(bo);
byte[] mybytearray = bo.toByteArray();
ByteArrayInputStream bais = new ByteArrayInputStream(mybytearray);
return ImageIO.read(bais);
David have a look at DominoDocument,http://public.dhe.ibm.com/software/dw/lotus/Domino-Designer/JavaDocs/XPagesExtAPI/8.5.2/com/ibm/xsp/model/domino/wrapped/DominoDocument.html
There you can wrap every Notes document
In the DominoDocument, there such as DominoDocument.AttachmentValueHolder where you can access the attachments.
I have explained it at Engage. It very powerful
http://www.slideshare.net/flinden68/engage-use-notes-objects-in-memory-and-other-useful-java-tips-for-x-pages-development
I have been augmenting the QR scanning library Zxing to save a photo instantly upon scan. I was advised to do so within the onPreviewFrame method within PreviewCallback.java as thus:
public void onPreviewFrame(byte[] data, Camera camera) {
Point cameraResolution = configManager.getCameraResolution();
Handler thePreviewHandler = previewHandler;
YuvImage im = new YuvImage(data, ImageFormat.NV21, 1200,
800, null);
Rect r = new Rect(0, 0, 1200, 800);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
im.compressToJpeg(r, 50, baos);
try {
FileOutputStream output = new FileOutputStream("/sdcard/test_jpg.jpg");
output.write(baos.toByteArray());
output.flush();
output.close();
System.out.println("Attempting to save file");
System.out.println(data);
} catch (FileNotFoundException e) {
System.out.println("Saving to file failed");
} catch (IOException e) {
System.out.println("Saving to file failed");
}
if (cameraResolution != null && thePreviewHandler != null) {
Message message = thePreviewHandler.obtainMessage(previewMessage, cameraResolution.x,
cameraResolution.y, data);
message.sendToTarget();
previewHandler = null;
} else {
Log.d(TAG, "Got preview callback, but no handler or resolution available");
}}
The result of running this code is a corrupt image at the set file directory. I believe this is due to the code being run every frame. Is there a way to limit this to every second or so if that will allow the full image to save, or is there a method I can use to cause the image to only save upon completed scan.
I have a less favourable working alternative, in that I can successfully save the black and white image that is shown upon scan; colour is the preferable option of course.
Update: Code changed to (in theory) accommodate camera resolution on any device. Image is still corrupt.
public void onPreviewFrame(byte[] data, Camera camera) {
Point cameraResolution = configManager.getCameraResolution();
Handler thePreviewHandler = previewHandler;
android.hardware.Camera.Parameters parameters = camera.getParameters();
android.hardware.Camera.Size size = parameters.getPictureSize();
int height = size.height;
int width = size.width;
YuvImage im = new YuvImage(data, ImageFormat.NV21, width,
height, null);
Rect r = new Rect(0, 0, width, height);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
im.compressToJpeg(r, 50, baos);
try {
FileOutputStream output = new FileOutputStream("/sdcard/test_jpg.jpg");
output.write(baos.toByteArray());
output.flush();
output.close();
System.out.println("Attempting to save file");
System.out.println(data);
} catch (FileNotFoundException e) {
System.out.println("Saving to file failed");
} catch (IOException e) {
System.out.println("Saving to file failed");
}
if (cameraResolution != null && thePreviewHandler != null) {
Message message = thePreviewHandler.obtainMessage(previewMessage, cameraResolution.x,
cameraResolution.y, data);
message.sendToTarget();
previewHandler = null;
} else {
Log.d(TAG, "Got preview callback, but no handler or resolution available");
}}
Having checked the width and height variable are correctly set to the Nexus 7's camera width and height of 1280x960. I am confident the issue comes from attempting to save the image every frame, as "Attempting to save to file" appears somewhat rapidly within the logcat; several times a second. It may also be worth noting that the corrupt image saved is square(ish).
Thanks in advance.
1200 x 800? are you sure this is your preview size? check your parameters. Probably it's 1280 x 720
For some reason, my camera app saves all photos rotated 90 degrees (pictures only look right when taken with camera on landscape mode) I believe onPictureTaken should rotate photos automatically but I read there is a problem with Samsung devices (I haven't been able to test it on another brand so I don't know if it's the case). This is my code:
public void onPictureTaken(byte[] data, Camera camera) {
// Generate file name
FileOutputStream outStream = null;
outStream = new FileOutputStream(filePath);
outStream.write(data);
outStream.close();
I was thinking it could be fixed by checking the orientation and rotating the byte array but there must be a more straightforward way to do it since handling byte arrays is a pain.
How can I make sure photos are saved matching the orientation they were taken?
Try something like this:
int orientation = Exif.getOrientation(data);
Log.d("#", "onPictureTaken().orientation = " + orientation);
if(orientation != 0) {
Bitmap bmpSrc = BitmapFactory.decodeByteArray(data, 0, data.length);
Bitmap bmpRotated = CameraUtil.rotate(bmpSrc, orientation);
bmpSrc.recycle();
try {
FileOutputStream localFileOutputStream = new FileOutputStream(filePath);
bmpRotated.compress(Bitmap.CompressFormat.JPEG, 90,localFileOutputStream);
localFileOutputStream.flush();
localFileOutputStream.close();
bmpRotated.recycle();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
} else {
try {
FileOutputStream localFileOutputStream = new FileOutputStream(filePath);
localFileOutputStream.write(data);
localFileOutputStream.flush();
localFileOutputStream.close();
} catch (IOException localIOException)
{
Log.e("#",localIOException.getMessage());
}
}
There are different ways to get your image, as data, as stream, as file, also it is different for camera and gallery and other apps. For each of them you have another way to access the orientation tag. Once you have the orientation, you can rotate your image.
You - or for that matter everyone - should really get a Nexus, they are nice and rotate the image for you and set orientation to 0, while the lazy ones like Samsung just store the image and set the orientation tag. ;)