I have a situation in which I need to transmit an object from back-end to front-end in this format:
{
filename: "filename",
type: "type",
src: "src",
bytes: "base64Representation"
}
The bytes property of the object consists in the base64 representation of a file stored in a repository in the remote server. Up until now I've worked with small files in the range 1-2MB and the code for converting a file to the corresponding base64 representation has worked correctly. But now I'm facing some problems with big files, larger than 100MB. I've checked solutions that try to convert the file chunk by chunk, but still at the end of the process I need all the chunks concatenated in a string and at this step I'm getting an OutOfMemory exception. I've also seen some suggestions to use OutputStreams, but I can't apply them because I need the data in the above format. Please does anyone have any suggestions on how can I bypass this situation?
You can use OutputStream and process on the fly in a servlet by wrapping response.getOutputStream(). I will give a working example with spring boot. I tested and it works.
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpServletResponse;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Base64;
#RestController
public class Base64Controller {
#RequestMapping(value = "/base64", method = RequestMethod.GET)
public void getBase64File(HttpServletResponse response) throws IOException {
response.setContentType("text/plain");
OutputStream wrap = Base64.getEncoder().wrap(response.getOutputStream());
FileInputStream fis = new FileInputStream("./temp.txt");
int bytes;
byte[] buffer = new byte[2048];
while ((bytes=fis.read(buffer)) != -1) {
wrap.write(buffer, 0, bytes);
}
fis.close();
wrap.close();
}
}
A JSON response is a kludge here, with Base64 having a payload of 6/8th per byte, you have 33% more data transfer as needed. Indeed a JSON DOM object is overstretching both the server as also the client side.
So convert it to a simple binary download, and stream it out; possibly throttled for large data.
This means a change in the API.
I never worked with struts, so i'm not sure will this work, but it should be something like that
public class DownloadB64Action extends Action{
private final static BUFFER_SIZE = 1024;
#Override
public ActionForward execute(ActionMapping mapping, ActionForm form,
HttpServletRequest request, HttpServletResponse response)
throws Exception {
response.setContentType("text/plain");
try
{
FileInputStream in =
new FileInputStream(new File("myfile.b64"));
ServletOutputStream out = Base64.getEncoder().wrap(response.getOutputStream());
byte[] buffer = new byte[BUFFER_SIZE];
while(in.read(buffer, 0, BUFFER_SIZE) != -1){
out.write(buffer, 0, BUFFER_SIZE);
}
in.close();
out.flush();
out.close();
}catch(Exception e){
//TODO handle exception
}
return null;
}
}
to make it JSON structure like you need, you might try to write directly to response.getOutputStream() "{\"filename\":\"filename\",\"type\":\"type\",\"src\":\"src\",\"bytes\": \"".getBytes() before b64 payload and "\"}".getBytes() after
}
Related
I want to expose an API to download a S3 bucket file content as stream to its consumers.
The API URL is like /downloadfile/** which is GET request.
What should be my return type right now I tried with accept header=
application/octet-stream which didn't work.
I don't want to write the content of file to any file and send it. It should be returned as a stream that's it.
Here is the controller pseudo code I wrote till now which is giving me 406 error all the time.
#GetMapping(value = "/downloadfile/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
public ResponseEntity<Object> downloadFile(HttpServletRequest request) {
//reads the content from S3 bucket and returns a S3ObjectInputStream
S3ObjectInputStream object = null;
object = publishAmazonS3.getObject("12345bucket", "/logs/file1.log").getObjectContent();
return object
}
Any suggestions here on the way of doing this and what I am doing wrong?
I was able to download the file as a stream by using StreamingResponseBody class from Spring.
Here is the code I used:
#GetMapping(value = "/downloadfile/**", produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE })
public ResponseEntity<StreamingResponseBody> downloadFile(HttpServletRequest request) {
//reads the content from S3 bucket and returns a S3ObjectInputStream
S3Object object = publishAmazonS3.getObject("12345bucket", "/logs/file1.log");
S3ObjectInputStream finalObject = object.getObjectContent();
final StreamingResponseBody body = outputStream -> {
int numberOfBytesToWrite = 0;
byte[] data = new byte[1024];
while ((numberOfBytesToWrite = finalObject.read(data, 0, data.length)) != -1) {
System.out.println("Writing some bytes..");
outputStream.write(data, 0, numberOfBytesToWrite);
}
finalObject.close();
};
return new ResponseEntity<>(body, HttpStatus.OK);
}
The way to test the streaming is done correctly or not is to have a file of around 400mb to download. Reduce your Xmx to 256mb by passing the in the vm options. Now, compare the download functionality with and without using StreamingResponseBody, you will get OutofMemoryError when using the conventional OutputStreams for writing the content
You can solve with the below example
import java.io.ByteArrayOutputStream;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;
import com.grokonez.s3.services.S3Services;
#RestController
public class DownloadFileController {
#Autowired
S3Services s3Services;
/*
* Download Files
*/
#GetMapping("/api/file/{keyname}")
public ResponseEntity<byte[]> downloadFile(#PathVariable String keyname) {
ByteArrayOutputStream downloadInputStream = s3Services.downloadFile(keyname);
return ResponseEntity.ok().contentType(contentType(keyname))
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + keyname + "\"")
.body(downloadInputStream.toByteArray());
}
private MediaType contentType(String keyname) {
String[] arr = keyname.split("\\.");
String type = arr[arr.length - 1];
switch (type) {
case "txt":
return MediaType.TEXT_PLAIN;
case "png":
return MediaType.IMAGE_PNG;
case "jpg":
return MediaType.IMAGE_JPEG;
default:
return MediaType.APPLICATION_OCTET_STREAM;
}
}
}
Given that it is now possible to handle binary data in Amazon Api Gateway and Amazon Lambda, I wanted to try to make an Amazon Lambda endpoint which returned an Excel spreadsheet. It is entirely possible to do so using node/js, as demonstrated here. Unfortunately, any time I try to do this using Java, it falls to pieces.
My initial attempt was to create a simple workbook using apache XSSFWorkbook, write it to the output stream provided by RequestStreamHandler, and done.
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class FileRequestHandler implements RequestStreamHandler {
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
throws IOException {
Workbook wb = new XSSFWorkbook();
String sheetName = "Problem sheet";
wb.createSheet(sheetName);
wb.write(outputStream);
}
}
When tested locally, the output stream can be piped to a file resulting in a valid output excel file.
import com.amazonaws.util.StringInputStream;
import org.junit.Test;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
public class FileRequestHandlerTest {
#Test
public void shouldCreateExcelFile() throws IOException {
FileRequestHandler fileRequestHandler = new FileRequestHandler();
InputStream inputStream = new StringInputStream("hello world");
String fileName = "FileRequestLambda";
String path = fileName + ".xlsx";
FileOutputStream fileOutputStream = new FileOutputStream(path);
fileRequestHandler.handleRequest(inputStream, fileOutputStream, TestUtils.createContext());
fileOutputStream.close();
}
}
But when I run it in Amazon Lambda, I get malformed binary output:
PKn��I_rels/.rels���j�0��}
�{㴃1F�^Ơ�2��l%1I,c�[��3�l
l�����H��4��R�l��·����q}*�2�������;�*��
t"�^�l;1W)�N�iD)ejuD�cKz[:}g����#:�
�3����4�7N�s_ni�G�M*7�����2R�+�� �2�/�����b��mC�Pp�ֱ$POyQ�抒�DsZ��IС�'un���~�PK����OPKn��I[Content_Types].xml�SMO1��+6��m��1���G%��β
�J[���MDL0�S;yo�{3i�Ӎ5�c��5lć�B'��nѰ��S}˪��)0�aÜg��`<�L��.�p'D�ZH�t��>Z�Tƅ ��#q=��]F��\4�=`+���P�!-!S.�v�#��+�����N�tEV=nHe7���S,;K]_h7Q+�W8߶Z��re��c�U�����}�����g�&A��,���H�$�B<��`�"�Jb���"���I�N�1���A���CI�#��܂v��?|\�{��`�b������$�c�D��|2�PKKB�>'PKn��IdocProps/app.xmlM��
�0D�~EȽ��ADҔ���A? ��6�lB�J?ߜ���0���ͯ��)�#��H6���V>��$;�SC
;̢(�ra�g�l�&�e��L!y�%��49��`_���4G���F��J��Wg
�GS�b����
~�PK�|wؑ�PKn��IdocProps/core.xmlm��J�0F��!�m�V����(���Ż��m��!�v}{ӺVP/g��a��wG5�wp~4��4�1-�u���n��c�גOFC����6��e�888c��<�홰
B��/P�g��q�b��!��'��W�)��"
�<p�S��I)Ŧ�onZR�#��Ќ�6�S�߅u��G?n�<��\�\����ۛ���t���p|��f� Q4��ac&ߓ��������i��"�UG+vV��z�ɯ���U�^�H#�����IM�$�&�PK����PKn��Ixl/sharedStrings.xml=�A� ﾂ��.z0Ɣ�`������,�����q2��o�ԇ���N�E��x5�z>�W���(R�K���^4{�����ŀ�5��y�V����y�m�XV�\�.��j�����
8�PKp��&x�PKn��I
xl/styles.xml���n� ��>bop2TQ��P)U��RWb�6*�����ӤS�Nw�s���3ߍ���t��(l��������ҝx�!N=#$ɀ��}��3c���ʰr`:i��2��w,�
�d
�T��R#�voc �;c�iE���Û��E<|��4Iɣ�����F#��n���B�z�F���y�j3y��yҥ�jt>���2��Lژ�!6��2F��OY��4#M�!���G��������1�t��y��p��" n����u�����a�ΦDi�9�&#��%I��9��}���cK��T��$?������`J������7���o��f��M|PK�1X#C�PKn��Ixl/workbook.xml���N�0��<��wj�E�8��J��P�;�����hmZ'Q�#����~;���;vCJ6 �Fà���"��|x|�}���#]����C�0�<֜'=�WiG��#y���O#�2i#������+`!��F�{��-�O�!/B�r)�;&h�����zOz�o����xO��I2����YuĔ��s�u��<J8Q�z6��Qm�:�,�c��Z�����PK1����dPKn��Ixl/_rels/workbook.xml.rels��Mk1#���0�nv-�R�^����0$����$dƯo���R�OC�ރ�-��������#Sՠ(�����ܼ?��b��p�����d�AJ�¾O�
#�/�f�iD�b�P6m�#Jy�N'�[�HO��E�k����3�W���ܑ`���Zri㪐����?�ض��e�������7p�wj�W5r���]������=�|���<:�[p��7�O�PK��4��9PKn��Ixl/worksheets/sheet1.xmleP�N�0���މ�V��THU$���$��j���[��c�����3��-v�nT���/a����7�Zߗ��z���]uQ���0 ��zJD�[�C3�3!� }|鈝�H��ab4�br�^���v�z���:�)P1v%ܭ#W�"|�8�?X�ܚ���C[B�'�~��ȅO������Tyb�bgN�<�|��$��ƙ��{#&����h��>��D�Ű�z�#��6��8�LF�dQ����,4�xS����/PK�_�Y�lPKn��I����O_rels/.relsPKn��IKB�>'[Content_Types].xmlPKn��I�|wؑ��docProps/app.xmlPKn��I����mdocProps/core.xmlPKn��Ip��&x��xl/sharedStrings.xmlPKn��I�1X#C�
nxl/styles.xmlPKn��I1����d�xl/workbook.xmlPKn��I��4��9xl/_rels/workbook.xml.relsPKn��I�_�Y�l$ xl/worksheets/sheet1.xmlPK ?Z
The output is about 5KB in size, while the output on my local computer is about 3KB in size. This appears to be a problem with binary output in general for Java on Amazon Lambda. When I do run some code that writes an image to the output string, it also works locally, but results in an image twice the size and garbled when run from Amazon Lambda.
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestStreamHandler;
import java.io.*;
import java.net.URL;
public class ImageRequestHandler implements RequestStreamHandler {
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
throws IOException {
String address = "https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/580px-AmazonWebservices_Logo.svg.png";
URL url = new URL(address);
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n;
while (-1!=(n=in.read(buf)))
{
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
outputStream.write(response);
}
}
The types of the was input and output streams are:
lambdainternal.util.NativeMemoryAsInputStream
lambdainternal.util.LambdaByteArrayOutputStream
Help?
I had the same problem with returning JPG image from Amazon Lambda and I found a work-around.
You need to encode an output stream with base64 encoding:
OutputStream encodedStream = Base64.getEncoder().wrap(outputStream);
encodedStream.write(response);
encodedStream.close();
Then you need to update Method Response and Integration Response of your function as described here: AWS Gateway API base64Decode produces garbled binary?
I've been trying for days to do this and got absolutely nowhere. I know it can be done, but I've been trawling SO for answers and got nothing working.
Upload a picture using my REST client
Insert that uploaded picture into the MySQL database.
What I have tried:
Following Load_File doesn't work, I'm using OS X so I don't know how to change ownership of folders etc... how do I do this? I never got an answer in my last post about this. How do I do this?
I've also tried doing it another way: http://examples.javacodegeeks.com/enterprise-java/rest/jersey/jersey-file-upload-example/
This does not work at all. I keep getting the error described in this post: Jersey REST WS Error: "Missing dependency for method... at parameter at index X", but the answer doesn't help me as I still don't know what it should be...
Can anyone please guide me through it?
I'm using a Jersey REST client in Java. Many of the tutorials to do this mention a pom.xml file, I don't have one or know what it is.
Thank you,
Omar
EDIT:
This is the file upload:
package com.omar.rest.apimethods;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import com.sun.jersey.core.header.FormDataContentDisposition;
import com.sun.jersey.multipart.FormDataParam;
#Path("/files")
public class FileUpload {
private String uploadLocationFolder = "/Users/Omar/Pictures/";
#POST
#Path("/upload")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(
#FormDataParam("file") InputStream fileInputStream,
#FormDataParam("file") FormDataContentDisposition contentDispositionHeader) {
String filePath = "/Users/Omar/Pictures/" + contentDispositionHeader.getFileName();
// save the file to the server
saveFile(fileInputStream, filePath);
String output = "File saved to server location : " + filePath;
return Response.status(200).entity(output).build();
}
// save uploaded file to a defined location on the server
private void saveFile(InputStream uploadedInputStream,
String serverLocation) {
try {
OutputStream outpuStream = new FileOutputStream(new File(serverLocation));
int read = 0;
byte[] bytes = new byte[1024];
outpuStream = new FileOutputStream(new File(serverLocation));
while ((read = uploadedInputStream.read(bytes)) != -1) {
outpuStream.write(bytes, 0, read);
}
outpuStream.flush();
outpuStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Schema for the table (one I created for testing):
image_id: int auto-incrementing PK, picture: BLOB.
I could make it a file link and just load the image on my website but I can't even get that far yet.
I would recommend storing your image in some kind of cheap, well permissioned flat storage like network storage, and then storing a path to that storage location in the database. If you're storing your image as a blob, the database is going to do something similar to this already anyways, but I believe there will be some overhead involved with making the database manage storing and retrieving these images. These images will eat through a lot of your database's disk space, and if you want to add more space for images, adding space to flat storage should be easier than adding space to a database.
I have a phonegap plugin I altered. The Java part outputs a base64 string:
package org.apache.cordova;
import java.io.ByteArrayOutputStream;
import java.io.File;
import org.apache.cordova.api.Plugin;
import org.apache.cordova.api.PluginResult;
import org.json.JSONArray;
import android.annotation.TargetApi;
import android.graphics.Bitmap;
import android.os.Environment;
import android.util.Base64;
import android.view.View;
public class Screenshot extends Plugin {
#Override
public PluginResult execute(String action, JSONArray args, String callbackId) {
// starting on ICS, some WebView methods
// can only be called on UI threads
final Plugin that = this;
final String id = callbackId;
super.cordova.getActivity().runOnUiThread(new Runnable() {
//#Override
#TargetApi(8)
public void run() {
View view = webView.getRootView();
view.setDrawingCacheEnabled(true);
Bitmap bitmap = Bitmap.createBitmap(view.getDrawingCache());
view.setDrawingCacheEnabled(false);
File folder = new File(Environment.getExternalStorageDirectory(), "Pictures");
if (!folder.exists()) {
folder.mkdirs();
}
File f = new File(folder, "screenshot_" + System.currentTimeMillis() + ".png");
System.out.println(folder);
System.out.println("screenshot_" + System.currentTimeMillis() + ".png");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, baos);
byte[] b = baos.toByteArray();
String base64String = Base64.encodeToString(b, Base64.DEFAULT);
String mytextstring = "data:image/png;base64,"+base64String;
System.out.println(mytextstring);
that.success(new PluginResult(PluginResult.Status.OK, mytextstring), id);
}
});
PluginResult imageData = new PluginResult(PluginResult.Status.NO_RESULT);
imageData.setKeepCallback(true);
System.out.println("imageData=============>>>>>"+imageData);
return imageData;
}
}
I then pass this to some Javascript and then send the string to a server. I have checked the string that the .php file receives, and the base64 string is identical. However when I decode the base64 string it seems corrupt. For a better example copy the contents of this text file into a decoder.
http://dl.dropbox.com/u/91982671/base64.txt
Note: When the .php file tries to decode it data:image/png;base64, is infront, I have just removed it for the ease of you pasting it into a decoder.
Decoder found here:
http://www.motobit.com/util/base64-decoder-encoder.asp
All I can think is that for some reason I may not be outputting the base64 string correctly from the Java. Does anyone have any idea whats going on? Or what may cause this?
I played about with this for a good few hours last night and took some of these suggestions into consideration.
Firstly I checked the image before I encoded it. It was fine.
However decoding it before it goes to the Javascript showed that it was corrupted, this meant it had to be something to do with the Java encoding process. To solve this, and I don't claim to 100% understand why it happens, but the the problem seems to lay with this code:
String mytextstring = "data:image/png;base64,"+base64String;
and the way I was adding "data:/image/png;base64," before I sent it to the Javascript and on to the PHP decoder. To resolve this I removed it from the Java code so it became:
String mytextstring = base64String;
And in my JavaScript function that sent it to the server I added it to the string there, this works and I received an uncorrupted image. Just in-case anyone wonders/cares the Javascript function where I add it instead is below:
function returnScreenshotImage(imageData) {
base64string = "data:image/png;base64,"+imageData;
console.log("String: "+base64string);
var url = 'http://www.websitename.co.uk/upload.php';
var params = {image: imageData};
document.basicfrm.oldscreenshotimg.value = document.basicfrm.screenshotimg.value;
// send the data
$.post(url, params, function(data) {
document.basicfrm.screenshotimg.value = data;
});
}
As you can see the line:
base64string = "data:image/png;base64,"+imageData;
Adds the section previously added by the Java. This works now. Hope this helps people in the future. If anyone would care to comment ad explain why this is if they know feel free. :)
I'm using PDF-Renderer to view PDF files within my java application. It's working perfectly for normal PDF files.
However, i want the application to be able to display encrypted PDF files. The ecrypted file will be decrypted with CipherInputStream, but i do not want to save the decrypted data on disk. Am trying to figure a way i can pass the decryted data from CipherInputStream to the PDFFile constructor without having to write the decryted data to file.
I will also appreciate if someone can help with a link to PDF-Renderer tutorial, so that i can read up more on it.
Thanks.
Try the following class:
import com.sun.pdfview.PDFFile;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
public class PDFFileUtility {
private static final int READ_BLOCK = 8192;
public static PDFFile getPDFFile(InputStream in) throws IOException {
ReadableByteChannel bc = Channels.newChannel(in);
ByteBuffer bb = ByteBuffer.allocate(READ_BLOCK);
while (bc.read(bb) != -1) {
bb = resizeBuffer(bb); //get new buffer for read
}
return new PDFFile(bb);
}
private static ByteBuffer resizeBuffer(ByteBuffer in) {
ByteBuffer result = in;
if (in.remaining() < READ_BLOCK) {
result = ByteBuffer.allocate(in.capacity() * 2);
in.flip();
result.put(in);
}
return result;
}
}
So call:
PDFFileUtility.getPDFFile(myCipherInputStream);