I need to build a webservice with Jersey that downloads a big file from another service and returns to the client.
I would like jersey to read some bytes into a buffer and write those bytes to client socket.
I would like it to use non blocking I/O so I dont keep a thread busy. (This could not be achieved)
#GET
#Path("mypath")
public void getFile(final #Suspended AsyncResponse res) {
Client client = ClientBuilder.newClient();
WebTarget t = client.target("http://webserviceURL");
t.request()
.header("some header", "value for header")
.async().get(new InvocationCallback<byte[]>(){
public void completed(byte[] response) {
res.resume(response);
}
public void failed(Throwable throwable) {
res.resume(throwable.getMessage());
throwable.printStackTrace();
//reply with error
}
});
}
So far I have this code and I believe Jersey would download the complete file and then write it to the client which is not what I want to do.
any thoughts??
The client side async request, isn't going to do much for your use case. It's more mean for "fire and forget" use cases. What you can do though is just get the InputStream from the client Response and mix with a server side StreamingResource to stream the results. The server will start sending the data as it is coming in from the other remote resource.
Below is an example. The "/file" endpoint is the dummy remote resource that serves up the file. The "/client" endpoint consumes it.
#Path("stream")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public class ClientStreamingResource {
private static final String INFILE = "Some File";
#GET
#Path("file")
public Response fileEndpoint() {
final File file = new File(INFILE);
final StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) {
try (FileInputStream in = new FileInputStream(file)) {
byte[] buf = new byte[512];
int len;
while ((len = in.read(buf)) != -1) {
out.write(buf, 0, len);
out.flush();
System.out.println("---- wrote 512 bytes file ----");
}
} catch (IOException ex) {
throw new InternalServerErrorException(ex);
}
}
};
return Response.ok(output)
.header(HttpHeaders.CONTENT_LENGTH, file.length())
.build();
}
#GET
#Path("client")
public void clientEndpoint(#Suspended final AsyncResponse asyncResponse) {
final Client client = ClientBuilder.newClient();
final WebTarget target = client.target("http://localhost:8080/stream/file");
final Response clientResponse = target.request().get();
final StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) {
try (final InputStream entityStream = clientResponse.readEntity(InputStream.class)) {
byte[] buf = new byte[512];
int len;
while ((len = entityStream.read(buf)) != -1) {
out.write(buf, 0, len);
out.flush();
System.out.println("---- wrote 512 bytes client ----");
}
} catch (IOException ex) {
throw new InternalServerErrorException(ex);
}
}
};
ResponseBuilder responseBuilder = Response.ok(output);
if (clientResponse.getHeaderString("Content-Length") != null) {
responseBuilder.header("Content-Length", clientResponse.getHeaderString("Content-Length"));
}
new Thread(() -> {
asyncResponse.resume(responseBuilder.build());
}).start();
}
}
I used cURL to make the request, and jetty-maven-plugin to be able to run the example from the command line. When you do run it, and make the request, you should see the server logging
---- wrote 512 bytes file ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
...
while cURL client is keeping track of the results
The point to take away from this is that the "remote server" logging is happening the same time as the client resource is logging. This shows that the client doesn't wait to receive the entire file. It starts sending out bytes as soon as it starts receiving them.
Some things to note about the example:
I used a very small buffer size (512) because I was testing with a small (1Mb) file. I really didn't want to wait for a large file for testing. But I would imagine large files should work just the same. Of course you will want to increase the buffer size to something larger.
In order to use the smaller buffer size, you need to set the Jersey property ServerProperties.OUTBOUND_CONTENT_LENGTH_BUFFER to 0. The reason is that Jersey keeps in internal buffer of size 8192, which will cause my 512 byte chunks of data not to flush, until 8192 bytes were buffered. So I just disabled it.
When using AsyncResponse, you should use another thread, as I did. You may want to use executors instead of explicitly creating threads though. If you don't use another thread, then you are still holding up the thread from the container's thread pool.
UPDATE
Instead of managing your own threads/executor, you can annotate the client resource with #ManagedAsync, and let Jersey manage the threads
#ManagedAsync
#GET
#Path("client")
public void clientEndpoint(#Suspended final AsyncResponse asyncResponse) {
...
asyncResponse.resume(responseBuilder.build());
}
Related
Got a bit of a problem atm. for my "inapp"-update im downloading the new base64 encoded .apk from my webspace. I have the functionality pretty much down, this is the code without decoding.
public void onResponse(Call call, Response response) throws IOException {
if(response.isSuccessful()){
ResponseBody body = response.body();
BufferedSource source = body.source();
source.request(Long.MAX_VALUE);
Buffer buffer = source.buffer();
String rString = buffer.clone().readString(Charset.forName("UTF-8"));
Log.i("Test: ", AppUtils.decodeBase64(rString));
if(rString.equals("xxx")){
EventBus.getDefault().post(new KeyNotValid());
dispatcher.cancelAll();
}else{
EventBus.getDefault().post(new SaveKey(apikey));
BufferedSink sink = Okio.buffer(Okio.sink(myFile));
sink.writeAll(source);
sink.flush();
sink.close();
}
}
}
The Buffer/Log is not really necessary, just using it to check the response during testing.
How would i go about decoding the bytes before i write them to the sink?
I tried doing it via. ByteString, but i couldn't find a way to write the decoded String back to a BufferedSource.
Most alternatives are pretty slow like reopening the file afterwards, reading the bytes into memory, decode and write them back.
Would really appreciate any help on this
cheers
You can already consume the response as an InputStream via ResponseBody.byteStream. You can decorate this stream with https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html and use it to read a stream of bytes and write it to the Sink for the file in chunks.
I know this answer arrives quite late and that Yuri's answer is technically correct, but I think the most idiomatic way to do that is to take advantage of the composition pattern promoted by Okio to create a Source that decodes from Base64 (or a Sink that encodes to Base64, if you need so).
Here's a little proof of concept (I'm sure it can be improved):
public class Base64Source implements Source {
private Source delegate;
private Base64.Decoder decoder; // Using Java 8 API, but it can be any library
public Base64Source(Source delegate) {
this(delegate, Base64.getDecoder());
}
public Base64Source(Source delegate, Base64.Decoder decoder) {
this.delegate = delegate;
this.decoder = decoder;
}
#Override
public long read(Buffer sink, long byteCount) throws IOException {
Buffer buffer = new Buffer();
long actualRead = this.delegate.read(buffer, byteCount);
if (actualRead == -1) {
return -1;
}
byte[] encoded = buffer.readByteArray(actualRead);
byte[] decoded = decoder.decode(encoded);
sink.write(decoded);
return decoded.length;
}
#Override
public Timeout timeout() {
return this.delegate.timeout();
}
#Override
public void close() throws IOException {
this.delegate.close();
}
}
And here's how it can be used
BufferedSource source = Okio.buffer(new Base64Source(originalSource));
BufferedSink sink = ... // create sink
sink.writeAll(source);
// Don't forget to close the source/sink to flush and free resources
sink.close();
source.close();
I want to retrieve the pdf (Stored as BLOB) from Database using jersey api
I am using mybatis as data base framework .
I am able to download the pdf but the problem is i get the input stream as database to which i save it as file and then pass that it in Response but i don't want to save that file in server , i want file directly to be downloaded to user .
Current Process :
DATABASE-------> input stream-----> File -----------> add to response ----->user downloads it
retrieving making file passing file user downloads
What i want :
DATABASE---------->input stream------------> add to response -------> user downloads it
retrieving passing file user downloads
I want remove File making in server as data is confidential
Resource interface
#GET
#Path("v1/download/{id}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response downloadFile(#PathParam("id") int id) throws IOException, SQLException;
Resource Impl
#Override
public Response downloadFile(int id) throws IOException, SQLException {
// TODO Auto-generated method stub
File file = fileUploadService.downloadFile(id);
ResponseBuilder response = Response.ok(file);
response.header("Content-Disposition", "attachment;filename=aman.pdf");
return response.build();
}
Service method
#Override
public File downloadFile(int id) throws IOException {
// TODO Auto-generated method stub
File fil=new File("src/main/resources/Sample.pdf");
FileUploadModel fm =mapper.downloadFile(id);
InputStream inputStream = fm.getDaFile();
outputStream = new FileOutputStream(fil);
int read = 0;
byte[] bytes = new byte[102400000];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
return fil;
}
This code is working but i want to remove making of file on server side i.e i want to remove File fil=new File("src/main/resources/Sample.pdf"), this operation which is in service method .
Thanks in advance.
Instead of using File, use ByteArrayOutputStream and write to it. Then return the result as a byte[] which you can pass to your Response.ok(content).
Didn't test this, but something like this:
public byte[] downloadFile(int id) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
FileUploadModel fm =mapper.downloadFile(id);
InputStream inputStream = fm.getDaFile();
int read = 0;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
return out.toByteArray();
}
Also, that's a lot of bytes to allocate to an array. You can experiment with what works for you, but something like 1024 would likely be totally sufficient.
You'll probably also want to add another header to your response for Content-Type.
I want to receive the multiple files uploaded from my client-side. I uploaded multiple files and request my server-side (Java) using JAX-RS(Jersey).
I have the following code,
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public void upload(#Context UriInfo uriInfo,
#FormDataParam("file") final InputStream is,
#FormDataParam("file") final FormDataContentDisposition detail) {
FileOutputStream os = new FileOutputStream("Path/to/save/" + appropriatefileName);
byte[] buffer = new byte[1024];
int length;
while ((length = is.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
}
How can i write the files separately in the server side as uploaded in the client side.
For eg. I uploaded files such as My_File.txt, My_File.PNG, My_File.doc.
I need to write as same as the above My_File.txt, My_File.PNG, My_File.doc in the server side.
How can I achieve this?
You could try something like this:
#POST
#Consumes(MediaType.MULTIPART_FORM_DATA)
public void upload(FormDataMultiPart formParams)
{
Map<String, List<FormDataBodyPart>> fieldsByName = formParams.getFields();
// Usually each value in fieldsByName will be a list of length 1.
// Assuming each field in the form is a file, just loop through them.
for (List<FormDataBodyPart> fields : fieldsByName.values())
{
for (FormDataBodyPart field : fields)
{
InputStream is = field.getEntityAs(InputStream.class);
String fileName = field.getName();
// TODO: SAVE FILE HERE
// if you want media type for validation, it's field.getMediaType()
}
}
}
There is a blog for the scenario you are looking for. Hope this helps
http://opensourzesupport.wordpress.com/2012/10/27/multiple-file-upload-along-with-form-data-in-jax-rs/
I want to stream directly from an Oracle database blobs files via WS with MTOM directly to the WS client.
I thought I found a way which is described here:
http://www.java.net/forum/topic/glassfish/metro-and-jaxb/mtom-best-practices
but after i took a look on InputStreamDataSource and javax.mail.util.ByteArrayDataSource i realized that they acutal hava a byte[] of the 'document' in memory meaning the streaming ideea is in vain, cause what i try to avoid is to have multiple docs in the same time fully in memory.
So how can I stream from DB via WS and MTOM to a WS client ?
Any idea ?
Thanks
Cris
I tried experimenting and finally i had some positive results.
In order to stream from DB directly to clients browser the above
things are valid but the InputStreamDataSource should be like this:
public class InputStreamDataSource implements DataSource {
private InputStream inputStream;
public InputStreamDataSource(InputStream inputStream) {
this.inputStream = inputStream;
}
public InputStream getInputStream() throws IOException {
return inputStream;
}
public OutputStream getOutputStream() throws IOException {
throw new UnsupportedOperationException("Not implemented");
}
public String getContentType() {
return "*/*";
}
public String getName() {
return "InputStreamDataSource";
}
}
What I was affraid is that once I closed the input stream myself...
the ws client did not received the binary content...
Than i check and actually the DataHandler creates a new thread and closes the input stream
I was able to stream 500MB from DB to client fast and with low memory footprint !
I've got a Java web service in JAX-WS that returns an OutputStream from another method. I can't seem to figure out how to stream the OutputStream into the returned DataHandler any other way than to create a temporary file, write to it, then open it back up again as an InputStream. Here's an example:
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create a temporary file to write to
File fTemp = File.createTempFile("my", "tmp");
OutputStream out = new FileOutputStream(fTemp);
// Method takes an output stream and writes to it
writeToOut(out);
out.close();
// Create a data source and data handler based on that temporary file
DataSource ds = new FileDataSource(fTemp);
DataHandler dh = new DataHandler(ds);
return dh;
}
}
The main issue is that the writeToOut() method can return data that are far larger than the computer's memory. That's why the method is using MTOM in the first place - to stream the data. I can't seem to wrap my head around how to stream the data directly from the OutputStream that I need to provide to the returned DataHandler (and ultimately the client, who receives the StreamingDataHandler).
I've tried playing around with PipedInputStream and PipedOutputStream, but those don't seem to be quite what I need, because the DataHandler would need to be returned after the PipedOutputStream is written to.
Any ideas?
I figured out the answer, along the lines that Christian was talking about (creating a new thread to execute writeToOut()):
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create piped output stream, wrap it in a final array so that the
// OutputStream doesn't need to be finalized before sending to new Thread.
PipedOutputStream out = new PipedOutputStream();
InputStream in = new PipedInputStream(out);
final Object[] args = { out };
// Create a new thread which writes to out.
new Thread(
new Runnable(){
public void run() {
writeToOut(args);
((OutputStream)args[0]).close();
}
}
).start();
// Return the InputStream to the client.
DataSource ds = new ByteArrayDataSource(in, "application/octet-stream");
DataHandler dh = new DataHandler(ds);
return dh;
}
}
It is a tad more complex due to final variables, but as far as I can tell this is correct. When the thread is started, it blocks when it first tries to call out.write(); at the same time, the input stream is returned to the client, who unblocks the write by reading the data. (The problem with my previous implementations of this solution was that I wasn't properly closing the stream, and thus running into errors.)
Sorry, I only did this for C# and not java, but I think your method should launch a thread to run "writeToOut(out);" in parralel. You need to create a special stream and pass it to the new thread which gives that stream to writeToOut. After starting the thread you return that stream-object to your caller.
If you only have a method that writes to a stream and returns afterwards and another method that consumes a stream and returns afterwards, there is no other way.
Of coure the tricky part is to get hold of such a -multithreading safe- stream: It shall block each side if an internal buffer is too full.
Don't know if a Java-pipe-stream works for that.
Wrapper pattern ? :-).
Custom javax.activation.DataSource implementation (only 4 methods) to be able to do this ?
return new DataHandler(new DataSource() {
// implement getOutputStream to return the stream used inside writeToOut()
...
});
I don't have the IDE available to test this so i'm only doing a suggestion. I would also need the writeToOut general layout :-).
In my application I use InputStreamDataSource implementation that take InputStream as constructor argument instead of File in FileDataSource. It works so far.
public class InputStreamDataSource implements DataSource {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private final String name;
public InputStreamDataSource(InputStream inputStream, String name) {
this.name = name;
try {
int nRead;
byte[] data = new byte[16384];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public String getContentType() {
return new MimetypesFileTypeMap().getContentType(name);
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(buffer.toByteArray());
}
#Override
public String getName() {
return name;
}
#Override
public OutputStream getOutputStream() throws IOException {
throw new IOException("Read-only data");
}
}