I've got a Java web service in JAX-WS that returns an OutputStream from another method. I can't seem to figure out how to stream the OutputStream into the returned DataHandler any other way than to create a temporary file, write to it, then open it back up again as an InputStream. Here's an example:
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create a temporary file to write to
File fTemp = File.createTempFile("my", "tmp");
OutputStream out = new FileOutputStream(fTemp);
// Method takes an output stream and writes to it
writeToOut(out);
out.close();
// Create a data source and data handler based on that temporary file
DataSource ds = new FileDataSource(fTemp);
DataHandler dh = new DataHandler(ds);
return dh;
}
}
The main issue is that the writeToOut() method can return data that are far larger than the computer's memory. That's why the method is using MTOM in the first place - to stream the data. I can't seem to wrap my head around how to stream the data directly from the OutputStream that I need to provide to the returned DataHandler (and ultimately the client, who receives the StreamingDataHandler).
I've tried playing around with PipedInputStream and PipedOutputStream, but those don't seem to be quite what I need, because the DataHandler would need to be returned after the PipedOutputStream is written to.
Any ideas?
I figured out the answer, along the lines that Christian was talking about (creating a new thread to execute writeToOut()):
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create piped output stream, wrap it in a final array so that the
// OutputStream doesn't need to be finalized before sending to new Thread.
PipedOutputStream out = new PipedOutputStream();
InputStream in = new PipedInputStream(out);
final Object[] args = { out };
// Create a new thread which writes to out.
new Thread(
new Runnable(){
public void run() {
writeToOut(args);
((OutputStream)args[0]).close();
}
}
).start();
// Return the InputStream to the client.
DataSource ds = new ByteArrayDataSource(in, "application/octet-stream");
DataHandler dh = new DataHandler(ds);
return dh;
}
}
It is a tad more complex due to final variables, but as far as I can tell this is correct. When the thread is started, it blocks when it first tries to call out.write(); at the same time, the input stream is returned to the client, who unblocks the write by reading the data. (The problem with my previous implementations of this solution was that I wasn't properly closing the stream, and thus running into errors.)
Sorry, I only did this for C# and not java, but I think your method should launch a thread to run "writeToOut(out);" in parralel. You need to create a special stream and pass it to the new thread which gives that stream to writeToOut. After starting the thread you return that stream-object to your caller.
If you only have a method that writes to a stream and returns afterwards and another method that consumes a stream and returns afterwards, there is no other way.
Of coure the tricky part is to get hold of such a -multithreading safe- stream: It shall block each side if an internal buffer is too full.
Don't know if a Java-pipe-stream works for that.
Wrapper pattern ? :-).
Custom javax.activation.DataSource implementation (only 4 methods) to be able to do this ?
return new DataHandler(new DataSource() {
// implement getOutputStream to return the stream used inside writeToOut()
...
});
I don't have the IDE available to test this so i'm only doing a suggestion. I would also need the writeToOut general layout :-).
In my application I use InputStreamDataSource implementation that take InputStream as constructor argument instead of File in FileDataSource. It works so far.
public class InputStreamDataSource implements DataSource {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private final String name;
public InputStreamDataSource(InputStream inputStream, String name) {
this.name = name;
try {
int nRead;
byte[] data = new byte[16384];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public String getContentType() {
return new MimetypesFileTypeMap().getContentType(name);
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(buffer.toByteArray());
}
#Override
public String getName() {
return name;
}
#Override
public OutputStream getOutputStream() throws IOException {
throw new IOException("Read-only data");
}
}
Related
Got a bit of a problem atm. for my "inapp"-update im downloading the new base64 encoded .apk from my webspace. I have the functionality pretty much down, this is the code without decoding.
public void onResponse(Call call, Response response) throws IOException {
if(response.isSuccessful()){
ResponseBody body = response.body();
BufferedSource source = body.source();
source.request(Long.MAX_VALUE);
Buffer buffer = source.buffer();
String rString = buffer.clone().readString(Charset.forName("UTF-8"));
Log.i("Test: ", AppUtils.decodeBase64(rString));
if(rString.equals("xxx")){
EventBus.getDefault().post(new KeyNotValid());
dispatcher.cancelAll();
}else{
EventBus.getDefault().post(new SaveKey(apikey));
BufferedSink sink = Okio.buffer(Okio.sink(myFile));
sink.writeAll(source);
sink.flush();
sink.close();
}
}
}
The Buffer/Log is not really necessary, just using it to check the response during testing.
How would i go about decoding the bytes before i write them to the sink?
I tried doing it via. ByteString, but i couldn't find a way to write the decoded String back to a BufferedSource.
Most alternatives are pretty slow like reopening the file afterwards, reading the bytes into memory, decode and write them back.
Would really appreciate any help on this
cheers
You can already consume the response as an InputStream via ResponseBody.byteStream. You can decorate this stream with https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html and use it to read a stream of bytes and write it to the Sink for the file in chunks.
I know this answer arrives quite late and that Yuri's answer is technically correct, but I think the most idiomatic way to do that is to take advantage of the composition pattern promoted by Okio to create a Source that decodes from Base64 (or a Sink that encodes to Base64, if you need so).
Here's a little proof of concept (I'm sure it can be improved):
public class Base64Source implements Source {
private Source delegate;
private Base64.Decoder decoder; // Using Java 8 API, but it can be any library
public Base64Source(Source delegate) {
this(delegate, Base64.getDecoder());
}
public Base64Source(Source delegate, Base64.Decoder decoder) {
this.delegate = delegate;
this.decoder = decoder;
}
#Override
public long read(Buffer sink, long byteCount) throws IOException {
Buffer buffer = new Buffer();
long actualRead = this.delegate.read(buffer, byteCount);
if (actualRead == -1) {
return -1;
}
byte[] encoded = buffer.readByteArray(actualRead);
byte[] decoded = decoder.decode(encoded);
sink.write(decoded);
return decoded.length;
}
#Override
public Timeout timeout() {
return this.delegate.timeout();
}
#Override
public void close() throws IOException {
this.delegate.close();
}
}
And here's how it can be used
BufferedSource source = Okio.buffer(new Base64Source(originalSource));
BufferedSink sink = ... // create sink
sink.writeAll(source);
// Don't forget to close the source/sink to flush and free resources
sink.close();
source.close();
I need to build a webservice with Jersey that downloads a big file from another service and returns to the client.
I would like jersey to read some bytes into a buffer and write those bytes to client socket.
I would like it to use non blocking I/O so I dont keep a thread busy. (This could not be achieved)
#GET
#Path("mypath")
public void getFile(final #Suspended AsyncResponse res) {
Client client = ClientBuilder.newClient();
WebTarget t = client.target("http://webserviceURL");
t.request()
.header("some header", "value for header")
.async().get(new InvocationCallback<byte[]>(){
public void completed(byte[] response) {
res.resume(response);
}
public void failed(Throwable throwable) {
res.resume(throwable.getMessage());
throwable.printStackTrace();
//reply with error
}
});
}
So far I have this code and I believe Jersey would download the complete file and then write it to the client which is not what I want to do.
any thoughts??
The client side async request, isn't going to do much for your use case. It's more mean for "fire and forget" use cases. What you can do though is just get the InputStream from the client Response and mix with a server side StreamingResource to stream the results. The server will start sending the data as it is coming in from the other remote resource.
Below is an example. The "/file" endpoint is the dummy remote resource that serves up the file. The "/client" endpoint consumes it.
#Path("stream")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public class ClientStreamingResource {
private static final String INFILE = "Some File";
#GET
#Path("file")
public Response fileEndpoint() {
final File file = new File(INFILE);
final StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) {
try (FileInputStream in = new FileInputStream(file)) {
byte[] buf = new byte[512];
int len;
while ((len = in.read(buf)) != -1) {
out.write(buf, 0, len);
out.flush();
System.out.println("---- wrote 512 bytes file ----");
}
} catch (IOException ex) {
throw new InternalServerErrorException(ex);
}
}
};
return Response.ok(output)
.header(HttpHeaders.CONTENT_LENGTH, file.length())
.build();
}
#GET
#Path("client")
public void clientEndpoint(#Suspended final AsyncResponse asyncResponse) {
final Client client = ClientBuilder.newClient();
final WebTarget target = client.target("http://localhost:8080/stream/file");
final Response clientResponse = target.request().get();
final StreamingOutput output = new StreamingOutput() {
#Override
public void write(OutputStream out) {
try (final InputStream entityStream = clientResponse.readEntity(InputStream.class)) {
byte[] buf = new byte[512];
int len;
while ((len = entityStream.read(buf)) != -1) {
out.write(buf, 0, len);
out.flush();
System.out.println("---- wrote 512 bytes client ----");
}
} catch (IOException ex) {
throw new InternalServerErrorException(ex);
}
}
};
ResponseBuilder responseBuilder = Response.ok(output);
if (clientResponse.getHeaderString("Content-Length") != null) {
responseBuilder.header("Content-Length", clientResponse.getHeaderString("Content-Length"));
}
new Thread(() -> {
asyncResponse.resume(responseBuilder.build());
}).start();
}
}
I used cURL to make the request, and jetty-maven-plugin to be able to run the example from the command line. When you do run it, and make the request, you should see the server logging
---- wrote 512 bytes file ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
---- wrote 512 bytes file ----
---- wrote 512 bytes client ----
...
while cURL client is keeping track of the results
The point to take away from this is that the "remote server" logging is happening the same time as the client resource is logging. This shows that the client doesn't wait to receive the entire file. It starts sending out bytes as soon as it starts receiving them.
Some things to note about the example:
I used a very small buffer size (512) because I was testing with a small (1Mb) file. I really didn't want to wait for a large file for testing. But I would imagine large files should work just the same. Of course you will want to increase the buffer size to something larger.
In order to use the smaller buffer size, you need to set the Jersey property ServerProperties.OUTBOUND_CONTENT_LENGTH_BUFFER to 0. The reason is that Jersey keeps in internal buffer of size 8192, which will cause my 512 byte chunks of data not to flush, until 8192 bytes were buffered. So I just disabled it.
When using AsyncResponse, you should use another thread, as I did. You may want to use executors instead of explicitly creating threads though. If you don't use another thread, then you are still holding up the thread from the container's thread pool.
UPDATE
Instead of managing your own threads/executor, you can annotate the client resource with #ManagedAsync, and let Jersey manage the threads
#ManagedAsync
#GET
#Path("client")
public void clientEndpoint(#Suspended final AsyncResponse asyncResponse) {
...
asyncResponse.resume(responseBuilder.build());
}
I want to stream directly from an Oracle database blobs files via WS with MTOM directly to the WS client.
I thought I found a way which is described here:
http://www.java.net/forum/topic/glassfish/metro-and-jaxb/mtom-best-practices
but after i took a look on InputStreamDataSource and javax.mail.util.ByteArrayDataSource i realized that they acutal hava a byte[] of the 'document' in memory meaning the streaming ideea is in vain, cause what i try to avoid is to have multiple docs in the same time fully in memory.
So how can I stream from DB via WS and MTOM to a WS client ?
Any idea ?
Thanks
Cris
I tried experimenting and finally i had some positive results.
In order to stream from DB directly to clients browser the above
things are valid but the InputStreamDataSource should be like this:
public class InputStreamDataSource implements DataSource {
private InputStream inputStream;
public InputStreamDataSource(InputStream inputStream) {
this.inputStream = inputStream;
}
public InputStream getInputStream() throws IOException {
return inputStream;
}
public OutputStream getOutputStream() throws IOException {
throw new UnsupportedOperationException("Not implemented");
}
public String getContentType() {
return "*/*";
}
public String getName() {
return "InputStreamDataSource";
}
}
What I was affraid is that once I closed the input stream myself...
the ws client did not received the binary content...
Than i check and actually the DataHandler creates a new thread and closes the input stream
I was able to stream 500MB from DB to client fast and with low memory footprint !
I have a utility class I created for my Spring controller to invoke to generate a CSV from a collection of beans using the SuperCSV library ( http://supercsv.sourceforge.net/ )
The utility class is pretty basic:
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename="+filePrefix+"_Data.csv");
OutputStream fout= response.getOutputStream();
OutputStream bos = new BufferedOutputStream(fout);
OutputStreamWriter outputwriter = new OutputStreamWriter(bos);
ICsvBeanWriter writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
}catch (Exception e){
e.printStackTrace();
}
};
The catch is, I'm getting different behaviors out of this operation and I don't know why. When I invoke it from one controller (we'll call it 'A'), I get the expected output of data.
When I invoke it from the other controller ('B'), I get a tiny blurb of unrecognizable binary data that cannot be opened by OO Calc. Opening it in Notepad++ yields an unreadable line of gibberish that I can only assume is an attempt by the reader to show me a binary stream.
Controller 'A' invocation (the one that works)
#RequestMapping(value="/getFullReqData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: ******************************Received request for full Req data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<Requirement> allRecords = reqService.getFullDataSet(projectID);
final String[] header = new String[] {
"ColumnA",
"ColumnB",
"ColumnC",
"ColumnD",
"ColumnE"
};
CSVExporter.export2CSV(response, header, projName+"_reqs_", allRecords);
};
...and here's the Controller 'B' invocation (the one that fails):
#RequestMapping(value="/getFullTCData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: Received request for full TCD data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<TestCase> allRecords = testService.getFullTestCaseList(projectID);
final String[] header = new String[] {
"ColumnW",
"ColumnX",
"ColumnY",
"ColumnZ"
};
CSVExporter.export2CSV(response, header, projName+"_tcs_", allRecords);
}
Observations:
Which controller I invoke first is irrelevant. 'A' always works and 'B' always produces gibberish
Both calls to this function have a list of header columns that are a subset of the total set of operations defined in the bean being passed in to CSVWriter
The simple Exception printStackTrace is working to detect when a bean's reflection field doesn't match the definition (i.e., can't find get() to get the value programmatically) suggesting that all column/variable matchups are succeeding.
In the debugger, I've verified the writer.write(Object, header) call is being hit the expected number of times based on the number of objects being passed and that these objects have the expected data
Any suggestions or insights would be greatly appreciated. I'm really stumped how to better isolate the issue...
You aren't closing the writer. Also, CsvBeanWriter will wrap the writer in a BufferedWriter, so you can probably simplify your outputwriter as well.
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
ICsvBeanWriter writer;
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition",
"attachment; filename="+filePrefix+"_Data.csv");
OutputStreamWriter outputwriter =
new OutputStreamWriter(response.getOutputStream());
writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
writer.close(); // closes writer and underlying stream
} catch (Exception e){}
}
};
Super CSV 2.0.0-beta-1 is out now! As well as adding numerous other features (including Maven support and a new Dozer extension), CSV writers now expose a flush() method as well.
I've serving a files from Android assets via Netty server (images, html).
Text files such a html is saved as .mp3 to disable compression (I need an InputStream!)
My pipeline is looking like this:
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("aggregator", new HttpChunkAggregator(65536));
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("chunkedWriter", new ChunkedWriteHandler());
pipeline.addLast("handler", new AssetsServerHandler(context));
My handler is:
public class AssetsServerHandler extends SimpleChannelUpstreamHandler {
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) {
// some checks
final FileInputStream is;
final AssetFileDescriptor afd;
try {
afd = assetManager.openFd(path);
is = afd.createInputStream();
} catch(IOException exc) {
sendError(ctx, NOT_FOUND);
return;
}
final long fileLength = afd.getLength();
HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
setContentLength(response, fileLength);
final Channel ch = e.getChannel();
final ChannelFuture future;
ch.write(response);
future = ch.write(new ChunkedStream(is));
future.addListener(new ChannelFutureListener() {
#Override
public void operationComplete(ChannelFuture future) throws Exception {
future.getChannel().close();
}
});
if (!isKeepAlive(request)) {
future.addListener(ChannelFutureListener.CLOSE);
}
}
// other stuff
}
With that handler i've got my resposes truncated by at least one byte. If I change ChunkedStream to ChunkedNioFile (and so use a is.getChannel() instead of is as a constructor to it) - everything works perfectly.
Please, help me understand what is wrong with ChunkedStream.
Your code looks right to me. Does the returned FileInputStream of AssetFileDescriptor contain "all the bytes" ? You could check this with a unit test. If there is no bug in it then its a bug in netty. I make heavy use of ChunkInputStream and never had such a problem yet, but maybe it really depends on the nature of the InputStream.
Would be nice if you could write a test case and open a issue at netty's github.