I noticed that the following code will use the same OutputStream object (same hash code on debug) if a curl request is ctrl+c, then re-run. Even with different parameters, it uses the same OutputStream. This causes very odd output obviously.
The original output stream starts to throw NullPointerException when writing to the OutputStream because the underlying HttpOutputStream no longer exists (broken pipe). This is why I think it is odd that a subsequent request would reuse the same OutputStream object.
Closing the output stream in a finally block fixes the issue. Some examples I've seen around the web don't explicitly close the output stream. Is the reuse of the OutputStream expected? Does anyone have any ideas on why I'd be seeing this behavior?
#RequestMapping(value= URI_ROOT, method= RequestMethod.POST, produces = {"text/event-stream"})
public StreamingResponseBody methodName(
... params
) {
return new StreamingResponseBody() {
#Override
public void writeTo(OutputStream outputStream) {
try {
... code ...
} catch (Exception e) {
LOGGER.info("Migration thread interrupted.",e);
} finally {
IoUtil.closeSilently(outputStream); // This fixes it.
}
}
};
}
Related
I am trying to read, in an async way, a file that i receive from the client. The idea is to recipt the file, validate it and if the validations are ok, send a response to the client saying that all it's ok, and process the file in background, so the user doesn't need to wait until the file is procesed.
For that I receive the file in my resource like an inputStrem:
#Override
#POST
#Path("/bulk")
#Consumes("text/csv")
#Produces(MediaType.APPLICATION_JSON)
public Response importEmployees(InputStream inputStream) {
if(fileIsNotValid(inputStream)){
throw exceptionFactoryBean.createBadRequestException("there was an error with the file");
}
try {
CompletableFuture.runAsync(() -> {
employeeService.importEmployees(inputStream);
}).exceptionally(e -> {
LOG.error(format(ERROR_IMPORTING_FILE, e.getMessage()));
return null;
});
} catch (RuntimeException e) {
LOG.error(format(ERROR_SENDING_EMAIL, e.getMessage()));
throw exceptionFactoryBean.createServiceException("payment-method.export.installment-schema.error");
}
return Response.ok().build();
}
For the async part I used the runAsync() method of CompletableFuture.
However, inside of my employeeService.importEmployees() method I tried to read the inputStream and I am getting a java.util.concurrent.CompletionException: java.lang.NullPointerException
public List<ImportResult> importEmployees(final InputStream inputStream) {
byte[] buffer = new byte[1024];
int len;
try {
while ((len = inputStream.read(buffer)) > -1) {
baos.write(buffer, NumberUtils.INTEGER_ZERO, len);
}
The inputStream is not null. And debuging in a low level, I can see that the wrapper of the class Http11InputBuffer is null when i try to read the inputStream.
Do you can see what errors i have or how i can set the wrapper attribute of the Http11InputBuffer previous to read the inputStream
You are not waiting for the result. So it will return the Responsebefore importEmployees is executed. You need to wait with a join/get before returning the response:
public Response importEmployees(InputStream inputStream) {
...
CompletableFuture.runAsync(() -> { ... }).get();
...
return Response.ok().build();
}
There is probably no point in making this code reactive, though.
I want to stream directly from an Oracle database blobs files via WS with MTOM directly to the WS client.
I thought I found a way which is described here:
http://www.java.net/forum/topic/glassfish/metro-and-jaxb/mtom-best-practices
but after i took a look on InputStreamDataSource and javax.mail.util.ByteArrayDataSource i realized that they acutal hava a byte[] of the 'document' in memory meaning the streaming ideea is in vain, cause what i try to avoid is to have multiple docs in the same time fully in memory.
So how can I stream from DB via WS and MTOM to a WS client ?
Any idea ?
Thanks
Cris
I tried experimenting and finally i had some positive results.
In order to stream from DB directly to clients browser the above
things are valid but the InputStreamDataSource should be like this:
public class InputStreamDataSource implements DataSource {
private InputStream inputStream;
public InputStreamDataSource(InputStream inputStream) {
this.inputStream = inputStream;
}
public InputStream getInputStream() throws IOException {
return inputStream;
}
public OutputStream getOutputStream() throws IOException {
throw new UnsupportedOperationException("Not implemented");
}
public String getContentType() {
return "*/*";
}
public String getName() {
return "InputStreamDataSource";
}
}
What I was affraid is that once I closed the input stream myself...
the ws client did not received the binary content...
Than i check and actually the DataHandler creates a new thread and closes the input stream
I was able to stream 500MB from DB to client fast and with low memory footprint !
I have a utility class I created for my Spring controller to invoke to generate a CSV from a collection of beans using the SuperCSV library ( http://supercsv.sourceforge.net/ )
The utility class is pretty basic:
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition","attachment; filename="+filePrefix+"_Data.csv");
OutputStream fout= response.getOutputStream();
OutputStream bos = new BufferedOutputStream(fout);
OutputStreamWriter outputwriter = new OutputStreamWriter(bos);
ICsvBeanWriter writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
}catch (Exception e){
e.printStackTrace();
}
};
The catch is, I'm getting different behaviors out of this operation and I don't know why. When I invoke it from one controller (we'll call it 'A'), I get the expected output of data.
When I invoke it from the other controller ('B'), I get a tiny blurb of unrecognizable binary data that cannot be opened by OO Calc. Opening it in Notepad++ yields an unreadable line of gibberish that I can only assume is an attempt by the reader to show me a binary stream.
Controller 'A' invocation (the one that works)
#RequestMapping(value="/getFullReqData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: ******************************Received request for full Req data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<Requirement> allRecords = reqService.getFullDataSet(projectID);
final String[] header = new String[] {
"ColumnA",
"ColumnB",
"ColumnC",
"ColumnD",
"ColumnE"
};
CSVExporter.export2CSV(response, header, projName+"_reqs_", allRecords);
};
...and here's the Controller 'B' invocation (the one that fails):
#RequestMapping(value="/getFullTCData.html", method = RequestMethod.GET)
public void getFullData(HttpSession session, HttpServletRequest request, HttpServletResponse response) throws IOException{
logger.info("INFO: Received request for full TCD data dump");
String projName= (String)session.getAttribute("currentProject");
int projectID = ProjectService.getProjectID(projName);
List<TestCase> allRecords = testService.getFullTestCaseList(projectID);
final String[] header = new String[] {
"ColumnW",
"ColumnX",
"ColumnY",
"ColumnZ"
};
CSVExporter.export2CSV(response, header, projName+"_tcs_", allRecords);
}
Observations:
Which controller I invoke first is irrelevant. 'A' always works and 'B' always produces gibberish
Both calls to this function have a list of header columns that are a subset of the total set of operations defined in the bean being passed in to CSVWriter
The simple Exception printStackTrace is working to detect when a bean's reflection field doesn't match the definition (i.e., can't find get() to get the value programmatically) suggesting that all column/variable matchups are succeeding.
In the debugger, I've verified the writer.write(Object, header) call is being hit the expected number of times based on the number of objects being passed and that these objects have the expected data
Any suggestions or insights would be greatly appreciated. I'm really stumped how to better isolate the issue...
You aren't closing the writer. Also, CsvBeanWriter will wrap the writer in a BufferedWriter, so you can probably simplify your outputwriter as well.
public static void export2CSV(HttpServletResponse response,
String[] header, String filePrefix, List<? extends Object> dataObjs) {
ICsvBeanWriter writer;
try{
response.setContentType("text/csv;charset=utf-8");
response.setHeader("Content-Disposition",
"attachment; filename="+filePrefix+"_Data.csv");
OutputStreamWriter outputwriter =
new OutputStreamWriter(response.getOutputStream());
writer = new CsvBeanWriter(outputwriter, CsvPreference.EXCEL_PREFERENCE);
// the actual writing
writer.writeHeader(header);
for(Object anObj : dataObjs){
writer.write(anObj, header);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
writer.close(); // closes writer and underlying stream
} catch (Exception e){}
}
};
Super CSV 2.0.0-beta-1 is out now! As well as adding numerous other features (including Maven support and a new Dozer extension), CSV writers now expose a flush() method as well.
I am doing conversion to image and PDF output. I need an input HTML document that is generated by our application JSPs. Essentially, I need to render the final output product of a JSP based application to a String or memory and then use that string for other processing.
What are some ways that I can just invoke the JSP renderer to get the final HTML content that is normally output to the user?
Ideally, I am looking for something that will work for multiple application servers like websphere. But something that is Tomcat specific will also work.
There are a couple of other different approaches, but I think rendering the JSP (which may include sub JSPs) is the best approach.
Optional Paths that I would rather stay away from.
I could perform a network request to the page using the Socket APIs and then read the final output that is rendered from that particular page. This is probably the next best option, but we work on multiple servers and JVMs, targeting the page I need would be complicated.
Use a filter to get that final page output. This Ok but I have always had problems with filters and illegalstateexceptions. It never seems to work 100% the way I need to.
It seems like this should be simple. The JSP compiler is essentially just a library for parsing an input JSP document and subdocuments and then output some HTML content. I would like to invoke that process through Java code. On the server and possibly as a standalone console application.
This is a downright irritating problem, one I've had to handle a few times and one I've never found a satisfactory solution to.
The basic problem is that the servlet API is of no help here, so you have to trick it. My solution is to write a subclass of HttpServletResponseWrapper which override the getWriter() and getOutput() methods and captures the data into a buffer. You then forward() your request to the URI of the JSP you want to capture, substituting your wrapper response for the original response. You then extract the data from the buffer, manipulate it, and write the end result back to the original response.
Here's my code that does this:
public class CapturingResponseWrapper extends HttpServletResponseWrapper {
private final OutputStream buffer;
private PrintWriter writer;
private ServletOutputStream outputStream;
public CapturingResponseWrapper(HttpServletResponse response, OutputStream buffer) {
super(response);
this.buffer = buffer;
}
#Override
public ServletOutputStream getOutputStream() {
if (outputStream == null) {
outputStream = new DelegatingServletOutputStream(buffer);
}
return outputStream;
}
#Override
public PrintWriter getWriter() {
if (writer == null) {
writer = new PrintWriter(buffer);
}
return writer;
}
#Override
public void flushBuffer() throws IOException {
if (writer != null) {
writer.flush();
}
if (outputStream != null) {
outputStream.flush();
}
}
}
The code to use it can be something like this:
HttpServletRequest originalRequest = ...
HttpServletResponse originalResponse = ...
ByteArrayOutputStream bufferStream = new ByteArrayOutputStream();
CapturingResponseWrapper responseWrapper = new CapturingResponseWrapper(originalResponse, bufferStream);
originalRequest.getRequestDispatcher("/my.jsp").forward(originalRequest, responseWrapper);
responseWrapper.flushBuffer();
byte[] buffer = bufferStream.toByteArray();
// now use the data
It's very ugly, but it's the best solution I've found. In case you're wondering, the wrapper response has to contain the original response because the servlet spec says that you cannot substitute a completely different request or response object when you forward, you have to use the originals, or wrapped versions of them.
I've got a Java web service in JAX-WS that returns an OutputStream from another method. I can't seem to figure out how to stream the OutputStream into the returned DataHandler any other way than to create a temporary file, write to it, then open it back up again as an InputStream. Here's an example:
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create a temporary file to write to
File fTemp = File.createTempFile("my", "tmp");
OutputStream out = new FileOutputStream(fTemp);
// Method takes an output stream and writes to it
writeToOut(out);
out.close();
// Create a data source and data handler based on that temporary file
DataSource ds = new FileDataSource(fTemp);
DataHandler dh = new DataHandler(ds);
return dh;
}
}
The main issue is that the writeToOut() method can return data that are far larger than the computer's memory. That's why the method is using MTOM in the first place - to stream the data. I can't seem to wrap my head around how to stream the data directly from the OutputStream that I need to provide to the returned DataHandler (and ultimately the client, who receives the StreamingDataHandler).
I've tried playing around with PipedInputStream and PipedOutputStream, but those don't seem to be quite what I need, because the DataHandler would need to be returned after the PipedOutputStream is written to.
Any ideas?
I figured out the answer, along the lines that Christian was talking about (creating a new thread to execute writeToOut()):
#MTOM
#WebService
class Example {
#WebMethod
public #XmlMimeType("application/octet-stream") DataHandler service() {
// Create piped output stream, wrap it in a final array so that the
// OutputStream doesn't need to be finalized before sending to new Thread.
PipedOutputStream out = new PipedOutputStream();
InputStream in = new PipedInputStream(out);
final Object[] args = { out };
// Create a new thread which writes to out.
new Thread(
new Runnable(){
public void run() {
writeToOut(args);
((OutputStream)args[0]).close();
}
}
).start();
// Return the InputStream to the client.
DataSource ds = new ByteArrayDataSource(in, "application/octet-stream");
DataHandler dh = new DataHandler(ds);
return dh;
}
}
It is a tad more complex due to final variables, but as far as I can tell this is correct. When the thread is started, it blocks when it first tries to call out.write(); at the same time, the input stream is returned to the client, who unblocks the write by reading the data. (The problem with my previous implementations of this solution was that I wasn't properly closing the stream, and thus running into errors.)
Sorry, I only did this for C# and not java, but I think your method should launch a thread to run "writeToOut(out);" in parralel. You need to create a special stream and pass it to the new thread which gives that stream to writeToOut. After starting the thread you return that stream-object to your caller.
If you only have a method that writes to a stream and returns afterwards and another method that consumes a stream and returns afterwards, there is no other way.
Of coure the tricky part is to get hold of such a -multithreading safe- stream: It shall block each side if an internal buffer is too full.
Don't know if a Java-pipe-stream works for that.
Wrapper pattern ? :-).
Custom javax.activation.DataSource implementation (only 4 methods) to be able to do this ?
return new DataHandler(new DataSource() {
// implement getOutputStream to return the stream used inside writeToOut()
...
});
I don't have the IDE available to test this so i'm only doing a suggestion. I would also need the writeToOut general layout :-).
In my application I use InputStreamDataSource implementation that take InputStream as constructor argument instead of File in FileDataSource. It works so far.
public class InputStreamDataSource implements DataSource {
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
private final String name;
public InputStreamDataSource(InputStream inputStream, String name) {
this.name = name;
try {
int nRead;
byte[] data = new byte[16384];
while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public String getContentType() {
return new MimetypesFileTypeMap().getContentType(name);
}
#Override
public InputStream getInputStream() throws IOException {
return new ByteArrayInputStream(buffer.toByteArray());
}
#Override
public String getName() {
return name;
}
#Override
public OutputStream getOutputStream() throws IOException {
throw new IOException("Read-only data");
}
}