I've written a basic REST Server using Jersey2 on top of Jetty, to test out HTTP Chunked Transfer-Encoding, and gzip Content-Encoding. However I've found that the recommended method of implementing a WriterInceptor to apply a GZIPOutputStream for gzip encoding results in the server blocking instead of sending through a gzip'd chunk.
I believe it is the GZIPOutputStream waiting for it's own buffer to fill up, so I tried overriding write() method in the WriterInterceptor to force a flush() after every write (as my server always writes one chunk at a time) but that made no difference. Is there a way of forcing the flush to occur whenever a write occurs?
App.java
public class App
{
public static int lineCount=0;
public static void main( String[] args ) {
System.out.println( "Hello World!" );
ResourceConfig config = new ResourceConfig();
config.packages("com.example.mockAPIjava");
ServletHolder servlet = new ServletHolder(new ServletContainer(config));
EncodingFilter.enableFor(config, GZipEncoder.class);
Server server = new Server(2222);
ServletContextHandler context = new ServletContextHandler(server, "/*");
context.addServlet(servlet, "/*");
try {
server.start();
server.join();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
server.destroy();
}
}
}
GZIPWriterInterceptor.java
#Provider
#Compress
public class GZIPWriterInterceptor implements WriterInterceptor {
#Override
public void aroundWriteTo(WriterInterceptorContext context)
throws IOException, WebApplicationException {
MultivaluedMap<String,Object> headers = context.getHeaders();
headers.add("Content-Encoding", "gzip");
final OutputStream outputStream = context.getOutputStream();
context.setOutputStream(new GZIPOutputStream(outputStream) {
#Override
public void write(final int b) throws IOException {
out.write(b);
out.flush();
}
#Override
public void write(final byte[] b) throws IOException {
out.write(b);
out.flush();
}
#Override
public void write(final byte[] b, final int off, final int len) throws IOException {
out.write(b, off, len);
out.flush();
}
});
context.proceed();
}
}
Resource.java
#Path("stream")
public class Resource {
#GET
#Path("test")
#Compress
#Produces(MediaType.APPLICATION_JSON)
public ChunkedOutput<String> helloWorld(#Context HttpHeaders header, #Context HttpServletResponse response) {
final ChunkedOutput<String> output = new ChunkedOutput<String>(String.class, "\r\n");
new Thread() {
public void run() {
BufferedReader br = null;
try {
String chunk;
// open file for reading
File file = new File("/tmp/stream.txt");
FileReader fr = new FileReader(file);
br = new BufferedReader(fr);
while ((chunk = getNextString(br)) != null) {
// write a chunk every second
output.write(chunk);
try {
Thread.sleep(1 * 1000);
} catch(InterruptedException ex) {
Thread.currentThread().interrupt();
}
}
} catch (Exception e) {
// IOException thrown when writing the
// chunks of response: should be handled
e.printStackTrace();
} finally {
try {
output.close();
// simplified: IOException thrown from
// this close() should be handled here...
if (br!=null) { br.close(); }
} catch (IOException e1){
e1.printStackTrace();
}
}
}
}.start();
// the output will be probably returned even before
// a first chunk is written by the new thread
return output;
}
private String getNextString(BufferedReader br) throws IOException, ParseException {
App.lineCount++;
return br.readLine();;
}
}
Compress.java
//#Compress annotation is the name binding annotation for the GZIPWriterInterceptor
#NameBinding
#Retention(RetentionPolicy.RUNTIME)
public #interface Compress {}
By overriding the write methods of GZIPOutputStream, you have just stopped it from gzipping!
public void write(final int b) throws IOException {
out.write(b);
out.flush();
}
Because you've overridden it to not invoke super.write (which you should have done), but rather out.write, you're sending directly to the context OutputStream, uncompressed.
Presumably, the receiving side is expecting gzip data and not receiving it, which may lead to all kinds of wrong behaviour.
Change the code to invoke super.write and flush:
public void write(final int b) throws IOException {
super.write(b);
flush();
}
etc.
Related
I check requests from the client and responses from the server with Charles. It sends the files (like CSS or js) fine, but doesn't with images: though the status is "200 OK" the transfer-encoding is chunked ... I have no idea what that means!? Does anybody know?
public class Server {
public static void main(String[] args) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress(8000), 0);
server.createContext("/guestBook", new MyHandler());
server.setExecutor(null);
server.start();
}
static class MyHandler implements HttpHandler {
#Override
public void handle(HttpExchange t) throws IOException {
File file = new File("/Users/rahul.joshi/Flower-Catalog/src/main/java/htmlpages/index.html");
t.sendResponseHeaders(200, file.length());
try (OutputStream os = t.getResponseBody()) {
Files.copy(file.toPath(), os);
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
}
I have a rest method for downloading files which works. But, it seems that the download doesn't start on the web client until the file is completely copied to the output stream, which can take a while for large files.
#GetMapping(value = "download-single-report")
public void downloadSingleReport(HttpServletResponse response) {
File dlFile = new File("some_path");
try {
response.setContentType("application/pdf");
response.setHeader("Content-disposition", "attachment; filename="+ dlFile.getName());
InputStream inputStream = new FileInputStream(dlFile);
IOUtils.copy(inputStream, response.getOutputStream());
response.flushBuffer();
} catch (FileNotFoundException e) {
// error
} catch (IOException e) {
// error
}
}
Is there a way to "stream" the file such that the download starts as soon as I begin writing to the output stream?
I also have a similar method that takes multiple files and puts them in a zip, adding each zip entry to the zip stream, and the download also only begins after the zip has been created:
ZipEntry zipEntry = new ZipEntry(entryName);
zipOutStream.putNextEntry(zipEntry);
IOUtils.copy(fileStream, zipOutStream);
You can use InputStreamResource to return stream result. I tested and it is started copying to output immediately.
#GetMapping(value = "download-single-report")
public ResponseEntity<Resource> downloadSingleReport() {
File dlFile = new File("some_path");
if (!dlFile.exists()) {
return ResponseEntity.notFound().build();
}
try {
try (InputStream stream = new FileInputStream(dlFile)) {
InputStreamResource streamResource = new InputStreamResource(stream);
return ResponseEntity.ok()
.contentType(MediaType.APPLICATION_PDF)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + dlFile.getName() + "\"")
.body(streamResource);
}
/*
// FileSystemResource alternative
FileSystemResource fileSystemResource = new FileSystemResource(dlFile);
return ResponseEntity.ok()
.contentType(MediaType.APPLICATION_PDF)
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + dlFile.getName() + "\"")
.body(fileSystemResource);
*/
} catch (IOException e) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
}
}
The second alternative is a partial download method.
#GetMapping(value = "download-single-report-partial")
public void downloadSingleReportPartial(HttpServletRequest request, HttpServletResponse response) {
File dlFile = new File("some_path");
if (!dlFile.exists()) {
response.setStatus(HttpStatus.NOT_FOUND.value());
return;
}
try {
writeRangeResource(request, response, dlFile);
} catch (Exception ex) {
response.setStatus(HttpStatus.INTERNAL_SERVER_ERROR.value());
}
}
public static void writeRangeResource(HttpServletRequest request, HttpServletResponse response, File file) throws IOException {
String range = request.getHeader("Range");
if (StringUtils.hasLength(range)) {
//http
ResourceRegion region = getResourceRegion(file, range);
long start = region.getPosition();
long end = start + region.getCount() - 1;
long resourceLength = region.getResource().contentLength();
end = Math.min(end, resourceLength - 1);
long rangeLength = end - start + 1;
response.setStatus(206);
response.addHeader("Accept-Ranges", "bytes");
response.addHeader("Content-Range", String.format("bytes %s-%s/%s", start, end, resourceLength));
response.setContentLengthLong(rangeLength);
try (OutputStream outputStream = response.getOutputStream()) {
try (InputStream inputStream = new BufferedInputStream(new FileInputStream(file))) {
StreamUtils.copyRange(inputStream, outputStream, start, end);
}
}
} else {
response.setStatus(200);
response.addHeader("Accept-Ranges", "bytes");
response.setContentLengthLong(file.length());
try (OutputStream outputStream = response.getOutputStream()) {
try (InputStream inputStream = new BufferedInputStream(new FileInputStream(file))) {
StreamUtils.copy(inputStream, outputStream);
}
}
}
}
private static ResourceRegion getResourceRegion(File file, String range) {
List<HttpRange> httpRanges = HttpRange.parseRanges(range);
if (httpRanges.isEmpty()) {
return new ResourceRegion(new FileSystemResource(file), 0, file.length());
}
return httpRanges.get(0).toResourceRegion(new FileSystemResource(file));
}
Spring Framework Resource Response Process
Resource response managed by ResourceHttpMessageConverter class. In writeContent method, StreamUtils.copy is called.
package org.springframework.http.converter;
public class ResourceHttpMessageConverter extends AbstractHttpMessageConverter<Resource> {
..
protected void writeContent(Resource resource, HttpOutputMessage outputMessage)
throws IOException, HttpMessageNotWritableException {
try {
InputStream in = resource.getInputStream();
try {
StreamUtils.copy(in, outputMessage.getBody());
}
catch (NullPointerException ex) {
// ignore, see SPR-13620
}
finally {
try {
in.close();
}
catch (Throwable ex) {
// ignore, see SPR-12999
}
}
}
catch (FileNotFoundException ex) {
// ignore, see SPR-12999
}
}
}
out.write(buffer, 0, bytesRead); sends data immediately to output (I have tested on my local machine). When whole data is transferred, out.flush(); is called.
package org.springframework.util;
public abstract class StreamUtils {
..
public static int copy(InputStream in, OutputStream out) throws IOException {
Assert.notNull(in, "No InputStream specified");
Assert.notNull(out, "No OutputStream specified");
int byteCount = 0;
int bytesRead;
for(byte[] buffer = new byte[4096]; (bytesRead = in.read(buffer)) != -1; byteCount += bytesRead) {
out.write(buffer, 0, bytesRead);
}
out.flush();
return byteCount;
}
}
Use
IOUtils.copyLarge(InputStream input, OutputStream output)
Copy bytes from a large (over 2GB) InputStream to an OutputStream.
This method buffers the input internally, so there is no need to use a BufferedInputStream.
The buffer size is given by DEFAULT_BUFFER_SIZE.
or
IOUtils.copyLarge(InputStream input, OutputStream output, byte[] buffer)
Copy bytes from a large (over 2GB) InputStream to an OutputStream.
This method uses the provided buffer, so there is no need to use a BufferedInputStream.
http://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/IOUtils.html
You can use "StreamingResponseBody" File download would start immediately while the chunks are written to the output stream. Below is the code snippet
#GetMapping (value = "/download-single-report")
public ResponseEntity<StreamingResponseBody> downloadSingleReport(final HttpServletResponse response) {
final File dlFile = new File("Sample.pdf");
response.setContentType("application/pdf");
response.setHeader(
"Content-Disposition",
"attachment;filename="+ dlFile.getName());
StreamingResponseBody stream = out -> FileCopyUtils.copy(new FileInputStream(dlFile), out);
return new ResponseEntity(stream, HttpStatus.OK);
}
I am trying to log all the outgoing Http requests in my spring based web application. Is there is interceptor for this purpose? I want to log all outgoing the contents and headers before it leaves the application. I am using spring-ws to send SOAP requests. So basically, I want to log not only the SOAP request xml (as mentioned here How can I make Spring WebServices log all SOAP requests?) but the http request as a whole.
Intercept the request/response using a ClientInterceptor on the WebServiceGatewaySupport:
// soapClient extends WebServiceGatewaySupport
soapClient.setInterceptors(new ClientInterceptor[]{new ClientInterceptor() {
#Override
public boolean handleRequest(MessageContext messageContext) throws WebServiceClientException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
try {
messageContext.getRequest().writeTo(os);
} catch (IOException e) {
throw new WebServiceIOException(e.getMessage(), e);
}
String request = new String(os.toByteArray());
logger.trace("Request Envelope: " + request);
return true;
}
#Override
public boolean handleResponse(MessageContext messageContext) throws WebServiceClientException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
try {
messageContext.getResponse().writeTo(os);
} catch (IOException e) {
throw new WebServiceIOException(e.getMessage(), e);
}
String response = new String(os.toByteArray());
logger.trace("Response Envelope: " + response);
return true;
}
...
To get the headers as well you need an instance of TransportOutputStream.
Unfortunately the class is abstract, so you need to subclass is. Here's how it might look:
class ByteArrayTransportOutputStream extends TransportOutputStream {
private ByteArrayOutputStream outputStream;
#Override
public void addHeader(String name, String value) throws IOException {
createOutputStream();
String header = name + ": " + value + "\n";
outputStream.write(header.getBytes());
}
public byte[] toByteArray() {
return outputStream.toByteArray();
}
#Override
protected OutputStream createOutputStream() throws IOException {
if (outputStream == null) {
outputStream = new ByteArrayOutputStream();
}
return outputStream;
}
}
I have an app to upload files to a cloud service using HttpURLConnection:
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setDoOutput(true);
httpCon.setRequestMethod("PUT");
OutputStreamWriter out = new OutputStreamWriter(
httpCon.getOutputStream());
FileInputStream fis = new FileInputStream("C:/Users/me/Pictures/_MG_9324.JPG");
IOUtils.copy(fis,httpCon.getOutputStream());
out.close();
InputStream is= httpCon.getInputStream();
IOUtils.copy(is,System.out);
The program didn't block at the first IOUtils.copy, instead, it blocks at the second IOUtils.copy. My understanding is that the program blocks since the upload didn't complete. So my questions during the upload, how do I monitor how much data is uploaded from client side?
The output is written to a ByteArrayOutputStream until you call getInputStream() or getResponseCode(), so that the Content-length header can be set correctly. So the second copy operation includes the upload. To avoid that, use chunked transfer mode.
Don't wrap your OutputStream in an OutputStreamWriter, instead wrap your OutputStream in the below class. You can pass in a callback function that'll listen for the bytes being written.
public class CallbackEnabledOutputStream extends OutputStream {
private OutputStreamListener outputStreamListener;
private OutputStream realOutputStream;
private int byteWriteThreshold;
private int bytesWritten = 0;
public CallbackEnabledOutputStream(OutputStreamListener outputStreamListener, OutputStream realOutputStream, int byteWriteThreshold) {
super();
this.outputStreamListener = outputStreamListener;
this.realOutputStream = realOutputStream;
this.byteWriteThreshold = byteWriteThreshold;
}
#Override
public void write(int b) throws IOException {
realOutputStream.write(b);
bytesWritten++;
if(bytesWritten > byteWriteThreshold) {
outputStreamListener.bytesWritten(bytesWritten);
bytesWritten = 0;
}
}
#Override
public void flush() throws IOException {
realOutputStream.flush();
}
#Override
public void close() throws IOException {
realOutputStream.flush();
}
}
OutputStreamListener
public class OutputStreamListener {
private int byteCount = 0;
public void bytesWritten(int byteCount) {
this.byteCount += byteCount;
}
public int getByteCount() {
return byteCount;
}
}
I'm curious if there is an annotation/filter/interceptor capability in JAX-RS to detect if PUT or POST method contains an empty body.
Currently I have method that, if request has empty body, possibly throws NPE.
#PUT
#Produces("application/json")
#Consumes("application/json")
#Path("/update/{id}")
public Response updateCustomer(#PathParam("id") final String customerIdStr, final CustomerJson customer) {
// if request body is empty -> customer == null
return Response.ok().build();
}
I can check customer for null . But since I have plenty of such methods, it's better to have filter to do such validation.
Please!
Did you try to use Bean Validation, using an #NotNull annotation on your CustomerJson method parameter ?
Interceptors read the HTTP body and I dont find a way to send the body for further processing. But you can do this by Servlet Filter and HTTP servlet request wrapper,
public class EmptyCheckFilter implements javax.servlet.Filter {
#Override
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException, ServletException {
HttpServletRequest req = (HttpServletRequest) request;
HttpServletResponse res = (HttpServletResponse) response;
if (req.getMethod().equals("POST") || req.getMethod().equals("PUT")) {
boolean dirty = false;
HttpRequestWrapper wrapper = new MyHTTPRequestWrapper(req);
try {
// check body is empty by wrapper.getBody() and set dirty = true;
} catch (Exception e) {
}
if (dirty) {
res.sendError(400, "Invalid input");
} else
chain.doFilter(wrapper, response);
} else
chain.doFilter(request, response);
}
#Override
public void destroy() {
}
#Override
public void init(FilterConfig arg0) throws ServletException {
}
}
public class MyHTTPRequestWrapper extends HttpServletRequestWrapper {
private final String body;
public MyHTTPRequestWrapper(HttpServletRequest request) throws IOException {
super(request);
StringBuilder stringBuilder = new StringBuilder();
BufferedReader bufferedReader = null;
try {
InputStream inputStream = request.getInputStream();
if (inputStream != null) {
if (request.getCharacterEncoding() != null)
bufferedReader = new BufferedReader(new InputStreamReader(inputStream, request.getCharacterEncoding()));
else
bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
char[] charBuffer = new char[128];
int bytesRead = -1;
while ((bytesRead = bufferedReader.read(charBuffer)) > 0) {
stringBuilder.append(charBuffer, 0, bytesRead);
}
} else {
stringBuilder.append("");
}
} catch (IOException ex) {
throw ex;
} finally {
if (bufferedReader != null) {
try {
bufferedReader.close();
} catch (IOException ex) {
throw ex;
}
}
}
body = stringBuilder.toString();
}
#Override
public ServletInputStream getInputStream() throws IOException {
final ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(
body.getBytes());
ServletInputStream servletInputStream = new ServletInputStream() {
public int read() throws IOException {
return byteArrayInputStream.read();
}
};
return servletInputStream;
}
#Override
public BufferedReader getReader() throws IOException {
return new BufferedReader(new InputStreamReader(this.getInputStream()));
}
public String getBody() {
return this.body;
}
}