We have a VPN that we sometimes use to connect to sites remotely only to find that the chunked encoding tomcat seems to use often ends up with a lot of JavaScript files being truncated. In an attempt to remedy this I wanted to build a Tomcat filter for JS files that sends them in one shot.
By reading through the articles here and here I tried to take my own shot at this and tried implementing it as follows.
I hooked the filter into my context with a custom filter map of
new CustomFilterMap(
new String[]{"*.js"},
new String[]{"REQUEST"}
)
This part has actually worked well and my filter seems to be applied to JS files. Now the part I haven't gotten working so well is the actual filter.
public class NoChunkEncodedJSFilter extends OncePerRequestFilter {
#Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
val unversionedServletPath = request.getRequestURI().substring(request.getRequestURI().indexOf("/js"));
val requestFilePath = getServletContext().getRealPath(unversionedServletPath);
if (requestFilePath != null) {
File requestedFile = new File(requestFilePath);
if (requestedFile.exists()) {
val fileSize = Math.toIntExact(requestedFile.length());
val out = response.getWriter();
val wrappedResponse = new NonEncodedResponse(response, fileSize);
filterChain.doFilter(request, wrappedResponse);
out.write(wrappedResponse.toString(), "UTF-8");
out.close();
return;
}
}
filterChain.doFilter(request, response);
}
}
With my NonEncodedResponse:
public class NonEncodedResponse extends HttpServletResponseWrapper {
private ByteArrayOutputStream baos;
public String toString() {
try {
return baos.toString("UTF-8");
} catch (UnsupportedEncodingException unsuportedEncodingException) {
return baos.toString();
}
}
/**
* Constructs a response adaptor wrapping the given response.
*
* #param response The response to be wrapped
* #throws IllegalArgumentException if the response is null
*/
public NonEncodedResponse(HttpServletResponse response, Integer fileSize) {
super(response);
this.setContentLength(fileSize);
this.setBufferSize(fileSize);
super.setContentLength(fileSize);
super.setBufferSize(fileSize);
baos = new ByteArrayOutputStream(fileSize);
}
#Override
public PrintWriter getWriter(){
return new PrintWriter(baos);
}
}
Now, originally I tried not wrapping the response at all and just calling response.setBufferSize(fileSize); and response.setContentLength(fileSize); but this seemed to have 0 actual effects on my output and when I looked at the headers I was still using a chunked transfer encoding without a fixed Content-Length. (I also tried setting a custom header like this and didn't see it appended on my response either. I'm assuming the response that goes into the filter in it's base form is some form of read only.)
I also tried using my wrapper and bypassing its output stream by reading and sending the bytes straight from the file
val fileContents = Files.readAllBytes(Paths.get(requestFilePath));
out.write(new String(fileContents));
because it seemed like even though my attempts to fix the content-length had failed I was still only seeing parts of files being sent as the whole thing in my browsers network tab while trying to debug.
All in all now I have it serving files like this (probably not ideal) and even less ideally it still says Transfer-Encoding: chunked despite all my efforts to put a fixed Content-Length on my wrapper and original response. I can now put a custom header on my content, so I know it's running through my filter. It seems like it's still just chunk encoded for some reason.
Can someone please tell me what I'm doing wrong or if filters can even be used to disable chunked encoding to begin with (I believe I saw things suggesting they could from google searches, but the truth is I just don't know). I would happily welcome any and all advice on this issue I no longer have ideas to try and I certainly don't know what I'm doing.
So I actually managed to get this working by reverting to my original approach and ditching my wrapper entirely and moving the filterChain.doFilter to the end of my code block. I'm not really sure why this works compared to what I was doing because I honestly have to confess I don't know what filterChain.doFilter actually does at all. This was my final end result that got things working.
public class NoChunkEncodedJSFilter extends OncePerRequestFilter {
#Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
val requestFilePath = this.getRealFilePath(request.getRequestURI());
File requestedFile = new File(requestFilePath);
if (requestedFile.exists()) {
val fileSize = Math.toIntExact(requestedFile.length());
val fileContents = Files.readAllBytes(Paths.get(requestFilePath));
response.reset();
response.setHeader("Content-Length", String.valueOf(fileSize));
response.setContentLength(fileSize);
ServletOutputStream sos = response.getOutputStream();
sos.write(fileContents);
sos.close();
}
filterChain.doFilter(request, response);
}
private String getRealFilePath(String requestURI) {
val unversionedServletPath = requestURI.substring(requestURI.indexOf("/js"));
return getServletContext().getRealPath(unversionedServletPath);
}
}
Now my only remaining question is, is there a smarter way to get my buffer data for my file, or do I have to re-read the file from disk every time? I'm assuming somewhere in there my file has probably already been loaded into memory ready to be served. Here I am loading it into memory a second time and I imagine this isn't very performant.
Related
I'm working on a GWT application which gives every team in the company an overview about what they have to do.
The program is working, but now we want that the Excel table which you can download will be a .xlsx and not a .xls.
This whole project is new for me and I consider myself as a beginner in GWT.
In the code, when the filename is given for the Excel table, there is a +".xls" at the end. When I change it to +".xlsx" and test the application, the download still works. However, when I try to open the file in Excel, it shows me an error message and tells me the file is corrupted. (.xls works)
Can you explain to me how a download works in GWT with a serverSite generated Excel?
Maybe you have some ideas what causes the file to be corrupted
(sadly the programmer of this application is on holiday, so I cannot ask him)
public class Download extends HttpServlet {
private static final long serialVersionUID = 5580666921970339383L;
#Override
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
doGet(request, response);
}
#Override
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String filename = (String)request.getSession().getAttribute(CrossReportConstants.ATTR_FILENAME);
byte[] data = (byte[])request.getSession().getAttribute(CrossReportConstants.ATTR_REPORT);
request.getSession().setAttribute(CrossReportConstants.ATTR_FILENAME, null);
request.getSession().setAttribute(CrossReportConstants.ATTR_REPORT, null);
response.setContentType("application/vnd.ms-excel");
response.setHeader("Content-Disposition", "attachment;filename=" + filename);
response.setHeader("Pragma", "no-cache");
response.setHeader("Expires", "0");
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate");
response.setContentLength(data.length);
try {
InputStream in = new ByteArrayInputStream(data);
ServletOutputStream out = response.getOutputStream();
byte[] outputByte = new byte[4096];
// copy binary contect to output stream
while (in.read(outputByte, 0, 4096) != -1) {
out.write(outputByte, 0, 4096);
}
in.close();
out.flush();
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Now as you provided code you question can be easily answered:
The shown code defines a HttpServlet. Somewhere in your project there is a file called web.xml. In this file the class you showed is mapped to an url pattern, therefore you server knows that a specific url should be handled by this servlet.
The servlet you showed first extracts the file name and the file content out of the session. Additional the http response is prepared and the file content is written out. Now you just have to replace the content type of the response with the one for xlsx.
response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
The browser which handles the http response now recognizes the download as a .xlsx file. The file extension does not really matter in this step, as you noticed.
When the original programmer of the servlet comes back from his hollidays, you should/could recommend him to use response.sendError() (with an appropriate http status code) instead of e.printStackTrace(). Then the user of the servlet can better understand if something do not work and who is to blame.
The following code demonstrates the difference in HttpServletRequest processing a request between Spring-Boot and a stand-alone servlet container (aka. Tomcat or Jetty). Both work, however I am less than happy with the Spring variant.
I have been unable to create a method to either copy the InputStream before it is processed by Spring or to suppress the processing of the request by Spring and just pass it "as is" to the Servlet. The servlet is interacting with a legacy API and needs to read/write bytes directly from the Request Body. Spring tries to map the payload, reads the InputStream and that requires the kludge of reading ParameterNames instead of just passing the Request Body on to the underlying socket to the legacy API. Any help turning this horrid spring code into something a bit more sane is greatly appreciated.
Things I've already tried (not necessarily correctly) that haven't worked. Filters to copy the InputStream, various consumes and produces annotations, etc... I also played with ByteArrayHttpMessageConverter to no avail.
The easiest solution in my opinion would be to flag the Servlet that works in Tomcat and just have Spring pass on the request via ServletRegistrationBean without modification. I'm sure I'm just missing something obvious here.
/*
* Works in Spring-Boot 1.2.5
*/
public void doIt(HttpServletRequest request, HttpServletResponse response)
{
response.setContentType("application/octet-stream");
response.setHeader("Cache-Control", "no-cache");
response.setContentLength(0);
Writer w = acquireWriter();
Enumeration<String> an = request.getParameterNames();
while(an.hasMoreElements()){
char[] msg = an.nextElement().toCharArray();
if(msg[0]=='w')
continue;
e.write(msg, 0, msg.length);
}
}
#Bean
public MyServlet myServlet(){
return new MyServlet();
}
#Bean
public ServletRegistrationBean servletRegistrationBean(){
ServletRegistrationBean bean = new ServletRegistrationBean(myServlet(), "/op");
bean.setOrder(Ordered.HIGHEST_PRECEDENCE);
return bean;
}
/*
* Works in Tomcat 7/8
*/
public void doIt(HttpServletRequest request, HttpServletResponse response)
{
response.setContentType("application/octet-stream");
response.setHeader("Cache-Control", "no-cache");
response.setContentLength(0);
Writer w = acquireWriter();
InputStreamReader input = new InputStreamReader(request.getInputStream(), "UTF-8");
char[] buffer = new char[8192];
int length=-1;
while((length = input.read(buffer, 0, buffer.length)) != -1) {
w.write(buffer, 0, length);
}
}
M. Deinum was right on the money. The problem was the HiddenHttpMethodFilter calling getParameter which triggers the preparation of the request and reads the InputStream. I played around with registering the HiddenHttpMethodFilter and a DispatcherServlet, but found it cleaner and easier to just override the HiddenHttpMethodFilter, but both solutions do work. Here is exactly what I did to pass the request unaltered to the servlet.
#Bean
public HiddenHttpMethodFilter hiddenHttpMethodFilter(){
return new HiddenHttpMethodFilter(){
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException
{
if("POST".equals(request.getMethod()) && request.getRequestURI().startsWith("/op")) {
filterChain.doFilter(request, response);
} else {
super.doFilterInternal(request, response, filterChain);
}
}
};
}
I am having trouble trying to get cross-domain ajax request working and despite the many solutions I have found on Stack Overflow I am unable to get it working.
$.ajax({
url : 'http://SERVER:PORT/CONTEXT/RESOURCE.html?someParameter=1234',
dataType : 'json',
success: function(xhr) {
alert('ok '+JSON.stringify(xhr));
},
error : function(xhr) {
alert('error '+JSON.stringify(xhr));
}
});
Doing just a standard $.ajax call with datatype "json" the server responds with a blank response and statusText "error", like so:
error {"readyState":0,"responseText":"","status":0,"statusText":"error"}
So I tried simply changing datatype to "jsonp" as suggested in other threads but this time it still goes to error condition with the following response:
error {"readyState":4,"status":200,"statusText":"success"}
and an error message of "parsererror"
Yet no data.
What gives?
Do I need to do something special on the server side because it is Spring MVC in Weblogic?
EDIT:
jQuery version 1.9.1
Spring-3 MVC
EDIT2: Oh yes I also tried $.getJSON but this command seems to do nothing - when I run the code replacing $.ajax with $.getJSON nothing happens. No response and I dont see any error occurring in console and no Network request seen going to the URL. I did also change the syntax in a 2nd try where I called it like $.getJSON(url, callback); but that did not change anything
EDIT3: I should also mention when I run the original code using "json" datatype and look in Firebug's Response tab, it is empty. But when I run the second code using "jsonp" I do see the JSON text in the Response tab. So it is strange why it still throws an error.
OK, upon more research I finally found the cause - yes I did need to do something on the Server side to support jsonp. I ended up writing a servlet filter which wraps the returning json string in the appropriate callback.
Learn something new everyday!
public class JsonPCallbackFilter extends OncePerRequestFilter {
Logger logger = Logger.getLogger(JsonPCallbackFilter.class);
#Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
//logger.debug("Filter: "+request.getRequestURI());
#SuppressWarnings("unchecked")
Map<String, String[]> parms = request.getParameterMap();
if(parms.containsKey("callback")) {
logger.debug("Wrapping response with JSONP callback '" + parms.get("callback")[0] + "'");
OutputStream out = response.getOutputStream();
ByteResponseWrapper wrapper = new ByteResponseWrapper(response);
chain.doFilter(request, wrapper);
StringBuffer sb = new StringBuffer();
sb.append(parms.get("callback")[0] + "(");
sb.append(new String(wrapper.getBytes()));
sb.append(new String(");"));
out.write(sb.toString().getBytes());
wrapper.setContentType("text/javascript;charset=UTF-8");
response.setContentLength(sb.length());
out.close();
} else {
chain.doFilter(request, response);
}
}
}
static class ByteOutputStream extends ServletOutputStream {
private ByteArrayOutputStream bos = new ByteArrayOutputStream();
#Override
public void write(int b) throws IOException {
bos.write(b);
}
public byte[] getBytes() {
return bos.toByteArray();
}
}
static class ByteResponseWrapper extends HttpServletResponseWrapper {
private PrintWriter writer;
private ByteOutputStream output;
public byte[] getBytes() {
writer.flush();
return output.getBytes();
}
public ByteResponseWrapper(HttpServletResponse response) {
super(response);
output = new ByteOutputStream();
writer = new PrintWriter(output);
}
#Override
public PrintWriter getWriter() {
return writer;
}
#Override
public ServletOutputStream getOutputStream() throws IOException {
return output;
}
}
<filter>
<filter-name>jsonpFilter</filter-name>
<filter-class>com.blahblah.JsonPCallbackFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>jsonpFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Lead by several examples and questions answered here ( mainly
http://www.javaworld.com/javaworld/jw-02-2009/jw-02-servlet3.html?page=3 ), I want to have server sending the response multiple times to a client without completing the request. When request times out, I create another one and so on.
I want to avoid long polling, since I have to recreate request every time I get the response. (and that quite isn't what async capabilities of servlet 3.0 are aiming at).
I have this on server side:
#WebServlet(urlPatterns = {"/home"}, name = "async", asyncSupported = true)
public class CometServlet extends HttpServlet {
public void doGet(final HttpServletRequest request, final HttpServletResponse response) throws IOException, ServletException {
AsyncContext ac = request.startAsync(request, response);
HashMap<String, AsyncContext> store = AppContext.getInstance().getStore();
store.put(request.getParameter("id"), ac);
}
}
And a thread to write to async context.
class MyThread extends Thread {
String id, message;
public MyThread(String id, String message) {
this.id = id;
this.message = message;
}
public void run() {
HashMap<String, AsyncContext> store = AppContext.getInstance().getStore();
AsyncContext ac = store.get(id);
try {
ac.getResponse().getWriter().print(message);
} catch (IOException e) {
e.printStackTrace();
}
}
}
But when I make the request, data is sent only if I call ac.complete(). Without it request will always timeout. So basically I want to have data "streamed" before request is completed.
Just to make a note, I have tried this with Jetty 8 Continuation API, I also tried with printing to OutputStream instead of PrintWriter. I also tried flushBuffer() on response. Same thing.
What am I doing wrong?
Client side is done like this:
var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://localhost:8080/home', true);
xhr.onreadystatechange = function () {
if (xhr.readyState == 3 || xhr.readyState == 4) {
document.getElementById("dynamicContent").innerHTML = xhr.responseText;
}
}
xhr.send(null);
Can someone at least confirm that server side is okay? :)
Your server-side and client-side code is indeed ok.
The problem is actually with your browser buffering text/plain responses from your web-server.
This is the reason you dont see this issue when you use curl.
I took your client-side code and I was able to see incremental responses, with only just one little change:
response.setContentType("text/html");
The incremental responses showed up immediately regardless of their size.
Without that setting, when my output was a small message, it was considered as text/plain and wasnt showing up at the client immediately. When I kept adding more and more to the client responses, it got accumulated until the buffer size reached about 1024 bytes and then the whole thing showed up on the client side. After that point, however, the small increments showed up immediately (no more accumulation).
I know this is a bit old, but you can just flushBuffer on the response as well.
I'm working on a Servlet and trying to log the requests. The crucial part of the code causing the error is the following:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException {
StringWriter writer = new StringWriter();
IOUtils.copy(request.getInputStream(), writer);
Message.Builder builder = something of type com.google.protobuf.GeneratedMessage.Builder;
builder.mergeFrom(writer.toString().getBytes());
}
The final line of code above results in the following Exception:
com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.
However, when the code is switched to:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException {
Message.Builder builder = something of type com.google.protobuf.GeneratedMessage.Builder;
builder.mergeFrom(request.getInputStream());
}
There is no error, and everything works fine. What could the problem be? I seem to need something similar to the first code snippet because I need to use the input stream a second time (once to write it to a file, and once to process the actual request).
How about this?
InputStream inputStream = request.getInputStream();
byte[] data = IOUtils.toByteArray(inputStream);
Message.Builder builder = something of type com.google.protobuf.GeneratedMessage.Builder;
builder.mergeFrom(data);
// then use inputStream for something else
Apparently, I needed to use coded input and output streams so that the data could be read correctly. See my answer to my other question.