I have a Spring Boot backend and I have just solved "ERR_CONNECTION_RESET", when uploading a file from Angular frontend, by configuring maxSwallowSize Tomcat property. I'm trying to understand what it does exactly. Tomcat documentation is not obvious to me:
The maximum number of request body bytes (excluding transfer encoding
overhead) that will be swallowed by Tomcat for an aborted upload. An
aborted upload is when Tomcat knows that the request body is going to
be ignored but the client still sends it. If Tomcat does not swallow
the body the client is unlikely to see the response. If not specified
the default of 2097152 (2 megabytes) will be used. A value of less
than zero indicates that no limit should be enforced.
https://tomcat.apache.org/tomcat-8.0-doc/config/http.html
Can I get some help?
If your servlet's service() method exits (normally or exceptionally) without consuming the whole client's request body, Tomcat will still accept maxSwallowSize bytes before resetting the connection. This is required since most browsers read the server's response only after they sent the entire request (cf. this question).
To consume the requests body you need to:
if the request is encoded as application/x-www-form-urlencoded, you need to call one of the getParameter* methods,
if the request is encoded as multipart/form-data, you need to call one of the getPart* methods,
in all other cases you need to read the entire InputStream.
An unconsumed request body is usually caused by errors, including those in the parsing of parameters or form parts.
Related
I'm streaming response of unknown size to client from a servlet (really a Spring Boot controller but it shouldn't matter).
I force sending headers early by calling flush() so that the client (browser) can properly show that the download is started.
What should I do if an error happens in the middle of the stream?
Right now client simply gets part of the file and has no idea it's bad.
According to What to do with errors when streaming the body of an Http request I should to TCP RST on Socket or write some garbage that is not a valid chunked encoded data.
Unfortunately I have not found a way of accessing underlying socket nor can I write malformed response - ServletResponse#getOutputStream does chunking internally and I couldn't find a way of bypassing that either.
This was fixed in https://github.com/apache/tomcat/commit/fa0f2567aae91a13137094ebcdb74d72b8222f3b
And is available since Tomcat to 10.1.0.
I need some assistance on a project I am working on. It's a library itself using Jersey 1.x (1.19.1) aiming at HTTP posting a JSON document and getting the corresponding JSON response from a server.
I am facing a problem when the response from the server is "big". The JSON document that is posted by my client application contains several jobs that must be executed by the server, and
the JSON document sent back by the server is made of the outputs of these jobs. The jobs can be considered independent from each other. The server works in streaming mode, which means it
starts to process the jobs before it receives the entire JSON document posted by the client. And it starts to send the outputs of the jobs as soon as they are finished. So the server
starts to reply to my client application while it is still posting the request. Here is my problem. When the request gets big so gets the response (more jobs to do), my application freezes
and at some point terminates.
I spent a lot of time trying to figure out what's happening and here is what is found and what I infered.
Jersey, for handling HTTP communication is using a class from the JDK (in rt.jar) I forgot the exact name and don't have access to my work right now but let's call it HttpConnection.
In this class there is a method checkError() that is invoked and throws a IOException with only a message saying it was impossible to write to server.
Debugging I was able to understand that an attribute of this class named trouble was set to true because a write() method caught an IOException before. checkError() throws a
IOException based on that trouble boolean flag. It's not possible to easily see the cause IOException because the classes of the JRE are compiled without the debugging symbols but
I managed to see that this IOExeption was a "connection reset by peer" problem.
Then I tried to understand why the server resets the connection. I used a HTTP proxy that captures the HTTP traffic between my client application and the server but this gave me no more clues,
it even seems that the proxy is unable to handle properly the connection with the server as well!
So I tried to use Wireshark to capture the traffic and see what's wrong. Here is what I found.
On client side, packets corresponding to the post of the request JSON document are sent and the server starts to reply shortly after, as explained above. The server side sends
more and more packets and I noticed that the buffer of the TCP layer (called TCP window in Wireshark) on client side has a size that decreases more and more as the server sends packets.
Until it beomes full (size: 0 byte). So the TCP layer on server side cannot send data to the TCP layer on client side anymore and thus becomes full too. The conversation, in the end is
only about retrying to send data, on both sides, failing again and again. Ultimately the server decides to send a reset packet. This corresponds to the cause IOExcpetion I mentioned
above I believe.
My understanding is: as long as the server does not start to stream the response everything is fine. When the server starts to send the response, the TCP buffer on client side starts to
get filled. But as the client application does not read the response yet, the content of this buffer is not consumed. When the server has sent enough data to fill this buffer it cannot
send anymore data and the buffer of its TCP layer gets full too because the server continues to push data. As a result, the client application cannot finish to send the request JSON
document. The communication is blocked on both sides and the server decides to reset the connection.
My conclusion is: the code, as currently written, does not support such full duplex communication, because the response from the server is not consumed as it is received. Indeed, walking
through the Jersey code that is executed by my library, by debugging, it is clear that the pattern is:
first: connection.getOutputStream().write()
and then: response.getInputStream().read()
In my opinion, the root cause of the problem is that the library I am working on uses Jersey in this synchronous manner which does not fit well the way the server works (streaming the
response while the request is still being sent to it).
I searched a lot on the Internet a solution keeping Jersey 1.19.1 for me to improve the library with as few impacts as possible but I failed. This is the reason why I am asking help
here now ;)
So basicaly my question is: is it possible to do what I need to do keeping Jersey client library 1.19.1 and if yes how? If not, what HTTP client library should I use for my library (to
write a post request and read the corresponding response at the same time) and if you could give me a basic example so I can be on track quickly it would be much appreciated.
One last thing: curl works just fine, I can fully post the exact same JSON document and get the response using it, so there is no problem on server side as I suspected at the very
beginning of my investigation. And it scales fine (I tried to send huge JSON documents). Of course I made sure the HTTP header of the post is the same in the case of my library and in the
curl case.
Thanks a lot for reading me and for your answers.
Best regards,
Loïc
I am using Jetty 8 in blocking I/O mode. The servlet doPost reads the request content from the request InputStream (which is actually an instance of Jetty HttpInput).
What if a client sends a very large request and does it very slowly ? Obviously, it will take a lot of time to read the request. So, I would like to cancel the reading after a few seconds and send an error response to the client. How can I do that without much changes i.e. using Jetty with blocking I/O and without continuation ?
Jetty 9+ answer
Use HttpServletResponse.sendError(-1) to terminate the connection.
Original Jetty 8 answer
Write a ServletFilter that does the detection of the slow request.
When you determine it is slow, ...
Obtain a reference to the current connection via a call to AbstractHttpConnection.getCurrentConnection()
Get a reference to the EndPoint via AbstractHttpConnection.getEndPoint()
Call EndPoint.close()
Also, look into the QoSFilter, it does a lot of this kind of stuff already.
I have recently been asked to configure CXF to the same parameters as our older XFire service.
One of those parameters was Keep-Alive: timeout=60, max=20.
However, I did some research and it appears that CXF uses the JVM HttpURLConnection object under the hood. From what I see, there has been some attempts to provide configuration for it but nothing has been commited for now.
I would prefer not to change the http.maxConnections parameter as it would impact all the server instead of the CXF web services only.
I found this interresting forum thread talking about it where Daniel Kulp says:
BTW: there is a way to control the connection pooling, but it's a
SERVER side thing. Basically, if the server sends back a header of:
Keep-Alive: timeout=60, max=5
then the Java client will respect those values. Right now in CXF,
you would probably need to write an interceptor to set those values.
I just made a commit to trunk that expands the http configuration to
include a setting to control this from the config file.
I could write an interceptor that modify the headers to do so. However my question is: How will the server react to this kind of change? Would not that be a problem if the server expects 5 connections max and the client performs more ?
According to what I read here, the keep-alive parameters can be controller either by system properties or directly in the HTTP headers:
The support for HTTP keep-Alive is done transparently. However, it can
be controlled by system properties http.keepAlive, and
http.maxConnections, as well as by HTTP/1.1 specified request and
response headers.
I'm running a servlet in Tomcat 6.0.26. The servlet accepts file upload from the client by HTTP POST. I'd like to stop the file uploading from the HttpServlet side. I tried the following methods with no luck:
close the request inputstream
send error code HttpServletResponse.SC_REQUEST_ENTITY_TOO_LARGE and flush response
do 1 and 2 in a Filter
I googled but found no direct answers. Please advise solutions.
Thanks.
This is not possible using the standard Servlet nor Commons FileUpload API's. Basically, to be able to abort the connection immediately, you should grab the underlying socket physically and close it. However, this socket is controlled by the webserver. See also this related question: How to explicitly terminate http connection from server with no response header.
Little tests have however confirmed that Commons FileUpload doesn't buffer up the entire file in memory when its size exceeds the limit. It will read the input stream, but just ignore and throw away the read bytes (also the ones which are already read). So memory efficiency isn't necessarily the problem here.
To fix the real problem, you'd basically like to validate the file size in the client side rather than the server side. This is possible with a Java Applet or a Flash Application. For example, respectively JumpLoader and SWFUpload.
This is not possible using standard APIs. And you're violating some protocol standards/RFC if you do. So why would you do that?
Instead, send a "Connection: close" response (http header) with no http body.
here is some crazy workaround: you can write (or find somewhere) some firewall standalone application based on Sockets that handles HTTP request, parses headers and if the request matches some your custom conditions - firewall forwards it to Tomcat, otherwise returns HTTP error response. Or you can try to tune Apache <-> Tomcat with some Apache rules.