I have a very strange scenario where, in a Linux server, CURL successfully retrieves a response from a web service. When that same request is issued by Tomcat on the same linux server used for the CURL command, for some reason Tomcat receives a 400 status code, which prevents me from doing our business logic.
Flow with CURL:
CURL issues request to Service A using Proxy A
Service A retrieves the data we need and returns it, as well as a 200 Status Code
CURL receives the correct data and 200 status code...
Flow with Tomcat:
Tomcat issues request to Service A using Proxy A
Service A retrieves the data we need and returns it, as well as a 200 Status Code
Tomcat receives a 400 status code and is not able to receive the correct data...
What could be causing this problem? Tomcat and CURL are using the same proxy and are in the same linux server... even the service is able to fetch the data successfully and return it to both. Only in the case of tomcat, the service is throwing this error after trying to write the data in the response:
2021-03-10 21:49:36.908 WARN 90623 --- [https-jsse-nio-8123-exec-10] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException]
Tomcat closes the socket because it sees a 400 response code, and doesn't even try reading the bytes from the response when I do con.getInputStream(). I don't really know where that 400 Status code is coming from.
Any help is appreciated. Thanks.
UPDATE 2021/03/11:
One thing I forgot to mention is that, Tomcat is able to perform other requests to that same service. The main difference here is the size of the response. It timesout when trying to read this large response, in comparison to other smaller responses we get.
UPDATE 2021/03/16:
After investigating deeper on what is happening I found out that, whenever I issue the request with Java, for some reason it timesout at exactly 2 minutes. That doesn't happen with CURL, only with Java. Is there anything I may be missing? I have already added these to my code:
JVM Arguments:
-Dsun.net.client.defaultReadTimeout=6000000 -Dsun.net.client.defaultConnectTimeout=6000000
Java code
con.setReadTimeout(Integer.MAX_VALUE);
con.setConnectTimeout(Integer.MAX_VALUE);
con.setDoOutput(true);
con.setDoInput(true);
I'm using an HttpURLConnection object.
http.converter.HttpMessageNotReadableException
The message that your tomcat sends to the service can not be read. Something happens with the message that your tomcat sends to the service. Does it have a json body included? If yes check that model class how it is serialized.
Related
I am supporting another vendors legacy application.
This is a J2EE application that runs on Glassfish v3.1.2.2. It has a REST API implemented using JAX-RS. I have limited visibility to the application and source.
The symptoms are:
make an HTTP request to a REST API
application has its own auditing system, this shows a successful request
no errors in GF logs
GF access log notes the request
0 bytes are returned from the request to the caller
This happens for both remote calls as well as from calls made using curl on localhost.
If we make the same requests to a different port over HTTPS they succeed. We are reluctant to move the calls to that other port without knowing a root cause. These failed intermittently last night and now fail constantly today.
A packet capture of the request shows:
- TCP overhead/handshake
- A GET request
- A single ACK from the application back to the caller
- then nothing after that
What would cause Glassfish v3 to successfully handle and process an HTTP request but return no data?
Is there a mechanism in Glassfish v3 to flush or reset an HTTP listener and its associated thread pool?
Since this happens on a curl request on the same server to localhost I think I can rule out the network being the issue.
The ports being used communicate directly with Glassfish. There is no proxy (like Apache or Nginx) between the caller and the app server.
Are there logging or monitoring settings I should be enabling in Glassfish to observe what the HTTP listener is doing relative to the application and the network stack?
I have obfuscated some examples that show the symptoms:
Glassfish Access log:
"0:0:0:0:0:0:0:1" "NULL-AUTH-USER" "25/Oct/2018:11:21:02 -0500" "GET /api/obfuscated/by/me HTTP/1.1" 200 9002
Curl response for that same call:
* Trying OFBBFUSCATED
* Connected to hostname.local (OFBBFUSCATED) port 11080 (#0)
> GET /api/obfuscated/by/me HTTP/1.1
> Host: hostname.local:11080
> User-Agent: curl/7.43.0
> Accept: */*
> Authorization: Basic asdfdsfsdfdsfsdafsdafsdafw==
>
* Empty reply from server
* Connection #0 to host hostname.local left intact
UPDATE I changed a timeout setting for the HTTP network listener. I bumped it from 30 to 35 seconds because I was seeing a packet capture where the app was sending a FIN after 30 seconds. After making this change it started to work again.
It is not clear if this somehow flushed or reset something or if I had some kind of race condition.
The apparent root cause was high I/O on the system running these services. The applications normally used 50MB/sec, a new process drove that usage to 250MB/sec. Once the I/O problem was resolved all of the HTTP errors went away and haven't come back.
I run a proxy on localhost:2080 for debugging purposes. I can see trafic going through my proxy using either:
curl http://localhost:8888/stuff --proxy localhost:2080
curl http://some.server.com:8888/stuff --proxy localhost:2080
Using a Spring application, or a bare bones HttpURLConnectionExample Java application, I see requests go through the proxy when I call some.server.com, but never on localhost, even though I set http.nonProxyHosts to a random value (to avoid the default of localhost).
I made sure my JVM arguments were taken into accounts by adding the following code:
System.out.println("java.version="+props.getProperty("java.version"));
System.out.println("http.proxyHost="+props.getProperty("http.proxyHost"));
System.out.println("http.proxyPort="+props.getProperty("http.proxyPort"));
System.out.println("http.nonProxyHosts="+props.getProperty("http.nonProxyHosts"));
That produces the following output:
java.version=1.8.0_131
http.proxyHost=localhost
http.proxyPort=2080
http.nonProxyHosts=dummy.snafu.com
https.proxyHost=localhost
https.proxyPort=2080
https.nonProxyHost=dummy.snafu.com
Testing 1 - Send Http GET request
Sending 'GET' request to URL : http://localhost:8888/stuff
Response Code : 200
hello
Why are HTTP requests to localhost not going through my proxy when run with the following JVM arguments?
-Dhttp.proxyHost=localhost
-Dhttp.proxyPort=2080
-Dhttp.nonProxyHosts=dummy.snafu.com
This is a limitation of DefaultProxySelector, It always appends local Uris(localhost, 127.*, etc) to whatever you configure as nonProxyHosts, essentially bypassing proxy for all local Uris(localhost|127. *|[::1]|0.0.0.0|[::0]). Fortunately, there is a workaround by creating your own proxy selector and registering it as default.
When I make a rest request and server throws an application exception like IllegalArgumentException, I get response with http status 404.
Wouldn't a response with http status 500 be better?
Or what would actually be the expected response when an application exception is thrown? Is there some default behaviour in resteasy, spring or tomcat itself?
I know I can use an ExceptionMapper for resteasy, but is this really best practice or is there a better alternative?
I'm using following setup:
spring
resteasy
tomcat 7
Update:
The problem I'm facing is that I have 2 web applications, one is returning http status 500 and the other is returning http status 400 when an IllegalArgumentException is thrown. I can't figure out why they behave different. It seems to me, that both web application have the same spring and resteasy configuration.
When I'm debugging, I see that resteasy is transforming the IllegalArgumentException to a org.jboss.resteasy.spi.UnhandledException, but the response and the response status code, respectively, is not touched.
So besides not knowing which http status code would be the expected one (400, as Jon Skeet and Stefano Cazzola already pointed out), I didn't know either why the 2 web applications behaves different.
I couldn't find any resteasy ExceptionMapper in both web application.
So is there some default behavior in resteasy, spring or tomcat, which is mapping an IllegalArgumentException to http status 404 or http status 500? Or how can this happen?
The response code is correlated to the error happened in the server. If the error is related to an invalid input received from the client, then the formally correct response status is 400 (Bad Request). The difference with 500 is that returning 400 is the correct behaviour for the server: means, the server processed correctly and responded with an error because the request was wrong. If the same request will ever be resubmitted, it will receive the same error response. The server is not supposed to correct this error, it is up to the client.
You can take this link as a reference
The pubsub example of atmosphere is deployed to weblogic 12c.
with the below small change i deployed the app
var request = {
url :document.location.toString()+'ws/pubsub/' + getElementByIdValue('topic'),
Gives the following error...
On server side
org.atmosphere.websocket.protocol.SimpleHttpProtocol BEA-000000 Status code higher or equal than 400 Status 405 Message Method Not Allowed
In script console(chrome)
Network
ERROR: 405:Method Not Allowed
ERROR: 405:Method Not Allowed
message=testing pubsub sent trying to use websocket
console
GET =1396276957934">http://localhost:7001/atmosphere/ws/pubsub/Test?X-Atmosphere-Transport=close&X-Atmosphere-tracking-id=&=1396276957934 400 (Bad Request) atmosphere.js:1816
WebSocket connection to 'ws://localhost:7001/atmosphere/ws/pubsub/Test?X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=2.1.4-javascript&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&X-Cache-Date=0&X-atmo-protocol=true' failed: Received a broken close frame containing a reserved status code.
Solved
document.location.toString()
needs to be modified to
document.location.origin+'/atmosphere/ws/pubsub/'
to skip the name of the html file(if not index.html) .
The atmosphere handler will take care of the operation and connects to websockets as intended.
The long polling needs to be addressed as a seperate url , as the weblogic-12c adds /ws/ to context path automatically, when it detects websocket stuff
refer to
complete solution
I have a tomcat server and java client with java.net.HttpUrlConnection sending messages. Usually all works fine, but there some requests the client sends out which are responded with a 400 http error.
The problem is that nothing is shown on tomcat logs. I even tried to put .level=fine and other log level changes, and still no exception or error was shown.
I'm suspecting bad encoding in request or something like that, but tomcat doesn't give me any clues.
By the way, the URL is OK. I also see the request reaches the server but not to my servlet.
It appears tomcat gets the message, declares it bad and doesn't log what's wrong.
Can anybody think of a way to see the 400 reason?
Thank you
I finally figured out -
I saw that the size of the total headers + size of data written to OutputStream - was exceptionally high - and so I defined in the Tomcat HTTP connector:
Connector port="8080" protocol="HTTP 1/1" maxPostSize="10485760" maxHttpHeaderSize="10485760"/
It is a shame Tomcat doesn't just tell me this, and I had to guess my - I will report this as a bug.
Thank you
Indeed, this was also the solution to the "http 400 bad request" error on Internet Explorer 11.
Adding
maxHttpHeaderSize="10485760" maxPostSize="10485760"
to the Connector-node of Tomcat's server.xml.
Seems my ajax-request (extjs) on IE11 are using a bigger request header then the default (8192 = 8 KB) max http header size on Tomcat 6?
With IE10 and Chrome I didn't experience this problem.
This is a very nice issue to be discussed about. I also faced the same issue in my organization's work. I was clueless about it as there were no good logging about it in catalina.out or in our application logs. Http calls made from client apps to the API that was running in Tomcat_8 were getting rejected by the server with 400 bad request. Only Tomcat locahost_access logs records it by printing the 400 http status fro the request. After I increased the http header size the issue got resolved.