We Keep Coding

Tomcat receives HTTP 400 but CURL receives 200

I have a very strange scenario where, in a Linux server, CURL successfully retrieves a response from a web service. When that same request is issued by Tomcat on the same linux server used for the CURL command, for some reason Tomcat receives a 400 status code, which prevents me from doing our business logic.
Flow with CURL:
CURL issues request to Service A using Proxy A
Service A retrieves the data we need and returns it, as well as a 200 Status Code
CURL receives the correct data and 200 status code...
Flow with Tomcat:
Tomcat issues request to Service A using Proxy A
Service A retrieves the data we need and returns it, as well as a 200 Status Code
Tomcat receives a 400 status code and is not able to receive the correct data...
What could be causing this problem? Tomcat and CURL are using the same proxy and are in the same linux server... even the service is able to fetch the data successfully and return it to both. Only in the case of tomcat, the service is throwing this error after trying to write the data in the response:
2021-03-10 21:49:36.908 WARN 90623 --- [https-jsse-nio-8123-exec-10] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.http.converter.HttpMessageNotReadableException: I/O error while reading input message; nested exception is org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException]
Tomcat closes the socket because it sees a 400 response code, and doesn't even try reading the bytes from the response when I do con.getInputStream(). I don't really know where that 400 Status code is coming from.
Any help is appreciated. Thanks.
UPDATE 2021/03/11:
One thing I forgot to mention is that, Tomcat is able to perform other requests to that same service. The main difference here is the size of the response. It timesout when trying to read this large response, in comparison to other smaller responses we get.
UPDATE 2021/03/16:
After investigating deeper on what is happening I found out that, whenever I issue the request with Java, for some reason it timesout at exactly 2 minutes. That doesn't happen with CURL, only with Java. Is there anything I may be missing? I have already added these to my code:
JVM Arguments:
-Dsun.net.client.defaultReadTimeout=6000000 -Dsun.net.client.defaultConnectTimeout=6000000
Java code
con.setReadTimeout(Integer.MAX_VALUE);
con.setConnectTimeout(Integer.MAX_VALUE);
con.setDoOutput(true);
con.setDoInput(true);
I'm using an HttpURLConnection object.

http.converter.HttpMessageNotReadableException
The message that your tomcat sends to the service can not be read. Something happens with the message that your tomcat sends to the service. Does it have a json body included? If yes check that model class how it is serialized.

J2EE app running on Glassfish v3 is not responding to HTTP requests. App logs success but no data sent back over HTTP

I am supporting another vendors legacy application.
This is a J2EE application that runs on Glassfish v3.1.2.2. It has a REST API implemented using JAX-RS. I have limited visibility to the application and source.
The symptoms are:
make an HTTP request to a REST API
application has its own auditing system, this shows a successful request
no errors in GF logs
GF access log notes the request
0 bytes are returned from the request to the caller
This happens for both remote calls as well as from calls made using curl on localhost.
If we make the same requests to a different port over HTTPS they succeed. We are reluctant to move the calls to that other port without knowing a root cause. These failed intermittently last night and now fail constantly today.
A packet capture of the request shows:
- TCP overhead/handshake
- A GET request
- A single ACK from the application back to the caller
- then nothing after that
What would cause Glassfish v3 to successfully handle and process an HTTP request but return no data?
Is there a mechanism in Glassfish v3 to flush or reset an HTTP listener and its associated thread pool?
Since this happens on a curl request on the same server to localhost I think I can rule out the network being the issue.
The ports being used communicate directly with Glassfish. There is no proxy (like Apache or Nginx) between the caller and the app server.
Are there logging or monitoring settings I should be enabling in Glassfish to observe what the HTTP listener is doing relative to the application and the network stack?
I have obfuscated some examples that show the symptoms:
Glassfish Access log:
"0:0:0:0:0:0:0:1" "NULL-AUTH-USER" "25/Oct/2018:11:21:02 -0500" "GET /api/obfuscated/by/me HTTP/1.1" 200 9002
Curl response for that same call:
* Trying OFBBFUSCATED
* Connected to hostname.local (OFBBFUSCATED) port 11080 (#0)
> GET /api/obfuscated/by/me HTTP/1.1
> Host: hostname.local:11080
> User-Agent: curl/7.43.0
> Accept: */*
> Authorization: Basic asdfdsfsdfdsfsdafsdafsdafw==
>
* Empty reply from server
* Connection #0 to host hostname.local left intact
UPDATE I changed a timeout setting for the HTTP network listener. I bumped it from 30 to 35 seconds because I was seeing a packet capture where the app was sending a FIN after 30 seconds. After making this change it started to work again.
It is not clear if this somehow flushed or reset something or if I had some kind of race condition.

The apparent root cause was high I/O on the system running these services. The applications normally used 50MB/sec, a new process drove that usage to 250MB/sec. Once the I/O problem was resolved all of the HTTP errors went away and haven't come back.

Https web service call failure

We have implemented webservice call using JAX-WS RI 2.1.6 in JDK 6 now problem comes when we enable https webservice call stops reaching server and java reports following error,
javax.xml.ws.WebServiceException: java.io.IOException: Async IO
operation failed (3), reason: RC: 55 The specified network resource
or device is no longer available.
Now I have tested this within SoapUI and response from the service is received there.
Looked into various solution where it tells us to provide timeout settings but nothing seems work.
#WebEndpoint(name = "RulesSoap")
public RulesSoap getRulesSoap() {
((BindingProvider)super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class)).getRequestContext().put("com.sun.xml.internal.ws.connect.timeout", 1000);
((BindingProvider)super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class)).getRequestContext().put("com.sun.xml.internal.ws.request.timeout", 1000);
return super.getPort(new QName("urn:decision:Rules", "RulesSoap"), RulesSoap.class);
}
And just for information JAX-WS implementation is following few simple lines,
of course we submit all necessary data into respective stubs and all but I am not submitting here because our http calls are getting through,
Rules rules = new Rules(new URL(url), new QName("urn:decision:Rules", "Rules"));
RulesSoap rulesSoap = rules.getRulesSoap();
CorticonResponse response = rulesSoap.processRequest(request);
Note : Our application server WebSphere Application Server and Version 7.0.0.19
Thanks in Advance.

After lots of efforts we resolved this. I will provide steps if anything related to this happens how to find root cause,
Step 1 :
First of all we enabled soap tracing in WebSphere Application Server by following setting,
Admin Console > Servers > Server Types > WebSphere Application Servers >
{your server} > Troubleshooting > Change Log Detail Levels > Runtime
In run time please put this , *=info: com.ibm.ws.websvcs.*=all: org.apache.axis2.jaxws.*=all
This step will create trace.log file in your logs folder.
Now any web service request which goes out of your server will add logs to this file and necessary props like endpoint, request, response etc.
Step 2 :
Reading this trace.log file we found following endpoint,
PropertyValid 1 org.apache.axis2.jaxws.client.PropertyValidator validate validate property=(javax.xml.ws.service.endpoint.address) with value=(http://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0)
HTTPConnectio 3 resetConnection : http://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0 Persistent : true
Now if you notice here that our soap has endpoint address javax.xml.ws.service.endpoint.address where protocol is still using http which causes to fail ssl handshake.
Step 3 :
Solution for this is to override endpoint inside your soap stubs which can be implemented by adding following line,
RulesSoap rulesSoap = rules.getRulesSoap();
((BindingProvider)rulesSoap).getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, "https://uxm.solutions.lnet.com:9445/axis/dswsdl/Rules/1/0");
Conclusion :
So here is what i think even we pass https url while we are creating objects but still does not take this https url on runtime, to me this looks like stubs creation issue with JAX-WS.
Thanks.

What protocol /ciphers are you using? You have mentioned there is connection to webservice on WAS7 with JDK6 and Java 6 does not support TLS1.2 (and TLS1.1 only from certain fixpack).
See this:
How to use TLS 1.2 in Java 6

Spring RestTemplate on Jelastic - Connection Refused when requesting external rest service

I have application that need to call Rest service, to do authentication process. The service and my application are in a same server. I used Spring RestTemplate to call it. When it tested in localhost, it works perfectly. But when I deploy it to the cloud server (in this case, Jelastic) I get this error:
I/O error on GET request for "[my_rest_service_address]":Connection refused; nested exception is java.net.ConnectException: Connection refused
All the code is follow this doc : http://docs.spring.io/spring/docs/3.0.x/javadoc-api/org/springframework/web/client/RestTemplate.html. Only the url that changed to my rest url.
Is it happened because jelastic server blocked outbound request? But, both the service and my application are in the same server, why it still blocked?
Please explain to me, what is the problem? Thanks

The http 401 code means that the user that made the request is not authenticated. It can be the user of your webapp or the user used for the rest call.
It can for example that different credentials are used in prod and test, different security schemes in each environment, etc.

WstxEOFException: Unexpected EOF in Prolog on JBoss only

The error I am getting are in a few forum posts, but all the scenarios seem slightly different than mine.
I am writing a JAX-WS web service client to communicate to a soap-based web service that uses basic authentication (http level only). I generated the client using wsimport on the wsdl.
There are two versions of the web service available to me for testing:
Port 8080 - no authentication
Port 80 - requires basic http level authentication
What the web service does:
This is just a simple web service that lets me send an base64 encoded xml payload into it.
I verified the following:
I can correctly send both web services (80/8080) using SOAPUI from my localhost
I can correctly send both web services using a test JAVA application from my localhost
What fails:
As soon as I try to deploy my web service client as a .war on jboss5.1... only the port 8080 web service works. When I try the web service on port 80, I get this error.
10:36:51,467 ERROR [CommonClient] Exception caught while (preparing for) performing the invocation:
javax.xml.ws.soap.SOAPFaultException: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]
at org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS.getSOAPFaultException(SOAPFaultHelperJAXWS.java:84)
at org.jboss.ws.core.jaxws.binding.SOAP11BindingJAXWS.throwFaultException(SOAP11BindingJAXWS.java:107)
The Core Code for my test client:
Service service = Service.create(url, qname);
ImportLoan port = service.getPort(ImportLoan.class);
BindingProvider bp = (BindingProvider) port;
bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, properties.getProperty("username"));
bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, properties.getProperty("password"));
ImportLoanRequestType requestType = new ImportLoanRequestType();
requestType.setData(strEncodedPayload);
ImportLoanResponseType responseType = port.importLoanApp(requestType);
What I'm Hoping For
I just need some ideas to triage here. I assume I am getting some invalid SOAP response back that cannot be parsed correctly. I just find it odd that it only happens when sending the service from JBoss (my app server), so it must be something on my end. SOAPUI works fine. The exact same code written in a standalone test.java file works fine as well.
Update... later in the evening
Of course I would find the answer the same day I ended up posting the question. I watched the http traffic and realized the HTTP500 error response (with the soap eof prolog error) came back after half the data was sent. Apparently JBoss chunks the data. That worked fine against the 8080 service, but the port 80 service didn't support chunking for some reason. I assumed it was an authentication error, but appears to be that the web service doesn't support chunking. I modified the standard-jaxws-client-config.xml (jbossws.deployer/META-INF folder) on my server. Set the chunksize from 2048 to 0, and voila... problem solved. Hope this helps somebody else out there someday.