We have Spring Boot applications which use Kubernetes and Istio. The application connects to AWS OpenSearch for CRUD functionality. Recently, the requests to OpenSearch started failing and we encountered this error repeatedly in the application logs:
method [POST], host [http://vpc-application-name.us-east-1.es.amazonaws.com], URI [/dummy/uri?param1=dummy1¶m2=dummy2], status line [HTTP/1.1 503 Service Unavailable] upstream connect error or disconnect/reset before headers. reset reason: connection failure
When we tried to hit this URL through Postman, it was working. We also tried to bash into the Kubernetes pods of this service and tried to curl this URL from there, but we were getting proper response. So it looks like an issue from the Java side. When we restarted the pods, the error was gone.
What could have caused this error?
How to identify or narrow down the root cause of this issue?
I am fairly new to Istio, Kubernetes and AWS, and I do not know what to expect. Any help is appreciated.
When I start up Solr, I get the following error
I/O exception caught when connecting to https://localhost:;
java.security.NoSuchAlgorithmException:
Error construct:sun.security.ssl.SSLContextImpl$DefaultSSLContext
Did anyone ever come across the same issue before?
What is causing this error and how can I fix it?
Try to connect using http://localhost. You are using https and it requires proper certificates and extra configuration. For localhost http is ok but is high recommend that you use https on production.
I am using nexus 2.10 having recently upgraded from nexus 2.8. The reason I upgraded was because I was getting the following exception in my logs and thus the jar was not downloaded and Nexus 2.10 gives a few more details in the log regarding the broken pipe error.
So in my logs I get the following error (replaced i[p address with string host name).
2014-10-21 15:18:22,556+0100 INFO [qtp-855494215-66] anonymous org.sonatype.nexus.proxy.storage.remote.httpclient.HttpClientRemoteStorage - Initializing remote transport for proxy repository "Central" [id=central]...
2014-10-21 15:23:59,440+0100 WARN [qtp-855494215-69] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx:8081/nexus/content/groups/public/org/springframework/spring-jdbc/4.1.1.RELEASE/spring-jdbc-4.1.1.RELEASE.jar]
2014-10-21 15:24:43,329+0100 WARN [qtp-855494215-67] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx5:8081/nexus/content/groups/public/org/springframework/spring-tx/4.1.1.RELEASE/spring-tx-4.1.1.RELEASE.jar]
2014-10-21 15:24:58,333+0100 WARN [qtp-855494215-66] anonymous org.sonatype.nexus.content.internal.ContentServlet - org.eclipse.jetty.io.EofException, caused by: java.io.IOException: Broken pipe [client=host.name,ua=m2e/3.9.1.v20130814-1242/1.4.0.20130601-0317/1.6.0_45,req=GET http://xx.xx:8081/nexus/content/groups/public/org/springframework/spring-context/4.1.1.RELEASE/spring-context-4.1.1.RELEASE-sources.jar]
Can anyone advise how to get past this error because currently some jars are not being downloaded. This does not apply to all jars for eg i just downloaded the nekohtml-0.9.5.jar no issues. Could this be possibly something to do with size? In my maven repository I just get a spring-test-4.1.1.RELEASE.jar.lastUpdated jar which is only 1kb in size.
Also I am running behind a firewall so have a proxy server configured in the nexus settings.
UPDATE:
this seems to be an issue only when done through eclipse. I was able to download sources and jars etc fine when I did it from the command line. I am using Eclipse keplar and the m2e plugin. Can anyone comment on this?
All advice appreciated
Thanks
The broken pipe exception occurs when nexus has received an http request from a client, prepares a response, but when it goes to write the response it finds the socket has already been closed.
Likely causes of this are:
An end user initiating a request and then canceling it (or
navigating away from a web page before it completes)
A client which does not properly close socket connections
A client's read timeout setting being reached
A reverse proxy in between nexus and the client (such as apache or
nginx) terminating the connection
Therefore you'll need to begin your investigation on the client side. If it turns out the client or reverse proxy's read timeout setting is triggering this then you'll need to investigate why nexus is responding slowly.
Your Nexus server is probably not able to reach some external repositories if it is running inside a network behind a proxy. You have to make sure you have the http proxy configured in Nexus and that any URL's of remote proxy repositories you want to access are allowed in the proxy.
You can use the "Browse Remote" feature to check if the access works. More details are in the Nexus book http://books.sonatype.com/nexus-book/reference/configxn-sect-customizing-server.html
I have a problem creating and connecting a Java client to a running Web Service.
I use the following code:
Service myService = null;
URL wsdlLocation = new URL("http://myservice?wsdl");
QName serviceName = new QName(wsdlLocation, "MyService");
Service myService = new Service(wsdlLocation, serviceName);
where the Service class was created with the following command:
wsimport -d gen -keep http://myservice?wsdl
I've also tried with a client generated by Apache cxf 2.4's wsdl2java, but got the same result.
(I've changed the WSDL location and Service class name only for this post, in the code I use the original ones.)
But I get an exception when I call the Web Service deployed on the appserver, when creating the service with the new Service() command.
But: I've tested the wsdl location with SOAP UI and it works perfectly.
Also, I've created a Mock Service using Soap UI and my Java client could connect to it, call it and get the results back.
The problem appears when I want' to call the web service running on the appserver.
Stacktrace:
javax.xml.ws.WebServiceException: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
at org.apache.cxf.jaxws.ServiceImpl.<init>(ServiceImpl.java:149)
at org.apache.cxf.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:98)
at javax.xml.ws.Service.<init>(Service.java:76)
at MyService.<init>(MyService.java:42)
at mypackage.createService(AClass.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
Caused by: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.
at org.apache.cxf.wsdl11.WSDLServiceFactory.<init>(WSDLServiceFactory.java:100)
at org.apache.cxf.jaxws.ServiceImpl.initializePorts(ServiceImpl.java:199)
at org.apache.cxf.jaxws.ServiceImpl.<init>(ServiceImpl.java:147)
... 12 more
Caused by: javax.wsdl.WSDLException: WSDLException: faultCode=PARSER_ERROR: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col,system-id]: [1,0,"http://myservice?wsdl"]
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:256)
at org.apache.cxf.wsdl11.WSDLManagerImpl.getDefinition(WSDLManagerImpl.java:205)
at org.apache.cxf.wsdl11.WSDLServiceFactory.<init>(WSDLServiceFactory.java:98)
... 14 more
Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col,system-id]: [1,0,"http://myservice?wsdl"]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:677)
at com.ctc.wstx.sr.BasicStreamReader.handleEOF(BasicStreamReader.java:2139)
at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2045)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1248)
at org.apache.cxf.staxutils.StaxUtils.readDocElements(StaxUtils.java:1142)
at org.apache.cxf.staxutils.StaxUtils.read(StaxUtils.java:1069)
at org.apache.cxf.wsdl11.WSDLManagerImpl.loadDefinition(WSDLManagerImpl.java:247)
... 16 more
Could someone please help me?
I encountered this error and found that it appeared to be due to using a URL that returned an HTTP 302 redirect instead of the WSDL directly.
The URL I was using was in the format /Service?wsdl, which redirected to a URL in the format /Service/wsdl/Service.wsdl. Once I used the redirect target URL directly, everything worked.
I remember reading that it could be related to the endpoint expecting a trailing '/'. I'm not sure if this valid, but please try it and post here it if works.
I had a similar error and when I checked the server logs - it was related to http server having encountered an unsupported http method in request. Due to this teh server returns an HTTP response that the SOAP client cannot handle ... hence Unexpected EOF in prolog
Below is a snippet my web-server log for tomcat "localhost.XXXX.log"
org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods For security constraints with URL pattern [/services/*] only the HTTP methods [POST GET] are covered. All other methods are uncovered.
This gave following error on client side
com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
In my case since I was downgrading a secured service to unsecured and my web.xml had following stray entry that was restricting GET and post with a security constraint and I wasnt sending needed security parameters in the request..
<security-constraint>
<web-resource-collection>
<web-resource-name>restricted web services</web-resource-name>
<url-pattern>/services/*</url-pattern>
<http-method>GET</http-method>
<http-method>POST</http-method>
</web-resource-collection>
I removed this constraint to get rid of this error :-)
In your case it may not related to security constraint - but for sure its related to server sending raw http response - Please check your server/ client configuration and ensure it sends appropriate http request that are liked by http server
I had a similar error and when I checked the server logs - it was related to http server having encountered an unsupported http method in request returns an HTTP response that the SOAP client cannot handle ... check your web-server logs
Surely this isn't due to character or format issue.
It's possible that the server didn't return any data, of which the Woodstox (wstx) was trying to parse into XML and failed, which resulted to this error.
I had similiar issue and i have resolved with debugging.
When i try to download WSDL in runtime i found that i could not get the WSDL because of the proxy.
Please check that you can access the WSDL in runtime like this code:
try {
String wsdl = org.apache.commons.io.IOUtils.toString(new URL("http://YOUR_WSDL_URL"));
System.out.println("WSDL => "+wsdl);
} catch (IOException e) {
e.printStackTrace();
}
One of the imports are missing. Please check routes to your xsd's.
I had the same problem. I had to use the full path to the WSDL file to make it works.
My contribution to this error is this:
I don't get this error when I am in debugging mode.
It seems that when the executor of the Java program that queries the WSDL Service (Java in my case) has the time to perform many requests, like in debugging mode, because of when sometimes the programs stops because I put some breakpoints, in this case I don't get this error.
When I run the program in production mode, yes.
But the requests are the same, the WSDL doesn't change, as well ad the endpoints.
It's like a limitation of the web Server, I think.
I am getting the following exception, when i am trying to hit the webservice over https using java client:
com.sun.xml.internal.messaging.saaj.client.p2p.HttpSOAPConnection post
SEVERE: SAAJ0009: Message send failed
com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: java.security.PrivilegedActionException: com.sun.xml.internal.messaging.saaj.SOAPExceptionImpl: Message send failed
at com.sun.xml.internal.messaging.saaj.client.p2p.HttpSOAPConnection.call(HttpSOAPConnection.java:157)
at
Also if i use the same code for http , it works fine. Webservices are deployed on weblogic server.
Could anyone help me in resolving the issue.
Regards
Gurleen.
The exception says that message cannot be sent.
Is it a firewall problem like for the same question here : http://www.coderanch.com/t/223309/Web-Services/java/Sending-Soap-requests-thru-Https