We Keep Coding

SSL Exceptions Seemingly caused by Apache Axis2

So my question is this: does axis2 create / override a global SSL factory that would impact other parts of a project that are communicating without using axis2?
I have a method making http calls out to an external page using org.apache.commons.httpclient.HttpClient (Below). It works. I send things out, and get responses back, everything is awesome.
PostMethod method = new PostMethod(url);
method.addRequestHeader("Content-Type", "application/json");
method.addRequestHeader("Authorization", "Bearer "+accessToken);
method.setRequestEntity(new StringRequestEntity(requestAsString, "application/json", "UTF-8"));
HttpClient client = new HttpClient();
client.getHttpConnectionManager().getParams().setSoTimeout(timeout);
int rCode = client.executeMethod(method);
I also have database connections using com.microsoft.sqlserver.jdbc. It works. Everything is still awesome.
The problems start with another service within the same project that uses axis2 (org.apache.axis2-1.6.2). After I attempt any communication using the axis 2 stub both the jdbc connection and the HttpClient connection both begin failing (The errors are below). Everything works fine up until axis2 gets involved.
I know axis2 is based on commons-httpclient-3.1 so the impact to the HttpClient piece isn't entirely unexpected but I cannot for the life of me figure out why I am seeing this behavior.
My assumption is that axis2 is setting some sort of global variable that is impacting ssl connections but having read the documentation and stepped through the source code, I can't find that happening anywhere.
The HttpClient error:
Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1917)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1874)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1857)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1783)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
at org.apache.commons.httpclient.methods.StringRequestEntity.writeRequest(StringRequestEntity.java:146)
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
... 10 more
Caused by: java.lang.RuntimeException: Could not generate DH keypair
at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:79)
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:696)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:277)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:936)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:871)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
... 16 more
Caused by: java.security.InvalidAlgorithmParameterException: parameter object not a ECParameterSpec
at org.bouncycastle.jce.provider.JDKKeyPairGenerator$EC.initialize(JDKKeyPairGenerator.java:345)
at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:74)
... 24 more
The jdbc error:
ERROR 13:27:33,321 [Thread-24] PID- M- TID- DAConnectionMgr_MSSQL -DB connection unavaliable to [master] as [Dev_User] failed on attempt [2], will automatically retry
com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "SQL Server returned an incomplete response. The connection has been closed. ClientConnectionId:a67d09bf-be47-4910-9d8c-fd040468a1cb".
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1667)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1668)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:1323)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:991)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:827)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1012)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.company.util.da.DAConnectionMgr_MSSQL.getConnection(DAConnectionMgr_MSSQL.java:299)
at com.company.baseserver.da.payments.product.productSSQL.openConnection(Product10DAMSSQL.java:499)
at com.company.baseserver.payments.product.ProductProcessor.process(ProductPaymentProcessor.java:988)
at com.company.baseserver.message.ProjectFunction.process(ProjectFunction.java:108)
at com.company.base.ClientProcessor.run(ClientProcessor.java:93)
at com.company.util.thread.PooledExecutor$Worker.run(PooledExecutor.java:774)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: SQL Server returned an incomplete response. The connection has been closed. ClientConnectionId:a67d09bf-be47-4910-9d8c-fd040468a1cb
at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.ensureSSLPayload(IOBuffer.java:651)
at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.readInternal(IOBuffer.java:708)
at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.read(IOBuffer.java:700)
at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.readInternal(IOBuffer.java:895)
at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.read(IOBuffer.java:883)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1618)
... 13 more

Jersey Server-Sent Events - write to broken connection does not throw exception

We are using Jersey Server-Sent Events (SSE) to allow remote components of our application to listen to events raised by our Jersey/Tomcat server. This works great.
However, it is crucial that our server have an accurate list of currently-connected listeners (our remote components). To this end, our server sends a tiny message to each caller (via eventOutput.write) once every five seconds. If our remote component is shut down while SSE-connected, or if the remote computer is powered off while SSE-connected, our server's eventOutput.write throws the ClientAbortException/SocketException exception shown below. That's perfect: we catch the exception, mark that caller as no longer connected, and move on.
Now, for the problem. As I mentioned, eventOutput.write throws an exception in cases where our remote component software is not running, or where the computer it runs on has been powered down. However, there are two cases where calling eventOutput.write to a no-longer-connected computer does NOT throw an exception: 1) if the Ethernet cable of the remote computer is simply pulled while the caller is SSE-connected, and 2) if the network adapter in the remote computer is turned off (i.e., by an administrative action) while the caller is SSE-connected. In these two cases, we can call eventOutput.write to the remote computer every five seconds for hours and no exception is thrown. This makes it impossible to detect that the remote computer is no longer connected.
I see that EventOutput (and ChunkedOutput) has very few methods and properties, but I wonder if there is any way to configure or use it that will cause an exception to be thrown when writing to a remote computer that has been disconnected by having its Ethernet cable pulled or network adapter turned off.
And here is the (good/useful) exception we get in cases where eventOutput.write DOES throw the exception we want:
org.apache.catalina.connector.ClientAbortException: null
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371) ~[catalina.jar:7.0.53]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) ~[catalina.jar:7.0.53]
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101) ~[catalina.jar:7.0.53]
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.flush(ResponseWriter.java:303) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.message.internal.CommittingOutputStream.flush(CommittingOutputStream.java:292) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.server.ChunkedOutput$1.call(ChunkedOutput.java:240) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.server.ChunkedOutput$1.call(ChunkedOutput.java:190) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.internal.Errors.process(Errors.java:242) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:347) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.server.ChunkedOutput.flushQueue(ChunkedOutput.java:190) ~[jaxrs-ri-2.13.jar:2.13.]
at org.glassfish.jersey.server.ChunkedOutput.write(ChunkedOutput.java:180) ~[jaxrs-ri-2.13.jar:2.13.]
at com.appserver.webservice.AgentSsePollingManager$ConnectionChecker.run(AgentSsePollingManager.java:174) ~[AgentSsePollingManager$ConnectionChecker.class:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_71]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[na:1.7.0_71]
at java.net.SocketOutputStream.write(SocketOutputStream.java:159) ~[na:1.7.0_71]
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215) ~[tomcat-coyote.jar:7.0.53]
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) ~[tomcat-coyote.jar:7.0.53]
at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119) ~[tomcat-coyote.jar:7.0.53]
at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799) ~[tomcat-coyote.jar:7.0.53]
at org.apache.coyote.Response.action(Response.java:174) ~[tomcat-coyote.jar:7.0.53]
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366) ~[catalina.jar:7.0.53]
... 19 common frames omitted

I do not think it will be possible to account for all the possible failures by adding the code around SSE sockets even if Jersey adds all information accessible from the socket interface. The only viable solution is proper two way communication. In case of SSE chunked output stream, pulled cable does not cause any interruption because nothing is supposed to tell it that remote host is now unreachable (until OS closes the socket).
Your first step is right - implement heartbeats every N seconds. Then all you need to do is to report back with another tiny http call every so often that you still listen. It is up to you to do acknowledgments every 5 seconds or every minute - depends on how fast do you need a problem detection.
You can do it in the same Jersey Resource by implementing #POST (in RESTful terms it reads "create new ack that you receive events").
Note: browsers are good at re-establishing SSE connection on their own in case of network interrupts, no need to fiddle with it.

app engine issue for java with frontend instances in production and blocking requests

My website is sometimes getting blocked and not working every few time (other times it works fine for a long while, with the same code), getting the log I show below. Each time this happens, I have to go to my console, refresh the cache, and change in application settings the Frontend Instance Class from F1 to F2 or F4 to make possible the first loading, and once it´s done, changing back to F1 (which works perfectly after the first request). But after a while, I got again the server error and my website doesn´t work.
...Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36" ms=60615 cpu_ms=41418 loading_request=1 exit_code=104 app_engine_release=1.8.4 instance=00c61b117cd6a25994d15ffe59bc99e20e5cb2bb
I 2013-09-06 17:06:12.201
javax.servlet.ServletContext log: Initializing Spring root WebApplicationContext
I 2013-09-06 17:06:50.336
javax.servlet.ServletContext log: Initializing Spring FrameworkServlet 'mvc-dispatcher'
E 2013-09-06 17:07:03.075
org.springframework.web.servlet.FrameworkServlet initServletBean: Context initialization failed
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/base/data/home/apps/s~comprarebookhoy/1.370036978402952713/WEB-INF/lib/objectify-4.0b2.jar!/com/googlecode/objectify/repackaged/gentyref/CaptureType.class]; nested exception is com.google.apphosting.api.DeadlineExceededException: This request (1de39e8e35c0b3f7) started at 2013/09/07 00:06:03.615 UTC and was still executing at 2013/09/07 00:07:03.073 UTC.
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:261)
at org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan(ClassPathBeanDefinitionScanner.java:242)
at org.springframework.context.annotation.ComponentScanBeanDefinitionParser.parse(ComponentScanBeanDefinitionParser.java:84)
at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:73)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1431)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1421)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:184)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:140)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:111)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:493)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:334)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:302)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:174)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:209)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:180)
at org.springframework.web.context.support.XmlWebApplicationContext.loadBeanDefinitions(XmlWebApplicationContext.java:125)
at org.springframework.web.context.support.XmlWebApplicationContext.loadBeanDefinitions(XmlWebApplicationContext.java:94)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:131)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:535)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:449)
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:638)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:595)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:514)
at org.springframework.web.servlet.FrameworkServlet.**initServletBean(FrameworkServlet.java:455)**
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:439)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:435)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.apphosting.api.**DeadlineExceededException: **This request (1de39e8e35c0b3f7) started at 2013/09/07 00:06:03.615 UTC and was still executing at** 2013/09/07 00:07:03.073 UTC.**
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:209)
at java.io.FilePermission.<init>(FilePermission.java:285)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.util.zip.ZipFile.<init>(ZipFile.java:217)
at java.util.zip.ZipFile.<init>(ZipFile.java:156)
at java.util.jar.JarFile.<init>(JarFile.java:153)
at java.util.jar.JarFile.<init>(JarFile.java:90)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:88)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
at org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:125)
at org.springframework.core.type.classreading.SimpleMetadataReader.<init>(SimpleMetadataReader.java:49)
at org.springframework.core.type.classreading.SimpleMetadataReaderFactory.getMetadataReader(SimpleMetadataReaderFactory.java:80)
at org.springframework.core.type.classreading.CachingMetadataReaderFactory.getMetadataReader(CachingMetadataReaderFactory.java:101)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:237)
... 48 more
E 2013-09-06 17:07:03.077
javax.servlet.ServletContext log: unavailable
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/base/data/home/apps/s~comprarebookhoy/1.370036978402952713/WEB-INF/lib/objectify-4.0b2.jar!/com/googlecode/objectify/repackaged/gentyref/CaptureType.class]; nested exception is com.google.apphosting.api.DeadlineExceededException: This request (1de39e8e35c0b3f7) started at 2013/09/07 00:06:03.615 UTC and was still executing at 2013/09/07 00:07:03.073 UTC.
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:261)
at org.springframework.context.annotation.ClassPathBeanDefinitionScanner.doScan(ClassPathBeanDefinitionScanner.java:242)
at org.springframework.context.annotation.ComponentScanBeanDefinitionParser.parse(ComponentScanBeanDefinitionParser.java:84)
at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:73)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1431)
at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1421)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:184)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:140)
at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:111)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:493)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:334)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:302)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:174)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:209)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:180)
at org.springframework.web.context.support.XmlWebApplicationContext.loadBeanDefinitions(XmlWebApplicationContext.java:125)
at org.springframework.web.context.support.XmlWebApplicationContext.loadBeanDefinitions(XmlWebApplicationContext.java:94)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:131)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:535)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:449)
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:638)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:595)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:514)
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:455)
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:439)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:435)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.apphosting.api.DeadlineExceededException: This request (1de39e8e35c0b3f7) started at 2013/09/07 00:06:03.615 UTC and was still executing at 2013/09/07 00:07:03.073 UTC.
at java.security.AccessController.doPrivileged(Native Method)
at java.io.FilePermission.init(FilePermission.java:209)
at java.io.FilePermission.<init>(FilePermission.java:285)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.util.zip.ZipFile.<init>(ZipFile.java:217)
at java.util.zip.ZipFile.<init>(ZipFile.java:156)
at java.util.jar.JarFile.<init>(JarFile.java:153)
at java.util.jar.JarFile.<init>(JarFile.java:90)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:88)
at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
at org.springframework.core.io.UrlResource.getInputStream(UrlResource.java:125)
at org.springframework.core.type.classreading.SimpleMetadataReader.<init>(SimpleMetadataReader.java:49)
at org.springframework.core.type.classreading.SimpleMetadataReaderFactory.getMetadataReader(SimpleMetadataReaderFactory.java:80)
at org.springframework.core.type.classreading.CachingMetadataReaderFactory.getMetadataReader(CachingMetadataReaderFactory.java:101)
at org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider.findCandidateComponents(ClassPathScanningCandidateComponentProvider.java:237)
... 48 more
W 2013-09-06 17:07:03.122
(my url)
javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$CancelledException: **The API call memcache.Get() was explicitly cancelled**.
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:268)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:439)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:435)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.apphosting.api.ApiProxy$CancelledException: The API call memcache.Get() was explicitly cancelled.
at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:386)
at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:383)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28)
at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51)
at com.google.apphosting.runtime.jetty.MemcacheSessionStore.getSession(MemcacheSessionStore.java:36)
at com.google.apphosting.runtime.jetty.SessionManager.loadSession(SessionManager.java:327)
at com.google.apphosting.runtime.jetty.SessionManager.getSession(SessionManager.java:301)
at org.mortbay.jetty.servlet.AbstractSessionManager.getHttpSession(AbstractSessionManager.java:237)
at org.mortbay.jetty.servlet.SessionHandler.setRequestedId(SessionHandler.java:246)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:136)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266)
... 16 more
W 2013-09-06 17:07:03.123
Nested in javax.servlet.ServletException: com.google.apphosting.api.ApiProxy$CancelledException: The API call memcache.Get() was explicitly cancelled.:
com.google.apphosting.api.ApiProxy$CancelledException: The API call memcache.Get() was explicitly cancelled.
at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:386)
at com.google.apphosting.runtime.ApiProxyImpl$5.get(ApiProxyImpl.java:383)
at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28)
at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51)
at com.google.apphosting.runtime.jetty.MemcacheSessionStore.getSession(MemcacheSessionStore.java:36)
at com.google.apphosting.runtime.jetty.SessionManager.loadSession(SessionManager.java:327)
at com.google.apphosting.runtime.jetty.SessionManager.getSession(SessionManager.java:301)
at org.mortbay.jetty.servlet.AbstractSessionManager.getHttpSession(AbstractSessionManager.java:237)
at org.mortbay.jetty.servlet.SessionHandler.setRequestedId(SessionHandler.java:246)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:136)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:439)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:435)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:442)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:186)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:306)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:298)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:439)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:722)
I 2013-09-06 17:07:03.131
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
W 2013-09-06 17:07:03.131
A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)
In my local environment I don´t get this problem. Looks like some kind of issues between the spring framework and the cache used by gae, but no idea what´s going on here... any suggestions please?
It also looks to be related with the front end instances, i think i have always one after the first request, but for some reason, at some moments goes shutdown and i have to change again to the F4 option to make it work, otherwise I got server error.
And from the logs, it´s possible objectify is making some kind of exception provoking the issue, i´m using version 4.0b2
I found also a related bug in app engine for the part: "ApiProxy$CancelledException: The API call memcache.Get() was explicitly cancelled."
https://code.google.com/p/googleappengine/issues/detail?id=7107

It looks like your app just takes too long to start. You're hitting the 60s deadline.
It takes 10X longer for your app to start on an F1 than it does on your desktop. In prod, not only is the server loading classes over a slow network, but F1 instances are really underpowered - your cellphone has more available CPU.
Your app is just on the edge of hitting the 60s deadline. When you switch to F2/F4 instances, the extra CPU power puts you under the deadline and your app starts. Changing the instance size doesn't immediately resize your instances; it only affects new instances that start up. So when you change back to F1, eventually GAE starts up a new instance and you're back to the same deadline problem.
Note that it's always dangerous to be "on the edge" because GAE performance is erratic. Variance can be up to 3X; it's dangerous to take more than 20-25s to start up.
Unfortunately there's no magic solution to this other than "optimize your startup". Go through it with a profiler. It will not be accurate because running locally does not give you the same behavior as you get server-side (as you've noticed, you don't get the same slowdowns) but it's a start. Avoid eager classloading if you can - although this is tricky with DI frameworks. And make sure to avoid doing any prefetching of data from remote sources (including the datastore or memcache) at startup.
One way you can help mitigate the problem (aside from using F2 or F4 instances) is to create a cron job that pings your app once per second. Cron tasks get a 10m deadline instead of 60s. At the very least, you'll get one instance off the ground.

Well what I can infer from your stacktrace is that there is process which is running inside your appengine application which the call any other process or may be some third party process and for that the timeout for the execution of the later process exceeds hence you are getting this exception.
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL
**[jar:file:/base/data/home/apps/s~comprarebookhoy/1.370036978402952713/WEB-INF/lib/objectify-4.0b2.jar!/com/googlecode/objectify/repackaged/gentyref/CaptureType.class];**
nested exception is com.google.apphosting.api.DeadlineExceededException:
This request (1de39e8e35c0b3f7) started at 2013/09/07 00:06:03.615 UTC and was still executing at 2013/09/07 00:07:03.073 UTC
There is no any problem with Objectify Library, Spring as well as with Appengine. It is the problem inside your code which is causing this exception to occur whenever that particular process comes under execution.
Also try to check in your appengine dashboard that what instances of appengine usage is high by your application. Also you can check many more things in your appengine dashboard itself.
Also check the last few lines of your stacktrace. It clearly explains the cause of this error.
I 2013-09-06 17:07:03.131
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
W 2013-09-06 17:07:03.131
A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

JAX-WS RI Metro 2.1.3 - SocketTimeout exception

I have developed a web service that is used to upload files on server, it has been created using Metro 2.1 JAX-WS engine and deployed on web sphere 7.0. I can connect to this service and transfer files successfuly but sometimes it throws java.net.SocketTimeoutException: Async operation timed out.
Following is the server stack trace -
It can be because of the bad n/w problem and files getting uploaded are also of reasonable size so we decided to increase the timeout value to 5 mins but i am not able to increase it after trying a lot of code changes. I have tried following methods to increase timeout values but nothing seems to work.
I have tried to set timeout values to '1' to test if timeout happens but web service seemed to be rolling fine without timeouts.
I would appreciate a quick help here.
Thanks,
((BindingProvider)port).getRequestContext().put(BindingProviderProperties.CONNECT_TIMEOUT, 1);
((BindingProvider)port).getRequestContext().put(BindingProviderProperties.REQUEST_TIMEOUT, 1);
AND
((BindingProvider)port).getRequestContext().put("com.sun.xml.ws.request.timeout", 100);
((BindingProvider)port).getRequestContext().put("com.sun.xml.ws.connect.timeout", 100);
AND
((BindingProvider)port).getRequestContext().put("com.sun.xml.internal.ws.request.timeout", 1);
((BindingProvider)port).getRequestContext().put("com.sun.xml.internal.ws.connect.timeout", 1);
java.net.SocketTimeoutException: Async operation timed out
at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:189)
at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.read(SSLReadServiceContext.java:265)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4142)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3374)
at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3494)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1705)
at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:373)
at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:266)
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:325)
at org.jvnet.mimepull.MIMEParser.fillBuf(MIMEParser.java:434)
at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:204)
at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:62)
at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:154)
at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:123)
at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:193)
at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:236)
at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:207)
at java.io.InputStream.read(InputStream.java:97)
at javax.activation.DataHandler.writeTo(DataHandler.java:307)
at com.ncr.java.service.ValidateRequest.validateCaptureFileContent(ValidateRequest.java:357)
at com.ncr.java.service.CaptureFileReceiverImpl.uploadCaptureFile(CaptureFileReceiverImpl.java:167)
at com.ncr.java.service.CaptureFileReceiverImplDelegate.uploadCaptureFile(CaptureFileReceiverImplDelegate.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:95)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:629)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:588)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:573)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:470)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:295)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:515)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:285)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:143)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:155)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:189)
at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:76)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:182)
... 64 more
Reporter BEGIN:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream#447e447e
java.io.InputStream::SKIP_BUFFER_SIZE:2048
java.io.InputStream::skipBuffer:null
com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream::logger:com.ibm.wsspi.webcontainer.logging.WebContainerLogger#2d512d51
com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream::CLASS_NAME:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream
com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream::isc BEGIN:com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl#4b574b57
com.ibm.ws.http.channel.impl.HttpServiceContextImpl::tc BEGIN:com.ibm.ejs.ras.TraceComponent#4b344b34
com.ibm.ejs.ras.TraceElement::ivLevel:10
com.ibm.ejs.ras.TraceElement::ivName:com.ibm.ws.http.channel.impl.HttpServiceContextImpl
com.ibm.ejs.ras.TraceElement::ivDebugEnabled:false
com.ibm.ejs.ras.TraceElement::ivEventEnabled:false
com.ibm.ejs.ras.TraceElement::ivEntryEnabled:false

The stacktrace says that webservice on server gets timeout when reading data from client.
In conrast setting timeout props on getRequestContext() affects ws-client when waiting response from server.
It seems that very low client timeout test "succeeded". Client got a timeout and aborted request too early. That caused timeout on server side when trying to analyse request's mime type.
Try increasing client timeout to desired value.

Axis 2 generated code in "Expanded Mode"

I'm having some problems understanding how to send a request to a web service that I have deployed. I have followed the following tutorial:
http://wso2.org/library/1719
Which I have successfully applied, and it worked well. The tutorial was straightforward, and I have easily generated a web service client. All I did was creating "request" objects from the exposed web service method, set its arguments, and then using the web service stub, I have passed the request object to the exposed web service method, and got a response.
Now, I have received a WSDL file from which I'm supposed to build a web service client as well. The problem is that the generated files from the WSDL differ greatly from that of the tutorial I followed. I learned later on that when generating clients using ADB, there is a mode called "Expanded Mode" that generates " a class for each of the outer elements and the named complex types " (http://axis.apache.org/axis2/java/core/docs/adb/adb-howto.html#gen_modes). The problem is that I'm unable to understand how to create a request object and get a response from the web service from the generated classes. My best shot was as follows:
GetMfrInventoryProductItemsListE wrapper = new GetMfrInventoryProductItemsListE();
GetMfrInventoryProductItemsList request = new GetMfrInventoryProductItemsList();
request.setArg0(deviceId);
request.setArg1(macAddresses);
request.setArg2(mfrKey);
wrapper.setGetMfrInventoryProductItemsList(request);
GetMfrInventoryProductItemsListResponseE wrapperResponse = wmService.getMfrInventoryProductItemsList(wrapper);
GetMfrInventoryProductItemsListResponse response = wrapperResponse.getGetMfrInventoryProductItemsListResponse();
return response.get_return();
In the above code getGetMfrInventoryProductItemsList is the name of my exposed web service method, and wmService is my stub.
The result was the following stack Trace :
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:197)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:404)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:231)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:443)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:406)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at wavemark.wmservice.soap.WMServiceStub.getMfrInventoryProductItemsList(WMServiceStub.java:3275)
at wavemark.linkserver.supplierconnect.controller.WsStub.getMfrInventoryProductItemsList(WsStub.java:107)
at wavemark.linkserver.supplierconnect.controller.WsManager.getProductItems(WsManager.java:384)
at wavemark.linkserver.supplierconnect.webservices.SupplierConnect.getProductItems(SupplierConnect.java:396)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:181)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:621)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
... 42 more
After trying to debug the problem, I have found out that this is the line that is throwing the exception:
GetMfrInventoryProductItemsListResponseE wrapperResponse = wmService.getMfrInventoryProductItemsList(wrapper);
I just want you guys to tell me if the way I'm using the request/response objects is correct.Your help is much appreciated.

Finally, I figured out how to solve the problem. Apparently the web service call that I was making, was taking a lot of time that it was timing out as the stack trace shows. So I added the following to my code to the section where I was instantiating my client stub:
wmService = new WMServiceStub();
Options options = new Options();
EndpointReference epr = new EndpointReference(url);
options.setTo(epr);
options.setProperty(HTTPConstants.SO_TIMEOUT, new Integer(300000));
wmService._getServiceClient().setOptions(options);
This results in increasing the Axis 2 web services' socket timeout to 5 minutes, which should be more than enough to most web service calls. I hope that this will be helpful for some developers out there.