I have an application that is attempting to call a service and the other service appears to be timing out. The problem is my application does not receive any timeout exceptions, although I do see an error printed out to the console:
[7/8/13 12:39:32:360 EDT] 00000005 TimeoutManage I WTRN0006W: Transaction 0000013FBF252E43000000010000000CE81CB4935851D5C13DECD3DBB2D463F0DBECAEE60000013FBF252E43000000010000000CE81CB4935851D5C13DECD3DBB2D463F0DBECAEE600000001 has timed out after 120 seconds.
[7/8/13 12:39:32:360 EDT] 00000005 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WebContainer : 1,5,main]. The stack trace of this thread when the timeout occurred was:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:334)
com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:182)
com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.parseResponseMessageSync(HttpOutboundServiceContextImpl.java:1657)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.readSyncResponse(HttpOutboundServiceContextImpl.java:725)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.startResponseReadSync(HttpOutboundServiceContextImpl.java:1775)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.finishRequestMessage(HttpOutboundServiceContextImpl.java:1195)
com.ibm.ws.websvcs.transport.http.out.HttpOutSyncWriter.finishBufferRequest(HttpOutSyncWriter.java:94)
com.ibm.ws.websvcs.transport.http.out.HttpOutWriter.writeBuffer(HttpOutWriter.java:136)
com.ibm.ws.websvcs.transport.http.out.HttpOutByteBufferOutputStream.finish(HttpOutByteBufferOutputStream.java:468)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendChunkedRequest(SOAPOverHTTPSender.java:890)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendSOAPRequest(SOAPOverHTTPSender.java:807)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.send(SOAPOverHTTPSender.java:611)
com.ibm.ws.websvcs.transport.http.HTTPTransportSender.invoke(HTTPTransportSender.java:364)
org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:531)
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:401)
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:228)
org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)
org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.execute(AxisInvocationController.java:581)
org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.doInvoke(AxisInvocationController.java:130)
org.apache.axis2.jaxws.core.controller.impl.InvocationControllerImpl.invoke(InvocationControllerImpl.java:93)
org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invokeSEIMethod(JAXWSProxyHandler.java:364)
org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invoke(JAXWSProxyHandler.java:185)
The client is created with these settings:
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.RESPONSE_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.CONNECTION_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.READ_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
MyService.TIME_OUT has a value of 20000 and I have verified that it is being set correctly.
The code that catches calls the service looks like this:
try
{
response = ((MyServicePortType) myService).doWebServiceOperation(request);
}
catch (Throwable e) //I know, catch Throwable is not very good but right now I'd be happy to catch ANYthing here!
{
log.error("Webservice reported error",e);
}
Even though I've changed my catch block to catch anything, I still don't catch any exceptions. WebSphere detects a transaction timeout, but I don't know why the application doesn't detect a timeout in the web service call. Is there something I'm missing that would cause a proper timeout exception to be thrown so that I can catch it and send the message to the application framework?
Well now I feel silly.
It seems that for WebSphere, these properties (RESPONSE_TIMEOUT_PROPERTY, CONNECTION_TIMEOUT_PROPERTY, etc...) should have their values set in seconds, and I was using milliseconds based on what I'd seen in online examples (that clearly were not intended for WebSphere).
Changing 20000 to 20 has resolved this problem.
The page that suggested I should be assuming seconds instead of milliseconds is this one: http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/rwbs_httptransportprop.html
Related
Context
I have a webservice writing a test id in a queue. Then, a listener reads the queue, searches the test and starts it. During those steps, it writes updates of the test in the database in order to be shown to the user. To be more precise: the test is launched in a docker container and at the end of it, I want to update the status of the test to FINISHED. For that, I use the docker java library with a callback.
Problem
When it comes to call the callback, I receive multiple error messages on the call to update the test (but it happens only once, if I try twice the second time works, but it still writes a lot of error messages from the transaction manager).
Here are the error messages logged:
2020-11-20 09:20:43,639 WARN [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#600268e6[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69 connection handles=1 lastReturned=1605860423264 lastValidated=1605860242146 lastCheckedOut=1605860443564 trackByTx=true pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#2efb0d3b mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#3e8e7a62[pool=ApplicationDS] xaResource=LocalXAResourceImpl#482fdad2[connectionListener=600268e6 connectionManager=4c83f895 warned=false currentXid=null productName=Oracle productVersion=Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0 jndiName=java:/ApplicationDS] txSync=TransactionSynchronization#1387480544{tx=Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4 status: ActionStatus.ABORTING >, owner=Local transaction context for provider JBoss JTA transaction provider) wasTrackByTx=true enlisted=true cancel=false}]: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
2020-11-20 09:20:43,647 INFO [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000302: Unregistered handle that was not registered: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#4f3c1cb2 for managed connection: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69
2020-11-20 09:20:43,656 WARN [docker-java-stream--1032099154] (com.arjuna.ats.jta) ARJUNA016031: XAOnePhaseResource.rollback for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4, node_name=1, branch_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db8, subordinatenodename=null, eis_name=java:/ApplicationDS > failed with exception: org.jboss.jca.core.spi.transaction.local.LocalXAException: IJ001160: Could not rollback local transaction
Caused by: org.jboss.jca.core.spi.transaction.local.LocalResourceException: IO Error: Socket read interrupted
at org.jboss.ironjacamar.jdbcadapters#1.4.22.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnection.rollback(LocalManagedConnection.java:139)
...
Caused by: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
at com.oracle.jdbc//oracle.jdbc.driver.T4CConnection.doRollback(T4CConnection.java:1140)
...
Caused by: java.io.InterruptedIOException: Socket read interrupted
at com.oracle.jdbc//oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:258)
...
Explanation
At the beginning, I thought about a connection problem, maybe the the transaction is no more available at the callback time (because the docker run took too long), maybe it has to be invalidated.
But at the end, as written in the console, it came from an interruption of the thread when it tries to acquire the lock to update the test and I discovered where this interruption came from: I took a look at the method executeAndStream in DefaultInvocationBuilder from the docker java library and I discovered this:
Thread thread = new Thread(() -> {
Thread streamingThread = Thread.currentThread();
try (DockerHttpClient.Response response = execute(request)) {
callback.onStart(() -> {
streamingThread.interrupt();
response.close();
});
sourceConsumer.accept(response);
callback.onComplete();
} catch (Exception e) {
callback.onError(e);
}
}, "docker-java-stream-" + Objects.hashCode(request));
thread.setDaemon(true);
thread.start();
And here it is, the closable given to onStart interrupts the thread. After that, I discovered in the method onComplete from ResultCallbackTemplate (that I was extending for my callback) a close on that closable:
#Override
public void onComplete() {
try {
close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Resolution
The problem finally came from the following code I wrote:
#Override
public void onComplete() {
super.onComplete();
updateTest(FINISHED);
}
I was calling the onComplete method from the parent without knowing what is does and as usual, first before doing anything else. To correct that, I only had to call the super method at the end:
#Override
public void onComplete() {
updateTest(FINISHED);
super.onComplete();
}
I have an odd issue with my Tomcat + Spring websocket application. When a user disconnects without sending a "closing" signal, due to power loss or a plug pull, the thread will block about 10 seconds later.
The thread blocks on this function :
org.springframework.web.socket.WebSocketSession.sendMessage(WebSocketMessage<?> wsm) throws IOException;
I have tried putting a line in my AppConfig to try and set a timeout of 3 seconds but it does not seem to work properly as the block seems to go on for upwards of 15 minutes before throwing an exception.
#Bean(name="servletServerContainerFactoryBean")
public int maxSessionIdleTimeout() {
return 3000;
}
Here is the stack trace after an eventual SocketTimeoutException
Step: 2304
SendB -> test isOpen -> sendMes -> Done -> Finished Send.
SendB -> test2 isOpen -> sendMes -> User closed connection during packet sending: s01
Propogating exception up!
java.io.IOException: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:315)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendPartialString(WsRemoteEndpointImplBase.java:223)
at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:49)
at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:197)
at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:102)
at org.infpls.noxio.game.module.game.session.NoxioSession.sendPacket(NoxioSession.java:40)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby.step(GameLobby.java:117)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby$GameLoop.run(GameLobby.java:274)
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:494)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:309)
... 8 more
## CRITICAL ## Ejecting player: test2 :: Exception thrown on packet send...
java.io.IOException: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:315)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendPartialString(WsRemoteEndpointImplBase.java:223)
at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:49)
at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:197)
at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:102)
at org.infpls.noxio.game.module.game.session.NoxioSession.sendPacket(NoxioSession.java:40)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby.step(GameLobby.java:117)
at org.infpls.noxio.game.module.game.dao.lobby.GameLobby$GameLoop.run(GameLobby.java:274)
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:81)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:494)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:309)
... 8 more
Finished Send. Step Finished.
Having threads be blocked for 15 minutes at a time is a major problem. Any info on why this happens and how to fix it would be greatly appreciated. Thank you!
Found an answer finally. It's actually a system setting.
Source
How many times to retry before deciding that something is wrong and
it is necessary to report this suspicion to network layer. Minimal RFC
value is 3, it is default, which corresponds to 3sec-8min depending on
RTO.
/proc/sys/net/ipv4/tcp_retries2
This question already has answers here:
What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
(4 answers)
Closed 5 years ago.
I get the following error while running my spark streaming application, we have a large application running multiple stateful (with mapWithState) and stateless operations. It's getting difficult to isolate the error since spark itself hangs and the only error we see is in the spark log and not the application log itself.
The error happens only after abount 4-5 mins with a micro-batch interval of 10 seconds.
I am using Spark 1.6.1 on an ubuntu server with Kafka based input and output streams.
Please note it's not possible for me to provide the smallest possible code to re-create this bug as it does not occur in unit test-cases, and the application itself is very large
Any direction you can give to solve this issue will be helpful. Please let me know if I can provide any more information.
Error inline below:
[2017-07-11 16:15:15,338] ERROR Error cleaning broadcast 2211 (org.apache.spark.ContextCleaner)
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:136)
at org.apache.spark.broadcast.TorrentBroadcast$.unpersist(TorrentBroadcast.scala:228)
at org.apache.spark.broadcast.TorrentBroadcastFactory.unbroadcast(TorrentBroadcastFactory.scala:45)
at org.apache.spark.broadcast.BroadcastManager.unbroadcast(BroadcastManager.scala:77)
at org.apache.spark.ContextCleaner.doCleanupBroadcast(ContextCleaner.scala:233)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$2.apply(ContextCleaner.scala:189)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$2.apply(ContextCleaner.scala:180)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:180)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1180)
at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:173)
at org.apache.spark.ContextCleaner$$anon$3.run(ContextCleaner.scala:68)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
Your exception message clearly says that its RPCTimeout due to default configuration of 120 seconds and adjust to optimal value as per your work load.
please see 1.6 configuration
your error messages org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds].
and
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76) confirms that.
For Better understanding please see the below code from
see RpcTimeout.scala
/**
* Wait for the completed result and return it. If the result is not available within this
* timeout, throw a [[RpcTimeoutException]] to indicate which configuration controls the timeout.
* #param awaitable the `Awaitable` to be awaited
* #throws RpcTimeoutException if after waiting for the specified time `awaitable`
* is still not ready
*/
def awaitResult[T](awaitable: Awaitable[T]): T = {
try {
Await.result(awaitable, duration)
} catch addMessageIfTimeout
}
}
Also see my answer in another context
my java code is like:
logger.info("start");
getJdbcTemplate().execute("call " + procedureName + "()");
and I got the exception:
org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [call PRMI_UPDATE_USER_LOGIN_INFO()]; Io ERROR: Connection reset; nested exception is java.sql.SQLException: Io ERROR: Connection reset
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:257)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:428)
Maybe it's caused by the long time waiting. I found that it printed "start" in log and after about 5 minutes I got the exception.
update at 2013-03-13:
I got that exception not only at calling oracle stored procedure but at druid's 'JdbcUtil.close(...)':
com.alibaba.druid.util.JdbcUtils.close:81 - close connection error
java.sql.SQLRecoverableException: Io Error: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:521)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:500)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3509)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:167)
at com.alibaba.druid.filter.stat.StatFilter.connection_close(StatFilter.java:254)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:163)
at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.close(ConnectionProxyImpl.java:115)
at com.alibaba.druid.util.JdbcUtils.close(JdbcUtils.java:79)
at com.alibaba.druid.pool.DruidDataSource.shrink(DruidDataSource.java:1876)
at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:1694)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:150)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:180)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:169)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1010)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:97)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:487)
The druid's JdbcUtil.close method is quite simple:
public static void close(Connection x) {
if (x == null) {
return;
}
try {
x.close();
} catch (Exception e) {
LOG.debug("close connection error", e);
}
}
the source code is :
https://github.com/alibaba/druid/blob/master/src/main/java/com/alibaba/druid/util/JdbcUtils.java
It should wait as long as it is needed. Forget about various hacks which try to "detect" deadlock based on timeout delay.
you should find also some ORA-XXXX error. Io ERROR: Connection reset does not look like Oracle error message, there should be some error number attached to it
the timeout 5 minutes is very strange value. Theoretically this can be setup also on database side. As profile parameter CPU_PER_CALL but in such a case you should get an error: ORA-02393: exceeded call limit on CPU usage. And you connection should NOT be lost
theoretically you can also have problems which dead connection detection, but 5 minutes timeout is too short for that
another possible source can be ORA-600 error. Oracle internal error, maybe your session process crashed and therefore TCP connection was lost
you should contact your local DBAs and ask then for cooperation. They should help you better than anonymous people on the Internet forum.
Maybe it's caused by the long time waiting
No it is not caused due to that
As Java Doc says about DataAccessResourceFailureException
Data access exception thrown when a resource fails completely: for
example, if we can't connect to a database using JDBC.
Occasionally I see this exception. It affects some of the crucial business processes in the application. What could it be? Did anyone have similar error?
I use WebSphere 6 with OJDBC14 & Hibernate 3.
[21.02.14 06:46:03:209 PST] 00000031 MCWrapper E J2CA0081E: Method cleanup failed while trying to execute method cleanup on ManagedConnection WSRdbManagedConnectionImpl#4d34b403 from ressource jdbc/MYDS. Caught exception: com.ibm.ws.exception.WsException: DSRA0080E: An exception was received by the Data Store Adapter. See original exception message: Cannot call 'cleanup' on a ManagedConnection while it is still in a transaction..
at com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException.<init>(DataStoreAdapterException.java:226)
at com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException.<init>(DataStoreAdapterException.java:177)
at com.ibm.ws.rsadapter.AdapterUtil.createDataStoreAdapterException(AdapterUtil.java:232)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cleanupTransactions(WSRdbManagedConnectionImpl.java:3392)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cleanup(WSRdbManagedConnectionImpl.java:3025)
at com.ibm.ejs.j2c.MCWrapper.cleanup(MCWrapper.java:1353)
at com.ibm.ejs.j2c.poolmanager.FreePool.returnToFreePool(FreePool.java:462)
at com.ibm.ejs.j2c.poolmanager.PoolManager.release(PoolManager.java:1543)
at com.ibm.ejs.j2c.MCWrapper.releaseToPoolManager(MCWrapper.java:2031)
at com.ibm.ejs.j2c.ConnectionEventListener.connectionClosed(ConnectionEventListener.java:263)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.processConnectionClosedEvent(WSRdbManagedConnectionImpl.java:1477)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.closeWrapper(WSJdbcConnection.java:724)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java(Compiled Code))
at org.hibernate.connection.DatasourceConnectionProvider.closeConnection(DatasourceConnectionProvider.java:74)
at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:445)
at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:379)
at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:318)
at org.hibernate.impl.SessionImpl.close(SessionImpl.java:293)
Update: This happens when I do session.close() in finally block. Something like this:
try {
tx = session.beginTransaction();
// some code
if (!tx.wasRolledBack() && !tx.wasCommitted()) {
tx.commit();
}
} catch (Exception ex) {
// rollback transaction in case of errors
}
finally {
session.close(); // Exception happens here!
}
I'm having some error, I found this explain https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014153732 .
He said: "I finally answered my own question. The problem was that my code was setting auto commit to false at the beginning of a transaction and did not set it back to true when the transaction was over. So when the code tried to execute single statements on the same connection object, they were not getting committed.".
I hope you help.