We Keep Coding

Socket read interrupted during callback from Docker Java

Context
I have a webservice writing a test id in a queue. Then, a listener reads the queue, searches the test and starts it. During those steps, it writes updates of the test in the database in order to be shown to the user. To be more precise: the test is launched in a docker container and at the end of it, I want to update the status of the test to FINISHED. For that, I use the docker java library with a callback.
Problem
When it comes to call the callback, I receive multiple error messages on the call to update the test (but it happens only once, if I try twice the second time works, but it still writes a lot of error messages from the transaction manager).
Here are the error messages logged:
2020-11-20 09:20:43,639 WARN [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#600268e6[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69 connection handles=1 lastReturned=1605860423264 lastValidated=1605860242146 lastCheckedOut=1605860443564 trackByTx=true pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#2efb0d3b mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#3e8e7a62[pool=ApplicationDS] xaResource=LocalXAResourceImpl#482fdad2[connectionListener=600268e6 connectionManager=4c83f895 warned=false currentXid=null productName=Oracle productVersion=Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0 jndiName=java:/ApplicationDS] txSync=TransactionSynchronization#1387480544{tx=Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4 status: ActionStatus.ABORTING >, owner=Local transaction context for provider JBoss JTA transaction provider) wasTrackByTx=true enlisted=true cancel=false}]: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
2020-11-20 09:20:43,647 INFO [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000302: Unregistered handle that was not registered: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#4f3c1cb2 for managed connection: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69
2020-11-20 09:20:43,656 WARN [docker-java-stream--1032099154] (com.arjuna.ats.jta) ARJUNA016031: XAOnePhaseResource.rollback for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4, node_name=1, branch_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db8, subordinatenodename=null, eis_name=java:/ApplicationDS > failed with exception: org.jboss.jca.core.spi.transaction.local.LocalXAException: IJ001160: Could not rollback local transaction
Caused by: org.jboss.jca.core.spi.transaction.local.LocalResourceException: IO Error: Socket read interrupted
at org.jboss.ironjacamar.jdbcadapters#1.4.22.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnection.rollback(LocalManagedConnection.java:139)
...
Caused by: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
at com.oracle.jdbc//oracle.jdbc.driver.T4CConnection.doRollback(T4CConnection.java:1140)
...
Caused by: java.io.InterruptedIOException: Socket read interrupted
at com.oracle.jdbc//oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:258)
...

Explanation
At the beginning, I thought about a connection problem, maybe the the transaction is no more available at the callback time (because the docker run took too long), maybe it has to be invalidated.
But at the end, as written in the console, it came from an interruption of the thread when it tries to acquire the lock to update the test and I discovered where this interruption came from: I took a look at the method executeAndStream in DefaultInvocationBuilder from the docker java library and I discovered this:
Thread thread = new Thread(() -> {
Thread streamingThread = Thread.currentThread();
try (DockerHttpClient.Response response = execute(request)) {
callback.onStart(() -> {
streamingThread.interrupt();
response.close();
});
sourceConsumer.accept(response);
callback.onComplete();
} catch (Exception e) {
callback.onError(e);
}
}, "docker-java-stream-" + Objects.hashCode(request));
thread.setDaemon(true);
thread.start();
And here it is, the closable given to onStart interrupts the thread. After that, I discovered in the method onComplete from ResultCallbackTemplate (that I was extending for my callback) a close on that closable:
#Override
public void onComplete() {
try {
close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Resolution
The problem finally came from the following code I wrote:
#Override
public void onComplete() {
super.onComplete();
updateTest(FINISHED);
}
I was calling the onComplete method from the parent without knowing what is does and as usual, first before doing anything else. To correct that, I only had to call the super method at the end:
#Override
public void onComplete() {
updateTest(FINISHED);
super.onComplete();
}

Acquired Connections grows abruptly leading to server crash

I have a Java application that schedules a cron job after every 1 min. It runs on Glassfish 4. We are using Hibernate with JTA Entity Manager which is container managed for executing the queries on SQL Server database.
JDBC Connection Pool Settings are:
Initial and Minimum Pool Size:16
Maximum Pool Size:64
Pool Resize Quantity:4
Idle Timeout:300
Max Wait Time:60000
JDBC Connection Pool Statistics after 22 Hours run:
NumConnUsed 0count
NumConnAcquired 14404count
NumConnReleased 14404count
NumConnCreated 16count
NumConnFree 16count
The number of acquired connections keeps on incrementing and the Glassfish 4 crashes after around 10 days with below exception.
RAR5117 : Failed to obtain/create connection from connection pool [ com.beonic.tiv5 ]. Reason : com.sun.appserv.connectors.internal.api.PoolingException: java.lang.RuntimeException: Got exception during XAResource.start:
Please suggest how to avoid Glassfish crash.

finally
{
em = null;
ic = null;
}
I think here is the problem you are never commiting or closing the transacction
Giving this example and documentation of JTA check 5.2.2
// BMT idiom
#Resource public UserTransaction utx;
#Resource public EntityManagerFactory factory;
public void doBusiness() {
EntityManager em = factory.createEntityManager();
try {
// do some work
...
utx.commit();
}
catch (RuntimeException e) {
if (utx != null) utx.rollback();
throw e; // or display error message
}
finally {
em.close();
}
This is the correct way of doing a transacction. But you are only nulling the values and nothing more, that's why you your pools and not being closed
Here is more documentation about Transactions

It's hard to tell what is the real cause of the problem, but the problem might be that all your connections have become stale because not used for a long time.
It is a good practice to set up connection validation, which ensures that connections are reopened when closed by the external server.
There is a thorough article about connection pools in Glassfish/Payara, checkout especially the section about Connection validation (using Derby DB in the example):
To turn on connection validation :
asadmin set
resources.jdbc-connection-pool.test-pool.connection-validation-method=custom-validation
asadmin set
resources.jdbc-connection-pool.test-pool.validation-classname=
org.glassfish.api.jdbc.validation.DerbyConnectionValidation
asadmin
set
resources.jdbc-connection-pool.test-pool.is-connection-validation-required=true

how long can Spring JdbcTemplate wait for an oracle stored procedure to finish

my java code is like:
logger.info("start");
getJdbcTemplate().execute("call " + procedureName + "()");
and I got the exception:
org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [call PRMI_UPDATE_USER_LOGIN_INFO()]; Io ERROR: Connection reset; nested exception is java.sql.SQLException: Io ERROR: Connection reset
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:257)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:428)
Maybe it's caused by the long time waiting. I found that it printed "start" in log and after about 5 minutes I got the exception.
update at 2013-03-13:
I got that exception not only at calling oracle stored procedure but at druid's 'JdbcUtil.close(...)':
com.alibaba.druid.util.JdbcUtils.close:81 - close connection error
java.sql.SQLRecoverableException: Io Error: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:521)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:500)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3509)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:167)
at com.alibaba.druid.filter.stat.StatFilter.connection_close(StatFilter.java:254)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:163)
at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.close(ConnectionProxyImpl.java:115)
at com.alibaba.druid.util.JdbcUtils.close(JdbcUtils.java:79)
at com.alibaba.druid.pool.DruidDataSource.shrink(DruidDataSource.java:1876)
at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:1694)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:150)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:180)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:169)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1010)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:97)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:487)
The druid's JdbcUtil.close method is quite simple:
public static void close(Connection x) {
if (x == null) {
return;
}
try {
x.close();
} catch (Exception e) {
LOG.debug("close connection error", e);
}
}
the source code is :
https://github.com/alibaba/druid/blob/master/src/main/java/com/alibaba/druid/util/JdbcUtils.java

It should wait as long as it is needed. Forget about various hacks which try to "detect" deadlock based on timeout delay.
you should find also some ORA-XXXX error. Io ERROR: Connection reset does not look like Oracle error message, there should be some error number attached to it
the timeout 5 minutes is very strange value. Theoretically this can be setup also on database side. As profile parameter CPU_PER_CALL but in such a case you should get an error: ORA-02393: exceeded call limit on CPU usage. And you connection should NOT be lost
theoretically you can also have problems which dead connection detection, but 5 minutes timeout is too short for that
another possible source can be ORA-600 error. Oracle internal error, maybe your session process crashed and therefore TCP connection was lost
you should contact your local DBAs and ask then for cooperation. They should help you better than anonymous people on the Internet forum.

Maybe it's caused by the long time waiting
No it is not caused due to that
As Java Doc says about DataAccessResourceFailureException
Data access exception thrown when a resource fails completely: for
example, if we can't connect to a database using JDBC.

WebService times out, but client receives no exception

I have an application that is attempting to call a service and the other service appears to be timing out. The problem is my application does not receive any timeout exceptions, although I do see an error printed out to the console:
[7/8/13 12:39:32:360 EDT] 00000005 TimeoutManage I WTRN0006W: Transaction 0000013FBF252E43000000010000000CE81CB4935851D5C13DECD3DBB2D463F0DBECAEE60000013FBF252E43000000010000000CE81CB4935851D5C13DECD3DBB2D463F0DBECAEE600000001 has timed out after 120 seconds.
[7/8/13 12:39:32:360 EDT] 00000005 TimeoutManage I WTRN0124I: When the timeout occurred the thread with which the transaction is, or was most recently, associated was Thread[WebContainer : 1,5,main]. The stack trace of this thread when the timeout occurred was:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:196)
com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:334)
com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:182)
com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.parseResponseMessageSync(HttpOutboundServiceContextImpl.java:1657)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.readSyncResponse(HttpOutboundServiceContextImpl.java:725)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.startResponseReadSync(HttpOutboundServiceContextImpl.java:1775)
com.ibm.ws.http.channel.outbound.impl.HttpOutboundServiceContextImpl.finishRequestMessage(HttpOutboundServiceContextImpl.java:1195)
com.ibm.ws.websvcs.transport.http.out.HttpOutSyncWriter.finishBufferRequest(HttpOutSyncWriter.java:94)
com.ibm.ws.websvcs.transport.http.out.HttpOutWriter.writeBuffer(HttpOutWriter.java:136)
com.ibm.ws.websvcs.transport.http.out.HttpOutByteBufferOutputStream.finish(HttpOutByteBufferOutputStream.java:468)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendChunkedRequest(SOAPOverHTTPSender.java:890)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.sendSOAPRequest(SOAPOverHTTPSender.java:807)
com.ibm.ws.websvcs.transport.http.SOAPOverHTTPSender.send(SOAPOverHTTPSender.java:611)
com.ibm.ws.websvcs.transport.http.HTTPTransportSender.invoke(HTTPTransportSender.java:364)
org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:531)
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:401)
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:228)
org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)
org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.execute(AxisInvocationController.java:581)
org.apache.axis2.jaxws.core.controller.impl.AxisInvocationController.doInvoke(AxisInvocationController.java:130)
org.apache.axis2.jaxws.core.controller.impl.InvocationControllerImpl.invoke(InvocationControllerImpl.java:93)
org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invokeSEIMethod(JAXWSProxyHandler.java:364)
org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.invoke(JAXWSProxyHandler.java:185)
The client is created with these settings:
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.RESPONSE_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.CONNECTION_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
bindProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.READ_TIMEOUT_PROPERTY , connectionProperties.getProperty(MyService.TIME_OUT));
MyService.TIME_OUT has a value of 20000 and I have verified that it is being set correctly.
The code that catches calls the service looks like this:
try
{
response = ((MyServicePortType) myService).doWebServiceOperation(request);
}
catch (Throwable e) //I know, catch Throwable is not very good but right now I'd be happy to catch ANYthing here!
{
log.error("Webservice reported error",e);
}
Even though I've changed my catch block to catch anything, I still don't catch any exceptions. WebSphere detects a transaction timeout, but I don't know why the application doesn't detect a timeout in the web service call. Is there something I'm missing that would cause a proper timeout exception to be thrown so that I can catch it and send the message to the application framework?

Well now I feel silly.
It seems that for WebSphere, these properties (RESPONSE_TIMEOUT_PROPERTY, CONNECTION_TIMEOUT_PROPERTY, etc...) should have their values set in seconds, and I was using milliseconds based on what I'd seen in online examples (that clearly were not intended for WebSphere).
Changing 20000 to 20 has resolved this problem.
The page that suggested I should be assuming seconds instead of milliseconds is this one: http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/rwbs_httptransportprop.html

Catched ActivitiException still gets thrown and breaks application

I'm having trouble understanding what is happening and why my loop doesn't continue. I'm creating a dashboard for all open activiti tasks. Now the trouble i am having is when someone closes a task while the dashboard is being created.
The code is like this:
List<Task> approvalTasks = taskQueryApproval.list();
for (Task task : approvalTasks) {
try {
ActivitiApplicationRequest activitiRequest = (ActivitiApplicationRequest) taskService
.getVariable(task.getId(), ACTIVITIREQUEST);
if (!dashboardValues.containsKey(activitiRequest.getGlobalRequestId())) {
GlobalRequest globalRequest = globalRequestDao
.findMinimalGlobalRequestForDashboardBySyscode(activitiRequest.getGlobalRequestId());
if (globalRequest != null) {
DashboardValueObject vo = new DashboardValueObject(globalRequest);
vo.setHasApproval(true);
dashboardValues.put(activitiRequest.getGlobalRequestId(), vo);
}
}
} catch (ActivitiException ex) {
LOGGER.debug("Approval already done, skipping activititask");
}
}
The trouble I'm having is I know when the task doesn't exist Activiti is going to throw an exception, that's why I placed a try-catch inside the for loop.
What I'm expecting is that when the error is catched it just continues with the rest of the list. While debugging I even saw the catch being done. But the loop still breaks and the application stops. With these exceptions :
SEVERE: Error while closing command context org.activiti.engine.ActivitiException: task 203039 doesn't exist
at org.activiti.engine.impl.cmd.GetTaskVariableCmd.execute(GetTaskVariableCmd.java:55)
at org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:42)
9-nov-2012 14:45:42 org.activiti.engine.impl.interceptor.CommandContext close
SEVERE: Error while closing command context
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: com.atomikos.jdbc.AtomikosSQLException: The transaction has timed out - try increasing the timeout if needed
### The error may exist in org/activiti/db/mapping/entity/Task.xml
### The error may involve org.activiti.engine.impl.persistence.entity.TaskEntity.selectTask
### The error occurred while executing a query
### SQL: select * from ACT_RU_TASK where ID_ = ?
### Cause: com.atomikos.jdbc.AtomikosSQLException: The transaction has timed out - try increasing the timeout if needed
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:8)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:81)
What am I doing wrong?