We Keep Coding

Socket read interrupted during callback from Docker Java

Context
I have a webservice writing a test id in a queue. Then, a listener reads the queue, searches the test and starts it. During those steps, it writes updates of the test in the database in order to be shown to the user. To be more precise: the test is launched in a docker container and at the end of it, I want to update the status of the test to FINISHED. For that, I use the docker java library with a callback.
Problem
When it comes to call the callback, I receive multiple error messages on the call to update the test (but it happens only once, if I try twice the second time works, but it still writes a lot of error messages from the transaction manager).
Here are the error messages logged:
2020-11-20 09:20:43,639 WARN [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#600268e6[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69 connection handles=1 lastReturned=1605860423264 lastValidated=1605860242146 lastCheckedOut=1605860443564 trackByTx=true pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#2efb0d3b mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool#3e8e7a62[pool=ApplicationDS] xaResource=LocalXAResourceImpl#482fdad2[connectionListener=600268e6 connectionManager=4c83f895 warned=false currentXid=null productName=Oracle productVersion=Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0 jndiName=java:/ApplicationDS] txSync=TransactionSynchronization#1387480544{tx=Local transaction (delegate=TransactionImple < ac, BasicAction: 0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4 status: ActionStatus.ABORTING >, owner=Local transaction context for provider JBoss JTA transaction provider) wasTrackByTx=true enlisted=true cancel=false}]: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
2020-11-20 09:20:43,647 INFO [docker-java-stream--1032099154] (org.jboss.jca.core.connectionmanager.listener.TxConnectionListener) IJ000302: Unregistered handle that was not registered: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#4f3c1cb2 for managed connection: org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#236f1a69
2020-11-20 09:20:43,656 WARN [docker-java-stream--1032099154] (com.arjuna.ats.jta) ARJUNA016031: XAOnePhaseResource.rollback for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db4, node_name=1, branch_uid=0:ffffac110002:-50a6b0bf:5fb6bdb9:73db8, subordinatenodename=null, eis_name=java:/ApplicationDS > failed with exception: org.jboss.jca.core.spi.transaction.local.LocalXAException: IJ001160: Could not rollback local transaction
Caused by: org.jboss.jca.core.spi.transaction.local.LocalResourceException: IO Error: Socket read interrupted
at org.jboss.ironjacamar.jdbcadapters#1.4.22.Final//org.jboss.jca.adapters.jdbc.local.LocalManagedConnection.rollback(LocalManagedConnection.java:139)
...
Caused by: java.sql.SQLRecoverableException: IO Error: Socket read interrupted
at com.oracle.jdbc//oracle.jdbc.driver.T4CConnection.doRollback(T4CConnection.java:1140)
...
Caused by: java.io.InterruptedIOException: Socket read interrupted
at com.oracle.jdbc//oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:258)
...

Explanation
At the beginning, I thought about a connection problem, maybe the the transaction is no more available at the callback time (because the docker run took too long), maybe it has to be invalidated.
But at the end, as written in the console, it came from an interruption of the thread when it tries to acquire the lock to update the test and I discovered where this interruption came from: I took a look at the method executeAndStream in DefaultInvocationBuilder from the docker java library and I discovered this:
Thread thread = new Thread(() -> {
Thread streamingThread = Thread.currentThread();
try (DockerHttpClient.Response response = execute(request)) {
callback.onStart(() -> {
streamingThread.interrupt();
response.close();
});
sourceConsumer.accept(response);
callback.onComplete();
} catch (Exception e) {
callback.onError(e);
}
}, "docker-java-stream-" + Objects.hashCode(request));
thread.setDaemon(true);
thread.start();
And here it is, the closable given to onStart interrupts the thread. After that, I discovered in the method onComplete from ResultCallbackTemplate (that I was extending for my callback) a close on that closable:
#Override
public void onComplete() {
try {
close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Resolution
The problem finally came from the following code I wrote:
#Override
public void onComplete() {
super.onComplete();
updateTest(FINISHED);
}
I was calling the onComplete method from the parent without knowing what is does and as usual, first before doing anything else. To correct that, I only had to call the super method at the end:
#Override
public void onComplete() {
updateTest(FINISHED);
super.onComplete();
}

Spring sql-error-codes.xml does not show correct exception on timeout

The following error occurs when an exception occurs for myJDBCTemplate.queryForList() , before which a setQueryTimeout(1) is set. I have a database which has 1.2 million rows, and looking for the timeout exception to be printed or occur in the case when the statement is executed. So, basically, the timeout occurs but the exception does not mention that.
I am using springFramework-version => 4.1.3.RELEASE in pom.xml
INFO: org.springframework.beans.factory.xml.XMLBeanDefinitionReader - Loading XML bean definition for class path resource [org/springframework/jdbc/support/sql-error-code.xml]
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [select * from myTable where userCategory='1']; SQL state [70100]; error code [1317]; Query execution was interrupted; nested exception is java.sql.SQLException: Query execution was interrupted
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:416)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:471)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:481)
……..
caused by java.sql.SQLExcepion: Query execution was interrupted.
From the answer found at Query execution was interrupted, error #1317 states, the interruption occurs because of timeout, which I think is the possible cause.
Also, the exception states it is caused by java.sql.SQLException, but there are no exact details, why it occurred? So, I am not sure is it because of timeout or something else.

The error is clear in your stack trace:-
error code [1317]; Query execution was interrupted
, which means your query is being interrupted by an execution time limit. This error occurs when your query takes an unexpectedly long time to execute.
The error can be solved by fetching the data in batches by executing the query repeatedly for a certain data range.

how long can Spring JdbcTemplate wait for an oracle stored procedure to finish

my java code is like:
logger.info("start");
getJdbcTemplate().execute("call " + procedureName + "()");
and I got the exception:
org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [call PRMI_UPDATE_USER_LOGIN_INFO()]; Io ERROR: Connection reset; nested exception is java.sql.SQLException: Io ERROR: Connection reset
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:257)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:428)
Maybe it's caused by the long time waiting. I found that it printed "start" in log and after about 5 minutes I got the exception.
update at 2013-03-13:
I got that exception not only at calling oracle stored procedure but at druid's 'JdbcUtil.close(...)':
com.alibaba.druid.util.JdbcUtils.close:81 - close connection error
java.sql.SQLRecoverableException: Io Error: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:521)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:500)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3509)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:167)
at com.alibaba.druid.filter.stat.StatFilter.connection_close(StatFilter.java:254)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:163)
at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.close(ConnectionProxyImpl.java:115)
at com.alibaba.druid.util.JdbcUtils.close(JdbcUtils.java:79)
at com.alibaba.druid.pool.DruidDataSource.shrink(DruidDataSource.java:1876)
at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:1694)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:150)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:180)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:169)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1010)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:97)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:487)
The druid's JdbcUtil.close method is quite simple:
public static void close(Connection x) {
if (x == null) {
return;
}
try {
x.close();
} catch (Exception e) {
LOG.debug("close connection error", e);
}
}
the source code is :
https://github.com/alibaba/druid/blob/master/src/main/java/com/alibaba/druid/util/JdbcUtils.java

It should wait as long as it is needed. Forget about various hacks which try to "detect" deadlock based on timeout delay.
you should find also some ORA-XXXX error. Io ERROR: Connection reset does not look like Oracle error message, there should be some error number attached to it
the timeout 5 minutes is very strange value. Theoretically this can be setup also on database side. As profile parameter CPU_PER_CALL but in such a case you should get an error: ORA-02393: exceeded call limit on CPU usage. And you connection should NOT be lost
theoretically you can also have problems which dead connection detection, but 5 minutes timeout is too short for that
another possible source can be ORA-600 error. Oracle internal error, maybe your session process crashed and therefore TCP connection was lost
you should contact your local DBAs and ask then for cooperation. They should help you better than anonymous people on the Internet forum.

Maybe it's caused by the long time waiting
No it is not caused due to that
As Java Doc says about DataAccessResourceFailureException
Data access exception thrown when a resource fails completely: for
example, if we can't connect to a database using JDBC.

J2CA0081E Websphere 6 Exception

Occasionally I see this exception. It affects some of the crucial business processes in the application. What could it be? Did anyone have similar error?
I use WebSphere 6 with OJDBC14 & Hibernate 3.
[21.02.14 06:46:03:209 PST] 00000031 MCWrapper E J2CA0081E: Method cleanup failed while trying to execute method cleanup on ManagedConnection WSRdbManagedConnectionImpl#4d34b403 from ressource jdbc/MYDS. Caught exception: com.ibm.ws.exception.WsException: DSRA0080E: An exception was received by the Data Store Adapter. See original exception message: Cannot call 'cleanup' on a ManagedConnection while it is still in a transaction..
at com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException.<init>(DataStoreAdapterException.java:226)
at com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException.<init>(DataStoreAdapterException.java:177)
at com.ibm.ws.rsadapter.AdapterUtil.createDataStoreAdapterException(AdapterUtil.java:232)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cleanupTransactions(WSRdbManagedConnectionImpl.java:3392)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.cleanup(WSRdbManagedConnectionImpl.java:3025)
at com.ibm.ejs.j2c.MCWrapper.cleanup(MCWrapper.java:1353)
at com.ibm.ejs.j2c.poolmanager.FreePool.returnToFreePool(FreePool.java:462)
at com.ibm.ejs.j2c.poolmanager.PoolManager.release(PoolManager.java:1543)
at com.ibm.ejs.j2c.MCWrapper.releaseToPoolManager(MCWrapper.java:2031)
at com.ibm.ejs.j2c.ConnectionEventListener.connectionClosed(ConnectionEventListener.java:263)
at com.ibm.ws.rsadapter.spi.WSRdbManagedConnectionImpl.processConnectionClosedEvent(WSRdbManagedConnectionImpl.java:1477)
at com.ibm.ws.rsadapter.jdbc.WSJdbcConnection.closeWrapper(WSJdbcConnection.java:724)
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java(Compiled Code))
at com.ibm.ws.rsadapter.jdbc.WSJdbcObject.close(WSJdbcObject.java(Compiled Code))
at org.hibernate.connection.DatasourceConnectionProvider.closeConnection(DatasourceConnectionProvider.java:74)
at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:445)
at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:379)
at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:318)
at org.hibernate.impl.SessionImpl.close(SessionImpl.java:293)
Update: This happens when I do session.close() in finally block. Something like this:
try {
tx = session.beginTransaction();
// some code
if (!tx.wasRolledBack() && !tx.wasCommitted()) {
tx.commit();
}
} catch (Exception ex) {
// rollback transaction in case of errors
}
finally {
session.close(); // Exception happens here!
}

I'm having some error, I found this explain https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014153732 .
He said: "I finally answered my own question. The problem was that my code was setting auto commit to false at the beginning of a transaction and did not set it back to true when the transaction was over. So when the code tried to execute single statements on the same connection object, they were not getting committed.".
I hope you help.

Illegal Access in WebappClassLoader.loadClass

I have a task scheduler implemented in my application. Basically, what i do is schedule some task's to be executed 4 times in a day (like 6 in 6 hours), so the system schedules it to: 00:00, 06:00, 12:00, 18:00.
Ok, i have a class (FlowJobController) which extends the Thread class and in the run() implementation i keep sleeping the thread in 60 to 60 seconds, so it's executed again to check if there's any task to be executed. If true, i run my Job.
Basically the main part it:
rSet = pStmt.executeQuery();
while (rSet.next()) {
long jobId = rSet.getLong("trf_codigo");
String ruleName = rSet.getString("reg_nome");
String ruleParameters = rSet.getString("trf_regra_parametros");
Job job = new Job();
job.setId(jobId);
job.setRuleName(ruleName);
job.setParameters(Functions.stringToList(ruleParameters, "\n"));
FlowJob flowJob = new FlowJob(this, job);
flowJob.start();
}
} catch (Exception ex) {
logger.error(WFRSystem.DEFAULT_USER, system.getCode(), ex);
} finally {
try {
DBConnection.close(pStmt);
DBConnection.close(rSet);
// executede 60 in 60 sec
Thread.sleep(60 * 1000);
} catch (InterruptedException ex) {
logger.error(WFRSystem.DEFAULT_USER, system.getCode(), ex);
}
}
The thing is: When the pStmt.executeQUery() returns records to be executed, it goes into the while and the error appears into the line: Job job = new Job();
The error is:
Exception in thread "FlowJobController" java.lang.NoClassDefFoundError: wfr/com/Job
at wfr.com.FlowJobController.run(FlowJobController.java:112)
Before this error i got this error:
25/09/2012 12:00:09 org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load wfr.com.Job.
The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at wfr.com.FlowJobController.run(FlowJobController.java:112)
The FlowJobController.java:112 is the Job job = new Job();
What am i doing wrong?

The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
If you are saying that your code deliberately threw the IllegalStateException exception, then that is the likely cause of the subsequent problems. If you throw an uncheck exception in static initialization, and that exception propagates, then the class initialization fails, and the class (and all classes that depend on it) are unusable. In this case, it looks like it also caused the webapp to be stopped.
If this is the problem, the solution is "don't do that". Fix whatever is causing the exception to be thrown.
The other possibility is that this is simply a missing class problem:
Well, that sounds not possible (to not be in the war) since i'm running into the eclipse environment and it's in the class folder directory and no errors is shown about not finding those classes...
The class loader won't be looking in the Eclipse classes folder. It will be looking in the webapp directory on your web container (Tomcat, Jetty, whatever).