We Keep Coding

Statement.setQueryTimeout doesn't work on Oracle 18c jdbc driver

We have a J2EE application on a payara 5.2020 server that executes a long running query (PL/SQL that executes for a couple of hours).
To avoid a timeout exception, we use this sentence at StatementLevel:
statement.setQueryTimeout(0);
This works using Oracle jdbc drivers version 12c, but when we have migrated to Oracle 18c, and we changed the driver to the version 18c, the query execution stops after 15 minutes with this exception. The code works with Oracle 12 and Oracle 18 is the change in the driver's jar what brings up the problem.
The problem has been reproduced in Linux and Windows machines:
2021-06-14T07:50:01.762+0200|SEVERE: java.sql.SQLRecoverableException: Error de E/S: Socket read interrupted
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:946)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1136)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3640)
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1318)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3752)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4242)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
at com.sun.gjc.spi.base.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:532)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.sun.gjc.spi.jdbc40.ProfiledConnectionWrapper40$1.invoke(ProfiledConnectionWrapper40.java:437)
at com.sun.proxy.$Proxy324.execute(Unknown Source)
at org.apache.jsp.index_jsp.callPL(index_jsp.java:49)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:108)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:411)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:473)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:377)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1636)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:259)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.InterruptedIOException: Socket read interrupted
at oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:262)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:184)
at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:544)
at oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:234)
at oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:174)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:122)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:100)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:86)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:762)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:427)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:394)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:255)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:610)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:249)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:82)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:924)
It seems the transport layer has been migrated to java.nio and the method setQueryTimeout is no longer
Things We've tried:
Setting the default Statement timeout to -1 in the JDBC Connection Pool Advanced Attributes screen in payara console.
Trying to set the time directly in the connection with conn.setNetworkTimeout(Executors.newFixedThreadPool(1), 0) didn't make any effect.
In different sources we have found that these properties below should affect the network timeout evaluation. We set them as JVM properties for payara startup (-Doracle.net.CONNECT_TIMEOUT=xxx) and as JDBC Connection Pool Properties, both cases with values 0 and -1. Didn't work in any case.
oracle.net.CONNECT_TIMEOUT
oracle.net.READ_TIMEOUT
oracle.jdbc.ReadTimeout
Sources:
Oracle 18c Net services best practices
Oracle 18c java jdbc reference. E.1.5 Using JDBC with Firewalls
4.- As we are accessing the DataSource through payara DataSource pool, we cannot cast the com.sun.gjc.spi.jdbc40.DataSource40 (class provided by payara) to an OracleDataSource, but we created the DataSorce programatically to set the connection properties as shown here and setting the properties seen in the image above but it doesn't work:
public static Properties oracleProperties() {
// Already tried -1 and 0
Properties properties = new Properties();
properties.put("Oracle.net.CONNECT_TIMEOUT", 0);
properties.put("Oracle.net.READ_TIMEOUT", 0);
properties.put("Oracle.jdbc.ReadTimeout", 0);
return properties;
}
public static OracleDataSource createDataSource() throws Exception {
OracleDataSource ods = new OracleDataSource();
ods.setURL("jdbc:oracle:thin:#itauc4602x:1521/BDExp");
ods.setUser("enevac");
ods.setPassword("enevac");
ods.setDataSourceName("OracleXADataSource");
ods.setLoginTimeout(0);
// default connection properties to avoid timeoutException
ods.setConnectionProperties(oracleProperties());
return ods;
}
Has anyone faced this problem, any idea on how to avoid the timeOut restriction?
Why 15 minutes?, according to the reference, the default value for oracle.net.ReadTimeout is 10 minutes.
Update:
To explain in more detail why I think the problem is in the driver and why I discard other possible origins of the exception, I assume the timeout can be raised from three sources:
Network timeout: I discard it cause I'm testing a payara server in my local machine against the development database, with no firewall in between.
Database server: the DBA has checked the Oracle net services configuration and there's no limit set that explains the 15 minutes cut. Besides, in these case, an SQLException would be expected with some kind of ORA-xxx error code.
JDBC: this can be set at connection level, statement level and transaction level. As I said at the beginning, the code works with oracle 12c drivers against Oracle 12 and Oracle 18 servers, it was the change of the driver's jar what make the code stop working.

Finally the problem was fixed configuring in the payara pool the "connectionProperties" custom property of the OracleDataSource. As #ibre5041 pointed, setting the property oracle.jdbc.javaNetNio=false changes the transport layer used by the driver and it starts working as the oracle 12c previous version.
According to Oracle reference, the OracleDataSource implementors can receive the connection properties as a java.util.Properties object.
Table 8-2 Oracle Extended Data Source Properties
Name: connectionProperties
Type: java.util.Properties
Description: Specifies the connection properties.
To set a multivalued property to the jdbc pool in the Payara Admin Console, you have to set the properties as (prop1=value1,prop2=value2), (Thank you again Ondro Mihályi). So in our case we set:
connectionProperties = (oracle.jdbc.ReadTimeout=0, oracle.jdbc.javaNetNio=false)
As a summary of what works and doesn't using Oracle 18c jdbc driver (every step tested separately)::
Setting timeout at statement level doesn't work:
statement.setQueryTimeout(0)
Setting timeout at connection level, with -1 or 0, doesn't work:
conn.setNetworkTimeout(Executors.newFixedThreadPool(1), timeout in ms)
Setting timeout properties programmatically in the java.util.connection using the OracleDataSource makes it work as indicated in the question.
Setting timeout properties as JVM properties makes it work if the limit is below 15min, but if you set a value > 15 minutes the exceptions is thrown, so setting to 0 or -1 has no effect:
So this makes the query stop after 10 secs:
-Doracle.net.CONNECT_TIMEOUT=10000 -Doracle.net.READ_TIMEOUT=10000 -Doracle.jdbc.ReadTimeout=10000
But with this stops after 15 minutes:
-Doracle.net.CONNECT_TIMEOUT=-1 -Doracle.net.READ_TIMEOUT=-1 -Doracle.jdbc.ReadTimeout=-1
Setting oracle.net.keepAlive=true as JVM property as #Nirmala suggested doesn’t work.
Setting oracle.jdbc.javaNetNio=false as JVM property as #ibre5041 makes it work. So it points to some problem with the java.nio transport layer.
Anyway, we opened a support issue to Oracle, cause the jdbc api statement.setQueryTimeout(0) should work without having to configure the datasource, I'll put the response when the case is closed.

The query execution could be stopped because of default tcp connection timeout. Can you set keepalive property "oracle.net.keepAlive" to “true” and verify?

How to solve "Socket read timed out" when using hikari connection pool

I am developing an application using play framework (version 2.8.0), java(version 1.8) with an oracle database(version 12C).
There is only zero or one hit to the database in a day, I am getting below error.
java.sql.SQLRecoverableException: IO Error: Socket read timed out
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:919)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2005)
at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:138)
at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$1(HikariPool.java:447)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Socket read timed out
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:174)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:62)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:908)
... 6 common frames omitted
db {
default {
driver=oracle.jdbc.OracleDriver
url="jdbc:oracle:thin:#XXX.XXX.XXX.XX:XXXX/XXXXXXX"
username="XXXXXXXXX"
password="XXXXXXXXX"
hikaricp {
dataSource {
cachePrepStmts = true
prepStmtCacheSize = 250
prepStmtCacheSqlLimit = 2048
}
}
}
}
It seems it is causing due to inactive database connection, How can I solve this?
Please let me know if any other information is required?

You can enable TCP keepalive for JDBC - either be setting directive or by adding "ENABLE=BROKEN" into connection string.
Usually Cisco/Juniper cuts off TCP connection when it is inactive for more that on hour.
While Linux kernel starts sending keepalive probes after two hours(tcp_keepalive_time). So if you decide to turn tcp keepalive on, you will also need root, to change this kernel tunable to lower value(10-15 minutes)
Moreover HikariCP should not keep open any connection for longer than 30 minutes - by default.
So if your FW, Linux kernel and HikariCP all use default settings, then this error should not occur in your system.
See HikariCP official documentation
maxLifetime:
This property controls the maximum lifetime of a connection in the
pool. An in-use connection will never be retired, only when it is
closed will it then be removed. On a connection-by-connection basis,
minor negative attenuation is applied to avoid mass-extinction in the
pool. We strongly recommend setting this value, and it should be
several seconds shorter than any database or infrastructure imposed
connection time limit. A value of 0 indicates no maximum lifetime
(infinite lifetime), subject of course to the idleTimeout setting. The
minimum allowed value is 30000ms (30 seconds). Default: 1800000 (30
minutes)

I have added the below configuration for hickaricp in configuration file and it is
working fine.
## Database Connection Pool
play.db.pool = hikaricp
play.db.prototype.hikaricp.connectionTimeout=120000
play.db.prototype.hikaricp.idleTimeout=15000
play.db.prototype.hikaricp.leakDetectionThreshold=120000
play.db.prototype.hikaricp.validationTimeout=10000
play.db.prototype.hikaricp.maxLifetime=120000

Why is Weblogic work manager is rejecting work

I am using commonJ work managers running in web logic environment.
My setup is as follows:
My spring application context
<bean id="workManagerTaskExecutor"
class="org.springframework.scheduling.commonj.WorkManagerTaskExecutor">
<property name="workManagerName" value="java:comp/env/wm/UVWorkManager" />
</bean>
My web.xml file in WEB-INF
<resource-ref>
<res-ref-name>wm/UVWorkManager</res-ref-name>
<res-type>commonj.work.WorkManager</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
and my weblogic.xml file in WEB-INF
<work-manager>
<name>wm/UVWorkManager</name>
<min-threads-constraint>
<name>min threads</name>
<count>30</count>
</min-threads-constraint>
<max-threads-constraint>
<name>max threads</name>
<count>200</count>
</max-threads-constraint>
<capacity>
<name>max capacity</name>
<count>20</count>
</capacity>
<work-manager-shutdown-trigger>
<max-stuck-thread-time>600</max-stuck-thread-time>
<stuck-thread-count>100</stuck-thread-count>
</work-manager-shutdown-trigger>
</work-manager>
After a while calling the service from SOAP-UI. I am getting the following error
commonj.work.WorkException: commonj.work.WorkRejectedException: [WorkManager:002912]OverloadManager max capacity rejected request as current length 20 exceeds max capacity of 20
I know as the maximum is set to it will be rejected but why doesn't it reclaim those threads? Even if I wait 15 minutes I cannot submit a new request. I have to reboot the service.

The capacity includes all requests, queued or executing, from the constrained work set.
To verify the status of the threads, do a thread dump of the JVM.
You should set the JVM flag -Dweblogic.StuckThreadHandling=true to append WorkManager's name in log files when the theard dump is generated.
kill -3 JVM_PID

Is it a good idea to put jdbc connection code in servlet class?

I am new in servlet and database.I have a core.java file in which I have a JDBC connection code.In the same directory I have a coreServlet.java file which take form value from HTML page.My questions are:
How can I make interaction between two java classes,for example how can I use variables that I got from HTML form in servlet(coreServlet.java) to my core.java file so that I can store that in my database ?
Is it is a better Idea to put my jdbc connection code in coreServlet.java ?
My project structure is like:
- aarya(project name)
|
- WEB-INF
|
| -web.xml
-src
|
-pkg
|
-CoreServlet.java(servlet to interact HTML form)
-Main.java
-Core.java(jdbc code is here to interact database)
|
-html
|
- core.html
|
- css
|
-core.css
|
-javascript
|
-core.js
|
- lib
|
-index.html

The database connection data should be in your JNDI Data Source and it would be better if you use a Connection Pool but never in a class. Since yours is a Web Application, note that the connection pool configuration depends heavily on the web application server.
As examples, this is very well explained in Tomcat 7 Database Connection Pool configuration and JBoss 7 Database Connection Pool configuration (there are other steps to configure the database connection pool on GlassFish and other Web application server, note that this is different on each server).
From both examples, you can see that you will have a XML file where you put the connection attributes: database URL, user, password, min and max pool connection size (how many connections to database will be open)
Tomcat way:
<Resource name="jdbc/ProjectX" auth="Container"
type="javax.sql.DataSource" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/projectx"
username="user" password="password" maxActive="20" maxIdle="10" maxWait="-1"/>
JBoss way:
<datasource jndi-name="jdbc/ProjectX" pool-name="MySqlDS">
<connection-url>jdbc:mysql://localhost:3306/projectx</connection-url>
<driver>com.mysql</driver>
<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
<pool>
<min-pool-size>10</min-pool-size>
<max-pool-size>100</max-pool-size>
<prefill>true</prefill>
</pool>
<security>
<user-name>user</user-name>
<password>password</password>
</security>
<statement>
<prepared-statement-cache-size>32</prepared-statement-cache-size>
<share-prepared-statements/>
</statement>
</datasource>
<drivers>
<driver name="com.mysql" module="com.mysql">
<xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
</driver>
</drivers>
In the end, if you have configured your database connection pool and it works, all you have to do in your code to recover the connection is call InitialContext#lookup to recover the resource using its JNDI resource name.
Knowing this, after configuring a JNDI resource to connect to MySQL database with name "jdbc/ProjectX", you can have a class which recovers the Connection like this:
public class DatabaseConnectivity {
public static Connection getConnection() throws NamingException, SQLException {
InitialContext cxt = new InitialContext();
DataSource ds = (DataSource) cxt.lookup("java:/comp/env/jdbc/ProjectX" );
return ds.getConnection();
}
}
By the way, I would use different names for packages to know the group of functionality of the classes. For example:
src
- edu.home.controller.servlet
+ CoreServlet.java
- edu.home.controller.filter
+ SessionFilter.java
- edu.home.model.entity
+ AnEntity.java
+ AnotherEntity.java
- edu.home.model.database
+ DatabaseConnectivity.java
- edu.home.model.service
+ AnEntityService.java
+ AnotherEntityService.java
(and on and on...)

MySQL clustering & JBOSS

I need to configure JBOSS for MySQL clustering with databases at two different machines (i.e different IPs).
Active-active configuration of db is desired with -
Both db to be updated simultaneously
Loadbalancing
Failover handling - to switch to the other db in case of failure of the 1st db
How do I configure mysql-ds.xml file to achieve all these ? Will it alone solve my problem or any other configuration changes need to be done ?
So far I have tried the following but without much success -
code sample 1 -
<local-tx-datasource>
<jndi-name>/abc</jndi-name>
<connection-url>jdbc:mysql:loadbalance://ip1:portno1,ip2:portno2/dbname?loadBalanceBlacklistTimeout=5000</connection-url>
<driver-class>com.mysql.jdbc.Driver</driver-class>
<user-name>def</user-name>
<password>defpassword</password>
<exception-sorter-class-name>path to exception sorter class</exception-sorter-class-name>
</local-tx-datasource>
code sample 2 -
<local-tx-datasource>
<jndi-name>/abc</jndi-name>
<connection-url>jdbc:mysql:loadbalance://ip1:portno1,ip2:portno2/dbname?loadBalanceBlacklistTimeout=5000</connection-url>
<url-delimiter>|</url-delimiter>
<autoReconnect>true</autoReconnect>
<failOverReadOnly>false</failOverReadOnly>
<maxReconnects>0</maxReconnects>
<initialTimeout>15</initialTimeout>
<idle-timeout-minutes>0</idle-timeout-minutes>
<connection-property name="readOnly">false</connection-property>
<min-pool-size>5</min-pool-size>
<max-pool-size>20</max-pool-size>
<driver-class>com.mysql.jdbc.Driver</driver-class>
<user-name>def</user-name>
<password>defpassword</password>
<exception-sorter-class-name>path to exception sorter class</exception-sorter-class-name>
</local-tx-datasource>
What more is required ?
Thanks

You can achieve this only by using mysql-ds.xml datasource file configuration. What you need to do is following.
Note: You will have to use mysql-connector-java-5.1.13-bin.jar and above for database driver. Earlier versions have some bug (http://bugs.mysql.com/bug.php?id=31053) which will pose you many issues so avoid them.
I am providing a snippet for mysql-ds.xml file.
<local-tx-datasource>
<jndi-name>DATA_SOURCE_NAME</jndi-name>
<connection-url>jdbc:mysql:loadbalance://IP1:port1,IP2:port2</connection-url>
<user-name>username</user-name>
<password>password</password>
<driver-class>org.gjt.mm.mysql.Driver</driver-class>
<check-valid-connection-sql>select count(*) from your_table_name</check-valid-connection-sql>
</local-tx-datasource>