We Keep Coding

Proper setting of application properties for "No operation allowed connection" exception

Application: Java spring boot + mysql
Story:
application log showed "Could not open JDBC Connecton for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure"
Recovery of the DB instance has been triggered since hardware problem occurred in RDS instance
After 10 mins, the recovery is done.
application log showed "Could not commit JDBC transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed." instead of previous exception.
After 45 mins, application resumed normal without manual operations.
Questions:
Will restarting the application solve the problem instantly?
Manual operation might not be the best to resolve such problem. But if I want it be auto resumed shortly, anything I can do to the setting datasource properties? i.e. set spring.datasource.justswap.time-between-eviction-runs-millis shorter?
Current datasource setting:
spring.datasource.xxx.maxActive=700
spring.datasource.xxx.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.xxx.test-on-borrow=false
spring.datasource.xxx.test-while-idle=true
spring.datasource.xxx.time-between-eviction-runs-millis=3600000
spring.datasource.druid.stat-view-servlet.enabled=false

Issue integrating mariaDB client with MySQL DB

I am running a java application with MySQL as the DB server. DB is a AWS RDS. The MySQL connector is published under GNU/GPL so it has to be substituted with something more lenient. Maria DB client claimed to be compatible. But ever since MariaDB client is being used we are seeing the below intermittent issue during DB intensive activities:
Excption:-
SEVERE: Servlet.service() for servlet [dispatcher] in context with path [/Projectname] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause
java.sql.SQLNonTransientConnectionException: (conn=1530488) Connection is closed
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:187)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1940)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=XXXXXXXXXX)(port=3306)(type=master) : (conn=1530487) could not load system variables
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1372)
I have tried increasing the time_wait and max_allowed_packet to the maximum allowed but the issue still exists.
Versions:-
MySQL - 5.7.26
MariaDB connector - 2.6.2

Just came across this, found an answer. On my side this was for an Aurora 1 MySQL DB (equiv to 5.6).
As per Diego Dupin in https://jira.mariadb.org/browse/CONJ-824:
only good solution is to use option `usePipelineAuth=false&useBatchMultiSend=false`
So your DB connection string might look like this:
jdbc:mysql://127.0.0.1:3307/mydbname?usePipelineAuth=false&useBatchMultiSend=false
More details:
Aurora proxy has a known race condition issue that results in skipping other
queries in proxy buffer. connection might be in hang waiting for query response.
During authentication, socket has a timeout, that will result throwing the error
you describe.
So if `usePipelineAuth` or `useBatchMultiSend` is enable, it might work ... or not.
All pipeline options (`usePipelineAuth` and `useBatchMultiSend`) must be disabled when using aurora.
Problem has been reported to aurora a few times without correction.

Spring 3 DataSourceTransactionManager sporadically timing out when getting transaction to MySQL

In a nutshell, when I try and get a connection after not having used a transaction for several minutes the first transaction setup fails.
When things are working, my logs show the following for a simple transaction:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] to manual commit
However, if I haven't had any activity for several minutes, instead I will get this message:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
My observations so far:
seems to be based on inactivity, but I've seen this behavior immediately after restarting my Tomcat although nothing else was hitting the database so I think it's inactivity against a network element such as my MySQL server.
when my application starts up, it makes a few non-transactional requests from the database which have not had any problems, so it seems related to transactions.
the timeout element in the #Transactional notation is not effective in this case. It seems to eventually time out, but takes 15 minutes (!).
while this transaction request is busy timing out, I can make subsequent requests successfully.
doesn't seem to be a starved local connection pool. I have seen this right after restarting the Tomcat.
When it finally times out (did I mention 15 minutes!) I get the following:
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [org.apache.commons.dbcp.PoolableConnection#3269c671] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [connection is closed] to manual commit
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Could not close JDBC Connection
ERROR: java.sql.SQLException: Already closed.
ERROR: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,312,604 milliseconds ago. The last packet sent successfully to the server was 924,748 milliseconds ago.
Caused by: java.net.SocketException: Connection timed out
Running Spring 3.1.1, mysql 5.1.32, commons-dbcp 1.4 and commons-pool 1.5.4.
Does anyone know what this is?

Your problem is that MySQL server timeouts idle JDBC connections. This has nothing to do with the TransactionManager set-up.
Have a look at your DataSource set-up. It shall tests connection on connection retrieval and/or validate idle connections in pool.
In commons-dbcp you can set-up test on connection retrieval via the testOnBorrow and validationQuery properties.

Migrating from MySql: MariaDB server closing client connections unexpectedly

We in the process of migrating from MySql to MariaDB due to licensing/commercial usage reasons.
We have successfully replaced the MySql connector jar with MariaDB client jar (first change) and are now trying to replace MySql server with MariaDB server without changing the data files.
All our applications run perfectly for about 8-12 hours after which we see the following exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection
Caused by:
org.hibernate.exception.JDBCConnectionException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:74)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:494)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:315)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:257)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209)
at $Proxy4.getMessageCountByStatus(Unknown Source)
at com.onmobile.cmfweb.monitoring.CmfMessagesMonitor.getMessageCounts(CmfMessagesMonitor.java:56)
at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:270)
at com.onmobile.cmfshare.MethodInvockingBean.invoke(MethodInvockingBean.java:28)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:212)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:79)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
at org.mariadb.jdbc.Driver.connect(Driver.java:114)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
... 21 more
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not connect to localhost:3306: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:509)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:669)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.<init>(MySQLProtocol.java:264)
at org.mariadb.jdbc.Driver.connect(Driver.java:110)
... 28 more
Caused by: java.io.EOFException: unexpected end of stream, read 0bytes from 4
at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:84)
at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92)
at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77)
at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67)
at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect(MySQLProtocol.java:467)
... 31 more
The exceptions disappear if MySql server is used again.
From what I have debugged so far MariaDB server is closing the client connections for some reason. I initially suspected of idle connections but we are using Hibernate configurations like 'testOnBorrow' so it shouldn't be the case.
Can anyone help us to find out the cause for this and help us fix the problem? Is there any specific configuration in MariaDB that I should be doing?
I've even run our applications by increasing the value of the param 'interactive_timeout' in the MariaDB server but it did not help.
Btw, we are using Spring-Hibernate and pooling our connections using Commons-pool jar.
Any kind of help will be deeply appreciated.

We encountered the same problem recently, due to a combination of incorrect parameters.
This error is caused by your web instance trying to use a connection that is no longer a valid one.
This can be resolved by making sure the following parameters are correct:
You have a validationQuery configured for your database connection. i.e. validationQuery="SELECT 1" in case of mariadb, in the server configuration.
You have the wait_timeout set to a reasonable value. 8 hours keepAlive seems a bit optimistic, we are now using wait_timeout=180
Make sure the validationInterval, which you can also set in the server configuration (server.xml in case of a tomcat setup), is set to a value that is lower than the wait_timeout value. I reduced the wait_timeout to 15s in one case, with the validationInterval value being higher than that, which caused the error to still appear at times.
Now it is set to validationInterval=60, combined with the wait_timeout=180 which should catch any broken connections in time.

You can try to increase 'wait_timeout' instead of 'interactive_timeout'. 8-12 hours until error correlates quite well with the default value of wait_timeout which is 10 hours. JDBC driver does not set interactive client flag when connecting, so change to 'interactive_timeout' might have no effect. I also replied to the cross-post in Maria KB ;)

According to the findings in this article you could get away with two options:
You should try turning innodb_file_per_table off
or...
Increasing the open file limits, this tells you the current value on yout machine $ ulimit -n, should you be running on high spec machines

I also had an error like this one Could not connect to localhost:3306: unexpected end of stream, read 0 bytes from 4
My architecture was an webapp of Tomcat that was inserting data into MariaDB.
I was using the connector version 1.5.1RC!
I switched to the 1.3.1 connector version and now runs fine!

I also faced this issue. The issue goes away with lowering the wait_timeout.
SHOW SESSION VARIABLES LIKE 'wait_timeout'; ->> would be 8 hours. which is huge.
SET session wait_timeout=300; ->> this should fix the issue.

Look at
SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE '%connect%';
Compare the my.cnf values between the two systems; you may see a difference in things like above that make a difference. If the problem is still not obvious, please provide those values for further discussion.

I think that the problem is with MariaDB Connector/J. Try to use a stable one like MariaDB Connector/J 1.2.3 for example. Also, study Failover behaviour with Basic failover when autoReconnect is set to true or Standard failover.

org.springframework.jdbc.CannotGetJdbcConnectionException

I am working in spring3 project. When I run my code sometimes i am getting this Jdbc Connection Exception for some particular functionality but other functionalities working very fine. And also this Exception is not occurring repetitively but sometimes, So I can't get where the mistake will be.
Please help me to come out of this problem.
<[weblogic.servlet.internal.WebAppServletContext#1b6e978 - appName: '_auto_generated_ear_', name: 'ae', context-path: '/ae'] Root cause of ServletException.
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
The Connection descriptor used by the client was:
localhost:1521:XE at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:674)
Truncated. see log file for complete stacktrace
java.sql.SQLException: Listener refused the connection with the following error:
ORA-12519, TNS:no appropriate service handler found
The Connection descriptor used by the client was:
localhost:1521:XE
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:261)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:420)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:165)
Truncated. see log file for complete stacktrace

There are 2 common reason for that symptoms
The connection pool is exhausted, too many active connections open and the next client cannot get it. This might be a connection pool leak
The connection pool setting doesn't test idle connection periodically / on borrow, hence when TCP connection truncated by OS (eg: because the OS thinks it's idle doing nothing), the pool still thinks it's a valid

To resolve this issue, just Restart your MySQL service
How to restart the MySQL service
On Windows: https://www.mysqltutorial.org/mysql-adminsitration/restart-mysql
On Linux:    https://linuxhint.com/restart-mysql-in-ubuntu/