Application: Java spring boot + mysql
Story:
application log showed "Could not open JDBC Connecton for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure"
Recovery of the DB instance has been triggered since hardware problem occurred in RDS instance
After 10 mins, the recovery is done.
application log showed "Could not commit JDBC transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed." instead of previous exception.
After 45 mins, application resumed normal without manual operations.
Questions:
Will restarting the application solve the problem instantly?
Manual operation might not be the best to resolve such problem. But if I want it be auto resumed shortly, anything I can do to the setting datasource properties? i.e. set spring.datasource.justswap.time-between-eviction-runs-millis shorter?
Current datasource setting:
spring.datasource.xxx.maxActive=700
spring.datasource.xxx.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.xxx.test-on-borrow=false
spring.datasource.xxx.test-while-idle=true
spring.datasource.xxx.time-between-eviction-runs-millis=3600000
spring.datasource.druid.stat-view-servlet.enabled=false
Related
I read a lot of questions here in stackoverflow about this subject and I ended with the following code in my application-prd.properties:
#PRD IP
spring.datasource.url=
spring.datasource.username=
spring.datasource.password=
# Fix for DB Connection
spring.datasource.tomcat.max-active=10
spring.datasource.tomcat.initial-size=5
spring.datasource.tomcat.max-idle=5
spring.datasource.tomcat.min-idle=1
spring.datasource.tomcat.test-while-idle=true
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.validation-query=SELECT 1 FROM DUAL
spring.datasource.tomcat.time-between-eviction-runs-millis=5000
spring.datasource.tomcat.min-evictable-idle-time-millis=60000
#Error fix for JSON conversion
spring.jackson.parser.allow-unquoted-control-chars=true
But, for some reason, the Tomcat with my Spring application suddenly loses the connection with the database, giving the following error:
Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
I don't know what should I do. Its been weeks that I'm struggling with it. Is there anything else that I can do? Or can I say that is a firewall issue and the infrastructure team should check something?
Thank you
Also check for connection leaks, might be some txns not closing connection properly that's why pool went out of threshold.
I am running a java application with MySQL as the DB server. DB is a AWS RDS. The MySQL connector is published under GNU/GPL so it has to be substituted with something more lenient. Maria DB client claimed to be compatible. But ever since MariaDB client is being used we are seeing the below intermittent issue during DB intensive activities:
Excption:-
SEVERE: Servlet.service() for servlet [dispatcher] in context with path [/Projectname] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause
java.sql.SQLNonTransientConnectionException: (conn=1530488) Connection is closed
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:187)
at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.cmdPrologue(AbstractQueryProtocol.java:1940)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=XXXXXXXXXX)(port=3306)(type=master) : (conn=1530487) could not load system variables
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1372)
I have tried increasing the time_wait and max_allowed_packet to the maximum allowed but the issue still exists.
Versions:-
MySQL - 5.7.26
MariaDB connector - 2.6.2
Just came across this, found an answer. On my side this was for an Aurora 1 MySQL DB (equiv to 5.6).
As per Diego Dupin in https://jira.mariadb.org/browse/CONJ-824:
only good solution is to use option `usePipelineAuth=false&useBatchMultiSend=false`
So your DB connection string might look like this:
jdbc:mysql://127.0.0.1:3307/mydbname?usePipelineAuth=false&useBatchMultiSend=false
More details:
Aurora proxy has a known race condition issue that results in skipping other
queries in proxy buffer. connection might be in hang waiting for query response.
During authentication, socket has a timeout, that will result throwing the error
you describe.
So if `usePipelineAuth` or `useBatchMultiSend` is enable, it might work ... or not.
All pipeline options (`usePipelineAuth` and `useBatchMultiSend`) must be disabled when using aurora.
Problem has been reported to aurora a few times without correction.
I have created a spring boot application. When I put load on the application after sometime load on database server started increasing and I got started socket read time out error.
2016-10-24 21:56:10 [pool-3-thread-296] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - IO Error: Socket read timed out
2016-10-24 21:56:10 [pool-3-thread-296] ERROR
Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
Here is the properties file details of Springboot application
spring.datasource.url=jdbc:oracle:thin:#<ip>:1521:<dbname>
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver
spring.datasource.initial-size=100
spring.datasource.maxActive=1500
spring.datasource.validationQuery=select 1 from dual
spring.jpa.hibernate.dialect = org.hibernate.dialect.Oracle10gDialect
spring.jpa.show-sql = true
spring.jpa.hibernate.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
spring.datasource.pool-prepared-statements=true
spring.datasource.pool-prepared-statements-cache-size=500
spring.datasource.max-open-prepared-statements=500
Since the error is very generic , I am not able to figure out why I am getting socket read timeout exception.
I also need to know what does "[pool-3-thread-296]" mean in exception log ?
Have you tried this? This worked for me
https://springframework.guru/configuring-spring-boot-for-oracle/
In a nutshell, when I try and get a connection after not having used a transaction for several minutes the first transaction setup fails.
When things are working, my logs show the following for a simple transaction:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] to manual commit
However, if I haven't had any activity for several minutes, instead I will get this message:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
My observations so far:
seems to be based on inactivity, but I've seen this behavior immediately after restarting my Tomcat although nothing else was hitting the database so I think it's inactivity against a network element such as my MySQL server.
when my application starts up, it makes a few non-transactional requests from the database which have not had any problems, so it seems related to transactions.
the timeout element in the #Transactional notation is not effective in this case. It seems to eventually time out, but takes 15 minutes (!).
while this transaction request is busy timing out, I can make subsequent requests successfully.
doesn't seem to be a starved local connection pool. I have seen this right after restarting the Tomcat.
When it finally times out (did I mention 15 minutes!) I get the following:
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [org.apache.commons.dbcp.PoolableConnection#3269c671] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [connection is closed] to manual commit
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Could not close JDBC Connection
ERROR: java.sql.SQLException: Already closed.
ERROR: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,312,604 milliseconds ago. The last packet sent successfully to the server was 924,748 milliseconds ago.
Caused by: java.net.SocketException: Connection timed out
Running Spring 3.1.1, mysql 5.1.32, commons-dbcp 1.4 and commons-pool 1.5.4.
Does anyone know what this is?
Your problem is that MySQL server timeouts idle JDBC connections. This has nothing to do with the TransactionManager set-up.
Have a look at your DataSource set-up. It shall tests connection on connection retrieval and/or validate idle connections in pool.
In commons-dbcp you can set-up test on connection retrieval via the testOnBorrow and validationQuery properties.
After a SQL Server database restart, my Tomcat 6 Spring web applications receives an SQL exception on the first attempt to query the database. Note: Tomcat 6 has continued to run during the SQL Server restart and I'm using myBatis. The exception is as follows:
org.springframework.dao.DataAccessResourceFailureException:
Error querying database. Cause: java.sql.SQLException: I/O Error: Connection reset
Any subsequent queries execute fine (without exception). How can I prevent this exception on the first query?
Set up your pool to test connections before giving them to your application
use testOnBorrow=true and (for instance) validationQuery="select 1" in your db pool connection