I have a spring boot, hibernate using java app. I deploy it on a jetty webserver with multiple instances. if I have too (greater than 10) many instances I get
com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: remaining connection slots are reserved for non-replication superuser connections
Many of the connections (10x instance) show up idle
ps:
postgres 9104 0.0 0.0 248968 4392 ? Ss 08:07 0:00 postgres: user my_db 127.0.0.1(60095) idle
Hikari trace log for an instance:
2017-02-21 10:59:47.578 DEBUG 7401 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
setting
hikari.leakDetectionThreshold: 30000
Doesnt log anything interesing. I think this looks interesting HikariCP - connection is not available
Any ideas how i could debug this? Also I am on java 7, so hikari 2.4.7
I think you must increase the max_connections parameter in your PostgreSQL configuration.
See https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
You cannot allow the pool to keep more connections than your PostgreSQL installation allows.
Related
We are facing now JDBC connection issue while we run load test. it goes well for few hours but suddenly thread starts going into pending state.
I have checked code and i could not find we are leaking any connection by intention. Only smoke I can see in logs is related to JPA call we make. but still it does not answer why we hitting no connection issue. any suggestion will be help ful.
Tech Stack -
We are using Spring Boot 2.4.7, Hibernate 5.4.25, Hikari CP 3.4.5, JPA 2.2
Logs -
1 [LogType:Usage] DEBUG [payment-service,,] [15] [HikariPool-1 connection adder] [com.zaxxer.hikari.pool.HikariPool] HikariPool-1 - Added connection org.mariadb.jdbc.MariaDbConnection#44a351b0
[HikariPool-1 housekeeper] [com.zaxxer.hikari.pool.ProxyLeakTask] Connection leak detection triggered for org.mariadb.jdbc.MariaDbConnection#44a351b0 on thread http-nio-8080-exec-10, stack trace follows \u2028java.lang.Exception: Apparent connection leak detected\u2028 at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)\u2028 at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)\u2028 at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)\u2028 at
[LogType:Usage] [com.zaxxer.hikari.pool.ProxyLeakTask] Previously reported leaked connection org.mariadb.jdbc.MariaDbConnection#44a351b0 on thread http-nio-8080-exec-10 was returned to the pool (unleaked)
[HikariPool-1 connection closer] [com.zaxxer.hikari.pool.PoolBase] HikariPool-1 - Closing connection org.mariadb.jdbc.MariaDbConnection#44a351b0: (connection has passed maxLifetime)
But in between we see and it is where our load test start failing ->
[LogType:Usage] DEBUG [14] [HikariPool-1 housekeeper] [com.zaxxer.hikari.pool.HikariPool] HikariPool-1 - Pool stats (total=15, active=15, idle=0, waiting=75)
Configuration -
connectionleakthreshold - 3000 ( we event tried with 30000 but found same behaviour)
connection-timeout - 30000
Please read for property - spring.jpa.open-in-view
https://raul8804.wordpress.com/2019/03/31/spring-boot-project-db-connection-leak-investigation/
This is very much strange how Spring adds transaction for all controllers.
I want to see what queries mongo java driver produce, but I'm not able to do that.
Using information from the official documentation I'm able just see in the log that update operation executes, but I don't see the query of this operation.
You can set the logger level for org.mongodb to DEBUG and your Java driver will emit detailed logging like this:
2018-01-18 16:51:07|[main]|[NA]|INFO |org.mongodb.driver.connection|Opened connection [connectionId{localValue:2, serverValue:39}] to localhost:27017
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.insert|Inserting 1 documents into namespace stackoverflow.sample on connection [connectionId{localValue:2, serverValue:39}] to server localhost:27017
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.insert|Insert completed
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.command|Sending command {find : BsonString{value='sample'}} to database stackoverflow on connection [connectionId{localValue:2, serverValue:39}] to server localhost:27017
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.command|Command execution completed
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.command|Sending command {findandmodify : BsonString{value='sample'}} to database stackoverflow on connection [connectionId{localValue:2, serverValue:39}] to server localhost:27017
2018-01-18 16:51:07|[main]|[NA]|DEBUG|org.mongodb.driver.protocol.command|Command execution completed
In the above log output you can see the details of a query submitted by the client:
org.mongodb.driver.protocol.command|Sending command {find : BsonString{value='sample'}}
Alternatively, you can enable profiling on the server side ...
db.setProfilingLevel(2)
... causes the MongoDB profiler to collect data for all operations against that database.
The profiler output (which includes the query submitted by the client) is written to the system.profile collection in whichever database profiling has been enabled.
More details in the docs but the short summary is:
// turn up the logging
db.setProfilingLevel(2)
// ... run some commands
// find all profiler documents, most recent first
db.system.profile.find().sort( { ts : -1 } )
// turn down the logging
db.setProfilingLevel(0)
If you're using Spring Boot 1.5.x (I'm on 1.5.19) you'll need to override the version of org.mongodb:mongodb-driver to at least version 3.7.0 to get the additional info in the logs.
See this ticket for more details: https://jira.mongodb.org/browse/JAVA-2698
We have an application that has been running fine with Java 7 for a few years, but have problems upgrading to Java 8. The database connections seems to be the problem.
This is the case:
The application accesses more than 200 databases. Or, technically, 200 different schemas on one database installation. We use Atomikos for the data sources. When starting up the application with Java 7, there is probably some sequential iteration over the data sources, because we have set the max pool size to 5. It takes a little bit of time, but really no issues.
When we try running with Java 8, my guess is that it tries to do everything in parallell. The application crashes on startup like this:
05:21:39.611 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit 'database.np'
05:21:41.389 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit
'database.np_000000000_0000' 05:21:41.796 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit
'database.np_000000001_0000' 05:22:11.890 WARN [main]
c.atomikos.jdbc.AtomikosSQLException - Connection pool exhausted - try
increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the
DataSourceBean. 05:22:41.991 WARN [main]
c.atomikos.jdbc.AtomikosSQLException - Connection pool exhausted - try
increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the
DataSourceBean. 05:22:41.998 ERROR [main]
o.h.tool.hbm2ddl.SchemaValidator - HHH000319: Could not get database
metadata com.atomikos.jdbc.AtomikosSQLException: Connection pool
exhausted - try increasing 'maxPoolSize' and/or
'borrowConnectionTimeout' on the DataSourceBean.
at com.atomikos.jdbc.AtomikosSQLException.throwAtomikosSQLException(AtomikosSQLException.java:46)
~[transactions-jdbc-3.9.3.jar:na]
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:90)
~[transactions-jdbc-3.9.3.jar:na]
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:85)
~[transactions-jdbc-3.9.3.jar:na]
So, we try the advice in the stacktrace, and increase maxPoolSize to a
suitable number above 200. Then we can actually start the application,
but... It seems like a complete overkill, and to make matters worse,
the application doesn't like having so many connections it doesn't
really use - the databases/schemas are accessed only now and then. We
see the following in the log:
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 358671ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 357953ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 357947ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 26425ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': current size: 1/4
11:08:29.983 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': trying to shrink pool
11:08:29.984 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': connection idle for
359790ms | (repeat 200 times...) | atomikos connection pool
'dataSourceRolp': connection idle for 263450ms 11:07:29.983
[Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool - atomikos
connection pool 'dataSourceRolp': connection idle for 263446ms
11:07:29.983 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': connection idle for
263168ms 11:07:29.983 [Atomikos:1] DEBUG
c.a.datasource.pool.ConnectionPool - atomikos connection pool
'dataSourceRolp': current size: 1/208
Any thoughts? For legal reasons, we need to have the 200+ schemas separate. Other than that, we are fairly free to choose technical solutions as we want.
What database are you using? I had a quite similar issue during our database migration(From Oracle to PostgreSQL).
We defined the data source as JNDI, and to make the data source we also removed some attributed out from the jdbc interceptor.
If you are using PostgreSQL, please check the type of bean you defined for the data source too.
I have app in SpringBoot with JPA.
When I lost connection my app send me error:
WARN 6812 --- [io-8080-exec-42] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 17002, SQLState: 08006
ERROR 6812 --- [io-8080-exec-42] o.h.engine.jdbc.spi.SqlExceptionHelper : IO Error: Socket read timed
out
After estabilishin connection I can't use my EntityManager, because I get:
WARN 6812 --- [io-8080-exec-50] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 17008, SQLState: 08003
ERROR 6812 --- [io-8080-exec-50] o.h.engine.jdbc.spi.SqlExceptionHelper : Closed Connection
My connection properties:
spring.datasource.driverClassName=oracle.jdbc.driver.OracleDriver
spring.datasource.url=jdbc:oracle:thin:#...:..:..
spring.datasource.username=...
spring.datasource.password=...
spring.datasource.test-on-borrow=true
spring.datasource.test-while-idle=true
spring.datasource.validation-query=SELECT 1;
What should I do to reconnect that DB connection?
Try adding an external connection pool, such as HikariCP, since Spring Boot will automatically detect and use it:
If HikariCP is available we will use it.
If your using a JDBC4 Driver, the connection pool will validate the connection before handing it to you. If you are using an older driver you need to set this property:
connectionTestQuery
If your driver supports JDBC4 we strongly recommend not setting this
property. This is for "legacy" databases that do not support the JDBC4
Connection.isValid() API. This is the query that will be executed just
before a connection is given to you from the pool to validate that the
connection to the database is still alive. Again, try running the pool
without this property, HikariCP will log an error if your driver is
not JDBC4 compliant to let you know. Default: none
In a nutshell, when I try and get a connection after not having used a transaction for several minutes the first transaction setup fails.
When things are working, my logs show the following for a simple transaction:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] to manual commit
However, if I haven't had any activity for several minutes, instead I will get this message:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
My observations so far:
seems to be based on inactivity, but I've seen this behavior immediately after restarting my Tomcat although nothing else was hitting the database so I think it's inactivity against a network element such as my MySQL server.
when my application starts up, it makes a few non-transactional requests from the database which have not had any problems, so it seems related to transactions.
the timeout element in the #Transactional notation is not effective in this case. It seems to eventually time out, but takes 15 minutes (!).
while this transaction request is busy timing out, I can make subsequent requests successfully.
doesn't seem to be a starved local connection pool. I have seen this right after restarting the Tomcat.
When it finally times out (did I mention 15 minutes!) I get the following:
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [org.apache.commons.dbcp.PoolableConnection#3269c671] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [connection is closed] to manual commit
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Could not close JDBC Connection
ERROR: java.sql.SQLException: Already closed.
ERROR: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,312,604 milliseconds ago. The last packet sent successfully to the server was 924,748 milliseconds ago.
Caused by: java.net.SocketException: Connection timed out
Running Spring 3.1.1, mysql 5.1.32, commons-dbcp 1.4 and commons-pool 1.5.4.
Does anyone know what this is?
Your problem is that MySQL server timeouts idle JDBC connections. This has nothing to do with the TransactionManager set-up.
Have a look at your DataSource set-up. It shall tests connection on connection retrieval and/or validate idle connections in pool.
In commons-dbcp you can set-up test on connection retrieval via the testOnBorrow and validationQuery properties.