We have an application that has been running fine with Java 7 for a few years, but have problems upgrading to Java 8. The database connections seems to be the problem.
This is the case:
The application accesses more than 200 databases. Or, technically, 200 different schemas on one database installation. We use Atomikos for the data sources. When starting up the application with Java 7, there is probably some sequential iteration over the data sources, because we have set the max pool size to 5. It takes a little bit of time, but really no issues.
When we try running with Java 8, my guess is that it tries to do everything in parallell. The application crashes on startup like this:
05:21:39.611 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit 'database.np'
05:21:41.389 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit
'database.np_000000000_0000' 05:21:41.796 INFO [main]
o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA
container EntityManagerFactory for persistence unit
'database.np_000000001_0000' 05:22:11.890 WARN [main]
c.atomikos.jdbc.AtomikosSQLException - Connection pool exhausted - try
increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the
DataSourceBean. 05:22:41.991 WARN [main]
c.atomikos.jdbc.AtomikosSQLException - Connection pool exhausted - try
increasing 'maxPoolSize' and/or 'borrowConnectionTimeout' on the
DataSourceBean. 05:22:41.998 ERROR [main]
o.h.tool.hbm2ddl.SchemaValidator - HHH000319: Could not get database
metadata com.atomikos.jdbc.AtomikosSQLException: Connection pool
exhausted - try increasing 'maxPoolSize' and/or
'borrowConnectionTimeout' on the DataSourceBean.
at com.atomikos.jdbc.AtomikosSQLException.throwAtomikosSQLException(AtomikosSQLException.java:46)
~[transactions-jdbc-3.9.3.jar:na]
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:90)
~[transactions-jdbc-3.9.3.jar:na]
at com.atomikos.jdbc.AbstractDataSourceBean.throwAtomikosSQLException(AbstractDataSourceBean.java:85)
~[transactions-jdbc-3.9.3.jar:na]
So, we try the advice in the stacktrace, and increase maxPoolSize to a
suitable number above 200. Then we can actually start the application,
but... It seems like a complete overkill, and to make matters worse,
the application doesn't like having so many connections it doesn't
really use - the databases/schemas are accessed only now and then. We
see the following in the log:
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 358671ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 357953ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 357947ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': connection idle for 26425ms
11:08:26.970 [Atomikos:0] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceNp': current size: 1/4
11:08:29.983 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': trying to shrink pool
11:08:29.984 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': connection idle for
359790ms | (repeat 200 times...) | atomikos connection pool
'dataSourceRolp': connection idle for 263450ms 11:07:29.983
[Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool - atomikos
connection pool 'dataSourceRolp': connection idle for 263446ms
11:07:29.983 [Atomikos:1] DEBUG c.a.datasource.pool.ConnectionPool -
atomikos connection pool 'dataSourceRolp': connection idle for
263168ms 11:07:29.983 [Atomikos:1] DEBUG
c.a.datasource.pool.ConnectionPool - atomikos connection pool
'dataSourceRolp': current size: 1/208
Any thoughts? For legal reasons, we need to have the 200+ schemas separate. Other than that, we are fairly free to choose technical solutions as we want.
What database are you using? I had a quite similar issue during our database migration(From Oracle to PostgreSQL).
We defined the data source as JNDI, and to make the data source we also removed some attributed out from the jdbc interceptor.
If you are using PostgreSQL, please check the type of bean you defined for the data source too.
Related
We are facing now JDBC connection issue while we run load test. it goes well for few hours but suddenly thread starts going into pending state.
I have checked code and i could not find we are leaking any connection by intention. Only smoke I can see in logs is related to JPA call we make. but still it does not answer why we hitting no connection issue. any suggestion will be help ful.
Tech Stack -
We are using Spring Boot 2.4.7, Hibernate 5.4.25, Hikari CP 3.4.5, JPA 2.2
Logs -
1 [LogType:Usage] DEBUG [payment-service,,] [15] [HikariPool-1 connection adder] [com.zaxxer.hikari.pool.HikariPool] HikariPool-1 - Added connection org.mariadb.jdbc.MariaDbConnection#44a351b0
[HikariPool-1 housekeeper] [com.zaxxer.hikari.pool.ProxyLeakTask] Connection leak detection triggered for org.mariadb.jdbc.MariaDbConnection#44a351b0 on thread http-nio-8080-exec-10, stack trace follows \u2028java.lang.Exception: Apparent connection leak detected\u2028 at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)\u2028 at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)\u2028 at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)\u2028 at
[LogType:Usage] [com.zaxxer.hikari.pool.ProxyLeakTask] Previously reported leaked connection org.mariadb.jdbc.MariaDbConnection#44a351b0 on thread http-nio-8080-exec-10 was returned to the pool (unleaked)
[HikariPool-1 connection closer] [com.zaxxer.hikari.pool.PoolBase] HikariPool-1 - Closing connection org.mariadb.jdbc.MariaDbConnection#44a351b0: (connection has passed maxLifetime)
But in between we see and it is where our load test start failing ->
[LogType:Usage] DEBUG [14] [HikariPool-1 housekeeper] [com.zaxxer.hikari.pool.HikariPool] HikariPool-1 - Pool stats (total=15, active=15, idle=0, waiting=75)
Configuration -
connectionleakthreshold - 3000 ( we event tried with 30000 but found same behaviour)
connection-timeout - 30000
Please read for property - spring.jpa.open-in-view
https://raul8804.wordpress.com/2019/03/31/spring-boot-project-db-connection-leak-investigation/
This is very much strange how Spring adds transaction for all controllers.
I have a spring boot, hibernate using java app. I deploy it on a jetty webserver with multiple instances. if I have too (greater than 10) many instances I get
com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: remaining connection slots are reserved for non-replication superuser connections
Many of the connections (10x instance) show up idle
ps:
postgres 9104 0.0 0.0 248968 4392 ? Ss 08:07 0:00 postgres: user my_db 127.0.0.1(60095) idle
Hikari trace log for an instance:
2017-02-21 10:59:47.578 DEBUG 7401 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
setting
hikari.leakDetectionThreshold: 30000
Doesnt log anything interesing. I think this looks interesting HikariCP - connection is not available
Any ideas how i could debug this? Also I am on java 7, so hikari 2.4.7
I think you must increase the max_connections parameter in your PostgreSQL configuration.
See https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
You cannot allow the pool to keep more connections than your PostgreSQL installation allows.
I have an app that uses spring-boot,jpa-hiberanate with mysql.I am getting this error log
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 56,006,037 milliseconds ago. The last packet sent successfully to the server was 56,006,037 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
Here is my application.properties
# DataSource settings: set here configurations for the database connection
spring.datasource.url = jdbc:mysql://localhost:3306/test
spring.datasource.username = test
spring.datasource.password = test
spring.datasource.driverClassName = com.mysql.jdbc.Driver
# Specify the DBMS
spring.jpa.database = MYSQL
# Show or not log for each sql query
spring.jpa.show-sql = true
# Hibernate settings are prefixed with spring.jpa.hibernate.*
spring.jpa.hibernate.ddl-auto = update
spring.jpa.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
spring.jpa.hibernate.naming_strategy = org.hibernate.cfg.ImprovedNamingStrategy
To solve this issue I can use
spring.datasource.testOnBorrow=true
spring.datasource.validationQuery=SELECT 1
But I checked that it's not recommended .So can anyone suggest me what should I do to overcome this error
The easiest way is to specify the autoReconnect property in the JDBC url, although this isn't the recommended approach.
spring.datasource.url = jdbc:mysql://localhost:3306/test?autoReconnect=true
This can give issues when you have an active connection and during a transaction something happens and a reconnect is going to happen. It will not give issues when the connection is validated at the start of the transaction and a new connection is acquired at the start.
However it is probably better to enable validation of your connections during the lifetime of your application. For this you can specify several properties.
First start by specifying maximum number of connections you allow for the pool. (For a read on determining the max poolsize read this).
spring.datasource.max-active=10
You also might want to specify the number of initial connections
spring.datasource.initial-size=5
Next you want to specify the min and max number of idle connections.
spring.datasource.max-idle=5
spring.datasource.min-idle=1
To validate connection you need to specify a validation-query and when to validate. As you want to validate periodically, instead of when a connection is retrieved from the pool (this to prevent broken connections in your pool).
spring.datasource.test-while-idle=true
spring.datasource.test-on-borrow=true
spring.datasource.validation-query=SELECT 1
NOTE: The usage of a validation-query is actually discouraged with as JDBC4 has a better/different way of doing connection validation. HikariCP will automatically call the JDBC validation method when available.
Now that you are also validating while a connection is idle you need to specify how often you want to run this query for the connections and when a connection is considered idle.
spring.datasource.time-between-eviction-runs-millis=5000 (this is the default)
spring.datasource.min-evictable-idle-time-millis=60000 (this is also default)
This all should trigger validation of your (idle) connections and when an exception occurs or the idle period has passed your connections will be removed from the pool.
Assuming you are using Tomcat JDBC as the connection pool this is a nice read of what and how to configure.
UPDATE: Spring Boot 2.x switched the default connection pool to HikariCP instead of Tomcat JDBC.
In a nutshell, when I try and get a connection after not having used a transaction for several minutes the first transaction setup fails.
When things are working, my logs show the following for a simple transaction:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [jdbc:mysql://dev-db.example.com:3306/example, UserName=foo#1.2.3.4, MySQL Connector Java] to manual commit
However, if I haven't had any activity for several minutes, instead I will get this message:
DEBUG: org.springframework.transaction.annotation.AnnotationTransactionAttributeSource - Adding transactional method 'getRecord' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Creating new transaction with name [com.example.services.Service.getRecord]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_30; ''
My observations so far:
seems to be based on inactivity, but I've seen this behavior immediately after restarting my Tomcat although nothing else was hitting the database so I think it's inactivity against a network element such as my MySQL server.
when my application starts up, it makes a few non-transactional requests from the database which have not had any problems, so it seems related to transactions.
the timeout element in the #Transactional notation is not effective in this case. It seems to eventually time out, but takes 15 minutes (!).
while this transaction request is busy timing out, I can make subsequent requests successfully.
doesn't seem to be a starved local connection pool. I have seen this right after restarting the Tomcat.
When it finally times out (did I mention 15 minutes!) I get the following:
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Acquired Connection [org.apache.commons.dbcp.PoolableConnection#3269c671] for JDBC transaction
DEBUG: org.springframework.jdbc.datasource.DataSourceTransactionManager - Switching JDBC Connection [connection is closed] to manual commit
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Returning JDBC Connection to DataSource
DEBUG: org.springframework.jdbc.datasource.DataSourceUtils - Could not close JDBC Connection
ERROR: java.sql.SQLException: Already closed.
ERROR: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,312,604 milliseconds ago. The last packet sent successfully to the server was 924,748 milliseconds ago.
Caused by: java.net.SocketException: Connection timed out
Running Spring 3.1.1, mysql 5.1.32, commons-dbcp 1.4 and commons-pool 1.5.4.
Does anyone know what this is?
Your problem is that MySQL server timeouts idle JDBC connections. This has nothing to do with the TransactionManager set-up.
Have a look at your DataSource set-up. It shall tests connection on connection retrieval and/or validate idle connections in pool.
In commons-dbcp you can set-up test on connection retrieval via the testOnBorrow and validationQuery properties.
I am trying to get tweets via hbc-twitter4j-v3 . Example code is : https://github.com/twitter/hbc/blob/master/hbc-example/src/main/java/com/twitter/hbc/example/Twitter4jSampleStreamExample.java
For enabling authentication on proxy, I have also set system properties for host,port and authentication. But it is showing following error-
[main] INFO com.twitter.hbc.httpclient.BasicClient - New connection executed: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Establishing a connection
[main] INFO com.twitter.hbc.httpclient.BasicClient - Stopping the client: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[main] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 exit event - Stopped by user: waiting for 5000 ms
[main] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Client thread failed to finish in 5000 millis
[main] INFO com.twitter.hbc.httpclient.BasicClient - Successfully stopped the client: hosebird-client-0, endpoint: /1.1/statuses/sample.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Unknown host - stream.twitter.com
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 failed to establish connection properly
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hosebird-client-0 Shutting down httpclient connection manager
Any help??
Thanks in advance
Hopefully I haven't overlooked something but this is how it appears to me...
If by setting properties you mean the http.proxy* ones, I don't think it will work as hosebird-client uses Apache's HTTP client under the hood which doesn't seem to use them.
From a cursory glance at the code, specifically around the ClientBuilder, it doesn't look like hbc supports proxy configuration - perhaps they have a good reason not to or just don't need the feature themselves, maybe try requesting it?
It looks like one of the ways you can get HttpClient to use a proxy is by adding it to the HttpParams object, e.g.:
HttpParams params = ...
HttpHost proxy = new HttpHost(hostname, port);
params.setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
Whilst the HttpParams object isn't exposed anywhere you could potentially extend the ClientBuilder in order to supply your proxy configuration. If you look at the ClientBuilder#build() method, you can see where the HttpParams object is being set up. Good luck!
EDIT: Additionally, this issue indicates there are no plans to add proxy support directly in hbc.