We Keep Coding

Mongo Connection issue with error "state should be: open"

I am running an event in Akka actor system, where we run multiple actors to query mongo db and retrieve data. Each actor queries for 1000 documents (each document's size is 9kb)
When running an event that is required to fire 14 actors to query for Mongo DB to retrieve 13000 documents.Once I experienced below exception, not sure why? Have anyone experienced this before?
2020-04-14 19:17:28,818 [erp-writer-actor-system-akka.actor.default-dispatcher-378] ERROR c.a.s.c.m.GlobalContextMongoClientService- 76cd7a80-83ef-4389-885a-be9caed77449 - Exception occured while reading data from cursor
java.lang.IllegalStateException: state should be: open
at com.mongodb.assertions.Assertions.isTrue(Assertions.java:70)
at com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:84)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:203)
at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:103)
at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46)
at com.xyz.smartconnect.commons.mongoclient.GlobalContextMongoClientService.findWorkers(GlobalContextMongoClientService.java:145)
at com.xyz.smartconnect.actors.QueryWorkersActor.lambda$createReceive$0(QueryWorkersActor.java:40)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at akka.actor.Actor$class.aroundReceive(Actor.scala:513)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:132)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:519)
at akka.actor.ActorCell.invoke(ActorCell.scala:488)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Suppressed: java.lang. IllegalStateException: state should be: open
at com.mongodb.assertions.Assertions.isTrue(Assertions.java:70)
at com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:84)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
at com.mongodb.operation.QueryBatchCursor.killCursor(QueryBatchCursor.java:261)
at com.mongodb.operation.QueryBatchCursor.close(QueryBatchCursor.java:147)
at com.mongodb.MongoBatchCursorAdapter.close(MongoBatchCursorAdapter.java:41)
at com.xyz.smartconnect.commons.mongoclient.GlobalContextMongoClientService.findWorkers(GlobalContextMongoClientService.java:149)

After running multiple tests and analyzing the logs carefully, I found the root cause. Below are the details.
While the application is using cursor to query data from mongoDb, connection has been released/closed. 'State should be : open' is complaining about a released connection.
In my case, my application experienced OutOfMemory, which caused disposing beans and releasing connections. Here is timeline of log events for this issue.
Since this is a memory issue for my case, fixing memory issue will fix below exception for me.
2020-04-19 12:57:32,981 [xyz-actor-system-akka.actor.default-dispatcher-72] ERROR a.a.ActorSystemImpl- - 413f9298-ca92-4744-913b-59934e4ce831 - exception on LARS’ timer thread
java.lang.OutOfMemoryError: GC overhead limit exceeded
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:269)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
2020-04-19 12:57:43,649 [Thread-19] INFO o.s.c.s.DefaultLifecycleProcessor- - - Stopping beans in phase 2147483647
2020-04-19 12:58:13,483 [Thread-19] INFO o.s.j.e.a.AnnotationMBeanExporter- - - Unregistering JMX-exposed beans on shutdown
2020-04-19 12:58:45,186 [localhost-startStop-2] INFO c.a.s.ApplicationContextListener- - - >>>>>>>>> Disposing beans
2020-04-19 12:59:00,182 [localhost-startStop-2] INFO c.a.s.c.SpringBeanDisposer- - - Mongo connections are released.
2020-04-19 12:59:09,591 [xyz-actor-system-akka.actor.default-dispatcher-73] ERROR c.a.s.c.m.GlobalContextMongoClientService- - 413f9298-ca92-4744-913b-59934e4ce831 - Exception occured while reading data from cursor
java.lang.IllegalStateException: state should be: open
at com.mongodb.assertions.Assertions.isTrue(Assertions.java:70)
at com.mongodb.connection.DefaultServer.getDescription(DefaultServer.java:114)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getServerDescription(ClusterBinding.java:81)
at com.mongodb.operation.QueryBatchCursor.initFromCommandResult(QueryBatchCursor.java:251)
at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:207)
at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:103)
at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46)

sql exceptions when trying to insert duplicate records into the database

I have the below method which checks if a user exists and then adds it to the database if it does not exist.
#Transactional(isolation = Isolation.READ_COMMITTED)
public Optional<User> create(User user) {
Optional<User> userOptional = userRepository.findByEmail(user.getEmail());
if (userOptional.isPresent()) {
return Optional.empty();
}
User savedUser = userRepository.save(user);
return Optional.of(savedUser);
}
two threads t1 and t2 are trying to save the same user with the same email(example#example.com). The following occurs:
t1 cant find user
t2 also cant find user
t1 adds user to database
t2 tries to add user to database but fails as record already exists and throws following error:
2019-10-02 18:28:43 WARN UKPC000029 --- [nio-8090-exec-2]
o.h.e.j.s.SqlExceptionHelper : SQL Error: 1062, SQLState: 23000
2019-10-02 18:28:43 ERROR UKPC000029 --- [nio-8090-exec-2]
o.h.e.j.s.SqlExceptionHelper : Duplicate entry 'example#example.com' for key 'UK_tcks72p02h4dp13cbhxne17ad'
2019-10-02 18:28:43 ERROR UKPC000029 --- [nio-8090-exec-2] o.h.i.ExceptionMapperStandardImpl : HHH000346: Error during managed flush [org.hibernate.exception.ConstraintViolationException: could not execute statement]
2019-10-02 18:28:43 DEBUG UKPC000029 --- [nio-8090-exec-2] o.s.o.j.JpaTransactionManager : Initiating transaction rollback after commit exception
org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [UK_tcks72p02h4dp13cbhxne17ad]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:296)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:253)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536)
In order to get around this error I tried to use serializble isolation level
#Transactional(isolation = Isolation.SERIALIZABLE)
However in the same scenario I got the following error:
2019-10-02 18:52:56 WARN UKPC000029 --- [nio-8090-exec-2] o.h.e.j.s.SqlExceptionHelper : SQL Error: 1213, SQLState: 40001
2019-10-02 18:52:56 ERROR UKPC000029 --- [nio-8090-exec-2] o.h.e.j.s.SqlExceptionHelper : Deadlock found when trying to get lock; try restarting transaction
2019-10-02 18:52:56 ERROR UKPC000029 --- [nio-8090-exec-2] o.h.i.ExceptionMapperStandardImpl : HHH000346: Error during managed flush [org.hibernate.exception.LockAcquisitionException: could not execute statement]
2019-10-02 18:52:56 DEBUG UKPC000029 --- [nio-8090-exec-1] o.s.o.j.JpaTransactionManager : Not closing pre-bound JPA EntityManager after transaction
2019-10-02 18:52:56 DEBUG UKPC000029 --- [nio-8090-exec-2] o.s.o.j.JpaTransactionManager : Initiating transaction rollback after commit exception
org.springframework.dao.CannotAcquireLockException: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.LockAcquisitionException: could not execute statement
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:287)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:253)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536)
I was under the impression that both transactions would be done 1 after the other?
The following link
isolation explained
SERIALIZABLE isolation level is the most restrictive of all isolation levels. Transactions are executed with locking at all levels (read, range and write locking) so they appear as if they were executed in a serialized way. This leads to a scenario where none of the issues mentioned above may occur, but in the other way we don't allow transaction concurrency and consequently introduce a performance penalty.
However both threads are still checking for the user email even though the 1st transaction hasnt finished? What is the best way to deal with this problem? Is using java locks acceptable in this case?

How to reconnect database if the connection closed in spring jpa?

I am using spring-boot, spring-jpa, mysql in my web application.When my application is running for some hours, I always got below exceptions:
2016-07-30 21:27:12.434 ERROR 13553 --- [http-nio-8090-exec-8] o.h.engine.jdbc.spi.SqlExceptionHelper : No operations allowed after connection closed.
2016-07-30 21:27:12.434 WARN 13553 --- [http-nio-8090-exec-5] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08003
2016-07-30 21:27:12.434 ERROR 13553 --- [http-nio-8090-exec-5] o.h.engine.jdbc.spi.SqlExceptionHelper : No operations allowed after connection closed.
2016-07-30 21:27:12.438 ERROR 13553 --- [http-nio-8090-exec-8] [.[.[.[.c.c.Go2NurseJerseyConfiguration] : Servlet.service() for servlet [com.cooltoo.config.Go2NurseJerseyConfiguration] in context with path [] threw exception [org.springframework.dao.DataAccessResourceFailureException: could not prepare statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not prepare statement] with root cause
java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3119) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3570) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4110) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2815) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155) ~[mysql-connector-java-5.1.25.jar!/:na]
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2322) ~[mysql-connector-java-5.1.25.jar!/:na]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82) ~[hibernate-core-4.3.11.Final.jar!/:4.3.11.Final]
I have checked that the database is running well. I have to restart my spring-boot application when that happened. How can I check what the problem is? Why the database connection got closed? If that happened, whether I can re-connect the database?
Below is my application.properties:
spring.datasource.url=jdbc:mysql://192.168.99.100:3306/test?characterEncoding=utf8
spring.datasource.username=admin
spring.datasource.password=123456
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.max-active=150

This seems like a common error with MySQL.
1) Add this to your application.properties and see how it goes:
spring.datasource.testOnBorrow=true
spring.datasource.validationQuery=SELECT 1
testOnBorrow is detailed in the spring doc and this other stackoverflow question. I'm however unable to find a reference on validationQuery in Spring's doc, but it seems to do the trick.
2) Or, you may use testWhileIdle as suggested here http://christoph-burmeister.eu/?p=2849
He suggests adding this to your application.properties:
spring.datasource.testWhileIdle = true
spring.datasource.validationQuery = SELECT 1
This solution is also mentionned in the other stackoverflow question, it was just not the accepted answer, but seems to be the solution for some.
3) In this case, they also added timeBetweenEvictionRunsMillis:
spring.datasource.testWhileIdle = true
spring.datasource.validationQuery = SELECT 1
spring.datasource.timeBetweenEvictionRunsMillis = 3600000
EDIT: Another stackoverflow question that covers this (with a very complete answer)

Spring sql-error-codes.xml does not show correct exception on timeout

The following error occurs when an exception occurs for myJDBCTemplate.queryForList() , before which a setQueryTimeout(1) is set. I have a database which has 1.2 million rows, and looking for the timeout exception to be printed or occur in the case when the statement is executed. So, basically, the timeout occurs but the exception does not mention that.
I am using springFramework-version => 4.1.3.RELEASE in pom.xml
INFO: org.springframework.beans.factory.xml.XMLBeanDefinitionReader - Loading XML bean definition for class path resource [org/springframework/jdbc/support/sql-error-code.xml]
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [select * from myTable where userCategory='1']; SQL state [70100]; error code [1317]; Query execution was interrupted; nested exception is java.sql.SQLException: Query execution was interrupted
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:416)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:471)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:481)
……..
caused by java.sql.SQLExcepion: Query execution was interrupted.
From the answer found at Query execution was interrupted, error #1317 states, the interruption occurs because of timeout, which I think is the possible cause.
Also, the exception states it is caused by java.sql.SQLException, but there are no exact details, why it occurred? So, I am not sure is it because of timeout or something else.

The error is clear in your stack trace:-
error code [1317]; Query execution was interrupted
, which means your query is being interrupted by an execution time limit. This error occurs when your query takes an unexpectedly long time to execute.
The error can be solved by fetching the data in batches by executing the query repeatedly for a certain data range.

Connection Pool configuration / error in SonarQube

from time to time I have the following error in my logs:
Error querying database. Cause: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
The error may exist in org.sonar.core.issue.db.ActionPlanMapper
The error may involve org.sonar.core.issue.db.ActionPlanMapper.findByKeys
The error occurred while executing a query
How should I adjust my connection pool settings so that this doesn't occur anymore?
sonar.jdbc.maxActive=25
sonar.jdbc.maxIdle=5
sonar.jdbc.minIdle=2
sonar.jdbc.maxWait=15000
sonar.jdbc.minEvictableIdleTimeMillis=600000
sonar.jdbc.timeBetweenEvictionRunsMillis=30000

According to http://jira.codehaus.org/browse/SONAR-188, you can try to update your connection settings in file: conf/sonar.properties
with adding "autoreconnect".
Example:
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8?autoReconnect=true