I am using hibernate lucene search(4.5.1) in my application in the cloud environment. For each tenant a separate hibernate configuration is maintained (all the properties are same except hibernate.search.default.indexBase property. Each tenant has separate filesystem location). While starting the application I made the logic to index some table data by unique location for each tenant (eg: d:/dbindex/tenant1/, d:/dbindex/tenant2/) by calling the method Search.getFullTextSession(session).createIndexer().startAndWait(). For the first tenant every thing is fine, index was perfectly done. For the second tenant the startAndWait is not completed. It doesn't comes out from the startAndWait() but some time it is working. Some time doesn't comes out from the startAndWait(). After a serious debugging I found that BatchIndexingWorkspace has two thread producer and consumer where producer take the list of Id from DB and put it in queue and Consumer takes and indexes it. At the producer side (IdentifierProducer) a method named inTransactionWrapper has a statement
Transaction transaction = Helper.getTransactionAndMarkForJoin( session );
transaction.begin();
The statement transaction.begin() gets hangs and transaction doesn't begins so producer is not produced so consumer didn't indexed and startAndWait freezes. After a long search some post says that pool size will make a dead lock. But I am using BoneCPConnectionProvider with maxConnectionsPerPartition with 50 (per tenant). I monitored the active connection while starting, it doesn't exceed 10. More connection are available. But I don't know what is the problem.
Related
Context:
Spring Boot application with Spring JPA and MS SQL DB.
User registration process with insuranceNumber
insuranceNumber is not unique in DB but only for certain status (PROSPECTIVE, PARTICIPANT)
there is a duplication check in a service to check this
REST Controller ->
RegistrationService (#Transactional)
- do duplicate check (select * from customer where status in (PROSPECTIVE,PARTICIPANT) and insuranceNumber = XYZ -> no results good)
- insert new customer to DB
Issue:
If tested, the duplication checks works but sometimes I have duplicates of insuranceNumber in PROSPECTIVE status
Assumption:
due to multiple REST requests in short time I have multiple threads (let's assume 3)
Thread 1 "duplicate check" - all fine
Thread 2 "duplicate check" - all fine (Thread 1 is not comitted yet)
Thread 1 insert do DB, commit TX
Thread 2 insert do DB, commit TX ## the issue, now customer with same insurance number in same status
Thread 3 "duplicate check" - fails - als expected
Possible Solutions:
Frontend: prevent this multiple requests. Out of scope. I want to be sure from backend
DB: create something on DB site (database trigger) to do the same duplication check again. Feels wrong as it duplicates the logic of the duplication check. Would also raise a different exception than if raised in Java already
Java code: RegistrationService with synchronized method. Would slow down registration for everybody. For me it would be enough that only one insurance number is allowed to enter the registration method.
Are there more ideas?
Play around with isolation levels for the DB?
Prevent to enter the registration method if one thread has already entered the method with same insurance number?
The only reliable approach to prevent duplicates in DB is to create unique index, in your particular case that should be filtered unique index:
CREATE UNIQUE NONCLUSTERED INDEX CUSTOMERS_UK
ON CUSTOMERS(insuranceNumber)
WHERE status IN ('PROSPECTIVE','PARTICIPANT')
Another options are:
application locks in MSSQL
lock by a key in Java, however, that won't work in case of multiple instance deployment
I have a rest API in Springboot using Hikari for connection pooling. Hikari is used with default configurations (10 connections in the pool, 30 sec timeout waiting for a connection). The API itself is very simple
It first makes a JPA repository query to fetch some data from a PostgresDB. This part takes about 15-20milliseconds.
It then sends this data to a remote REST API that is slow and can take upwards of 120seconds.
Once the remote API responds, my API returns the result back to the client. A simplified version is shown below.
public ResponseEntity analyseData(int companyId) {
Company company = companyRepository.findById(companyId);//takes 20ms
Analysis analysis = callRemoteRestAPI(company.data) //takes 120seconds
return ResponseEntity.status(200).body(analysis);
}
The code does not have any #Transactional annotations. I find that the JDBC connection is held for the entire duration of my API (i.e ~ 120s). And hence if we get more than 10 requests they timeout waiting on the hikari connection pool (30s). But strictly speaking my API does not need the connection after the JPA query is done (Step 1 above).
Is there a way to get spring to release this connection immediately after the query instead of holding it till the entire API finishes processing ? Can Spring be configured to get a connection for every JPA request ? That way if I have multiple JPA queries interspersed with very slow operations the server throughput is not affected and it can handle more than 10 concurrent API requests. .
Essentially the problem is caused by the Spring OpenSessionInViewFilter that "binds a Hibernate Session to the thread for the entire processing of the request" This essentially acquires a connection from the pool when the first JPA query is executed and then holds on to it till the request is processed.
This page - https://www.baeldung.com/spring-open-session-in-view provides a clear and concise explanation about this feature. It has its pros and cons and the current opinion seems to be divided on its use.
Problem Statement
We have been using H2 in embedded mode for a while now. It has a connection pool configured above it. Following is the current pool configuration:
h2.datasource.min-idle=10
h2.datasource.initial-size=10
h2.datasource.max-active=200
h2.datasource.max-age=600000
h2.datasource.max-wait=3000
h2.datasource.min-evictable-idle-time-millis=60000
h2.datasource.remove-abandoned=true
h2.datasource.remove-abandoned-timeout=60
h2.datasource.log-abandoned=true
h2.datasource.abandonWhenPercentageFull=100
H2 config:
spring.h2.console.enabled=true
spring.h2.console.path=/h2
h2.datasource.url=jdbc:h2:file:~/h2/cartdb
h2.server.properties=webAllowOthers
spring.h2.console.settings.web-allow-others=true
h2.datasource.driver-class-name=org.h2.Driver
*skipping username and password properties.
We have verified that the above configuration takes effect by logging the pool properties.
The issue with this setup is that we are observing regular(though intermittent) connection pool exhaustion and once the pool hits the max limit it starts throwing the following exception for some queries.
SqlExceptionHelper.logExceptions(SqlExceptionHelper.java:129) - [http-apr-8080-exec-38] Timeout: Pool empty. Unable to fetch a connection in 3 seconds, none available[size:200; busy:200; idle:0; lastwait:3000].
And thereafter it fails to recover from this state even after many hours until we restart the web server(tomcat in this case).
H2 driver dependency:
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.196</version>
<scope>runtime</scope>
</dependency>
Query Pattern & Throughput
We use h2 to load up some data for every request, then execute a few(about 50) SELECT queries and finally delete the data. This results into consistent 30k-40k calls per minute(except off hours) on h2(according to new relic monitoring).
Every read operation acquires a new connection and releases the same after execution.
EntityManager entityManager = null;
try {
entityManager = entityManagerFactory.createEntityManager();
Query query = entityManager.createNativeQuery(sqlQuery);
query.setParameter("cartId", cartId);
List<String> resultList = query.getResultList();
return resultList;
} finally {
if(null != entityManager) { entityManager.close(); }
}
Observations
After application restart the pool utilization is minimal until at one moment when the pool utilization abruptly shoots up and eventually reaches max limit. This happens over the course of 1-2 days.
Once the pool hits the maximum connection limit, the borrowed connection count increases at a faster pace as compared the the returned connection count which otherwise remains very close to one another.
At the same time the abandoned connection count also starts increasing along with the abandon logs.
Interestingly the query response times remains the same after pool exhaustion. So this kind of rules out slow query.
This issue has happened at even the oddest of the hours when the traffic is minimum. So it has no relation to the traffic.
Please guide us in the right direction to solve this issue.
UPDATE
Recently we discovered the following causes in our stack trace when one such incident occured:
Caused by: org.h2.jdbc.JdbcSQLException: Database may be already in
use: null. Possible solutions: close all other connection(s); use the
server mode [90020-196]
Caused by: java.lang.IllegalStateException:The file is locked:
nio:/root/h2/cartdb.mv.db [1.4.196/7]
Caused by: java.nio.channels.OverlappingFileLockException
So after digging into this we have decided to move to in-memory mode as we don't require to persist the data beyond the application's life time. As a result, the file lock should not occur thereby reducing or eradicating this issue.
Will come back and update in either case.
Since the last update on the question:
After observing the performance for quite some time we have come to the conclusion that using H2 in file-mode(embedded) was somehow leading to file lock exceptions periodically(though irregular).
Since our application does not require to persist the data beyond the application's lifetime, we decided to move to pure in-memory mode.
Though the file lock exception's mystery still needs to disclosed.
In JBOSS EAP6 I've got a long running MDB thread listening to a JMS Queue. It received a Text Message with a DB key of work it should process (loop).
During its execution I noticed that new threads spawn new MDB instances, leading to inconsistencies. I do want to prevent that in a programmatic manner or in a configuration manner whithout changing performace. So, for instance check in the onMessage that work is ongoing. I can't change the DB Model.
Since I'm running in a single VM I'm on the verge (last resort) of using a static Set that stores the DB key. (I'm a bit under time pressure to fix this).
The problem was caused by the fact I forgot the specify the transaction time-out. Hence the default time-out seems to kick in.
The problem was solved by adding the transaction time out:
#ActivationConfigProperty( propertyName = "transactionTimeout", propertyValue = "10800" )
I have a service which is running using executor in java.In the main method of that service is as follows
public void method()
{
// will get some records from database
process records one by one
}
For example in my database I have 100 records , after 49 records processed I stopped my server.When I restart the server again it is running from starting means from 1st record.
Is there any possibility to start the service from 50th record.
Possible solution:
whenever server started need to check in previous iteration how many records processed by looking into the database (by maintaining a flag ).Once I found that records , I can skip those records .
Is there any alternative for this or any framework in java which can handle in a proper manner.Please correct me if my solution is not correct or any better solution is available
NOTE:Here we don't require any transaction management