I have two servers connected to the same database. Both have scheduled jobs I don't really care which one runs the scheduled jobs as long as only one does. So the idea was to keep a key-value pair in DB and whichever reads the value as 0 first gets to run the scheduled job.
Ideally this would work as so:
App A and App B run the scheduled job at the same time.
App A access the DB first, locks the table for reading & writing.
App A sets value to 1 and releases the lock.
App A starts working on the scheduled job.
App B reads the value 1 from it's DB request and does not run the scheduled job.
I have a config table where I keep status on my locks.
config:
name: VARCHAR(55)
value: VARCHAR(55)
The repository:
#Repository
public interface ConfigRepository extends CrudRepository<Config, Long> {
#Lock(LockModeType.PESSIMISTIC_READ)
Config findOneByName(String name);
#Lock(LockModeType.PESSIMISTIC_WRITE)
<S extends Config> S save(S entity);
}
The service:
#Service
public class ConfigService {
#Transactional
public void unlock(ConfigEnum lockable) {
Config lock = configRepository.findOneByName(lockable.getSetting());
lock.setValue("0");
configRepository.save(lock);
}
#Transactional
public void lock(ConfigEnum lockable) {
Config lock = configRepository.findOneByName(lockable.getSetting());
lock.setValue("1");
configRepository.save(lock);
}
#Transactional
public boolean isLocked(ConfigEnum lockable) {
Config lock = configRepository.findOneByName(lockable.getSetting());
return lock.getValue().equals("1");
}
}
The Scheduler:
#Component
public class JobScheduler {
#Async
#Scheduled("0 0 1 * * *")
#Transactional
public void run() {
if (!configService.isLocked(ConfigEnum.CNF_JOB.getJobName())) {
configService.lock(ConfigEnum.CNF_JOB.getJobName());
jobService.run();
configService.unlock(ConfigEnum.CNF_JOB.getJobName());
}
}
}
However I have noticed that the scheduled jobs still run at the same time on both apps. At times one will throw a deadlock but it appears that Spring retries the transaction if it hits a deadlock. At which time it appears that the one app has finished so this one begins the same job again (not sure).
The tasks are not that short that a lock could be established, table updated, task run and lock released. I would like to keep this really simple without involving additional libraries like Quartz or ShedLock.
I think your transactions are too short. You don't start a transaction in the run method, but each ConfigService method is transactional. Most likely each method gets a new transaction and commits when done. A commit will release the lock, so there is a race condition between isLocked and lock.
Combine isLocked and lock:
#Transactional
public boolean tryLock(ConfigEnum lockable) {
Config lock = configRepository.findOneByName(lockable.getSetting());
if("1".equals(lock.getValue()) {
return false;
}
lock.setValue("1");
configRepository.save(lock);
return true;
}
This checks and writes in the same transaction and should work.
As a side note it is a dangerous method. What happens if the node that has the lock dies? There are many possible solutions. One is to lock a specific record and keep that lock throughout the job. The other node cannot proceed and if the first one dies the lock will be released. Another is to use a timestamp instead of 1 and require the timestamp to be updated on a regular basis by the owner. Or you could introduce something like Zookeeper.
Related
public class A {
String name = "foo";
}
#Service
public class B {
private final Repository repo;
public void someMethod(){
A a = repo.findByName("foo");
a.name = Thread.currentThread().getName();
repo.save(a);
}
}
Imagine that 2 thread executed someMethod at the same time. Especially in jpa hibernate implementation.
My opinion is there is a race condition.
First and second threads are obtained same object with name foo. If i am not wrong, without optimistic lock there will be error in this scenario.
Also optimistic lock throws exception so do i need to use pessimistic lock to work correctly?
Also if i open second level cache with distributed in memory cache(Redis, hazelcast), what will happen?
It is a IOT project and million devices are calling this api and service.
Do i need to approach with eventual consistency?
In a Spring boot with data JPA project with Hibernate on a PostgreSQL database, multiple tasks are executing simultaneously. There's a TaskExecutor pool and a database connection pool.
Sometimes these tasks require some of the same objects (update: with object we mean objects stored in the database). In an attempt to ensure the tasks don't conflict (update: "don't try to access/modify the same records at the same time"), a locking service was created. A task gets a lock on the objects it requires and only releases the lock when the task is done, at which time the next task can get a lock on them and start its work.
In practice, this isn't working. One particular case of a record being deleted in task A and still being visible during part of task B keeps popping up. The actual exception is a foreign key constraint not being fulfilled: task B first selects the (deleted) object as one for which a relationship is to be created (so task B still sees the deleted object at this point!), but then upon creation of the relationship in task B it fails because the deleted object is no longer present.
After consultation with a colleague, the idea came up that flushing a repository isn't quite the same as committing. Hence, task A unlocks when its logic is done and changes are flushed, but the changed data has not yet been actually committed to the database. In the mean time, task B gets a lock and starts reading the data and only a little bit later the commit for task A happens.
To make sure the lock of task A is only released after its database changes have been committed, I tried this code:
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
logDebugMessage("after completion method called");
// Release the locks
if(lockObtained) {
logDebugMessage("releasing locks");
lockService.releaseLocks(taskHolder.getId());
logDebugMessage("done releasing locks");
}
else
logDebugMessage("no locks to release");
}
});
That, in itself, didn't make the issue disappear. Next brainwave was that the next task, task B, already has a transaction open while it's waiting for a lock. When it gets a lock, it reads using this already-open transaction and then ?for some reason? reads data from before the commit. I admit this doesn't make much sense, but some desperation is beginning to set in. Anyway, with this additional idea, every task is now run as such:
#Override
public void run() {
try{
// Start the progress
taskStartedDateTime = LocalDateTime.now();
logDebugMessage("started");
// Let the task determine which objects need to be locked
List<LockableObjectId> objectsToLock = getObjectsToLock();
// Try to obtain a lock
lockObtained = locksObtained(objectsToLock);
if(lockObtained) {
// Do the actual task in a new transaction
TransactionTemplate transactionTemplate = new TransactionTemplate(transactionManager);
transactionTemplate.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
#Override
protected void doInTransactionWithoutResult(TransactionStatus status) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
#Override
public void afterCompletion(int status) {
logDebugMessage("after completion method called");
// Release the locks
if(lockObtained) {
logDebugMessage("releasing locks");
lockService.releaseLocks(taskHolder.getId());
logDebugMessage("done releasing locks");
}
else
logDebugMessage("no locks to release");
}
});
try{
// Run the actual task
runTask();
But this still doesn't resolve the issue.
Is it possible the commit to the database was done from Java side and task B reads the database before the commit is done in the database itself? Does the afterCompletion method get called after Java sent the commit, but before the database has actually executed it? If so, is there a way to get a database confirmation that the commit has actually been executed?
Or are we entirely on the wrong track here?
currently, I have a web application based on Java 7, tomcat 7 and Spring 4 that invokes a thread on tomcat startup.
This thread is always alive and the java code is:
public class Scheduler {
Queue<Long> queue = new ArrayBlockingQueue<Long>();
private static class ThreadExecutor implements Runnable
{
.......
#Override
public void run()
{
while(true)
{
Long ID = queue.get();
if(ID != null)
{
Object o = webFacade.get(ID);
//Exec....
}
else
{
try
{
Thread.sleep(30000);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
}
}
An external event fills the queue with the Object's ID.
With one tomcat this thread works well, but now I nedd to add onother tomcat, so I want to introduce Quartz in clustered mode.
I've configured Quartz in my project and it seems to work, but now how can I "translate" this class using Quartz?I want that only one thread is active because it is very expensive for my Database.
Thanks in advance
In general Quartz while being run in a cluster mode guarantees that the job will be triggered (and handled) on one server only.
So Job will be the task that you execute (in other words, what should be executed).
Now Quartz also introduces the concept of Trigger which basically defines when the job will be triggered.
From your code snippet, you run the job every 30000 ms = 30 seconds. So you'll trigger your stuff every 30 seconds (SimpleTrigger will do the job).
So, the 'while' loop goes away, it will be handled by quartz automatically.
In job you'll only work with a queue. Its unclear who fills this queue, but it looks like a different question.
It's hard to say exactly how you translate the queue, but in general job should
Get from queue
Call webFacade just like now
That's it. Last but not the least, Spring has a beautiful integration with Quartz. See Chapter 33.6.
I am using Apache TomEE 1.5.2 JAX-RS, pretty much out of the box, with the predefined HSQLDB.
The following is simplified code. I have a REST-style interface for receiving signals:
#Stateless
#Path("signal")
public class SignalEndpoint {
#Inject
private SignalStore store;
#POST
public void post() {
store.createSignal();
}
}
Receiving a signal triggers a lot of stuff. The store will create an entity, then fire an asynchronous event.
public class SignalStore {
#PersistenceContext
private EntityManager em;
#EJB
private EventDispatcher dispatcher;
#Inject
private Event<SignalEntity> created;
public void createSignal() {
SignalEntity entity = new SignalEntity();
em.persist(entity);
dispatcher.fire(created, entity);
}
}
The dispatcher is very simple, and merely exists to make the event handling asynchronous.
#Stateless
public class EventDispatcher {
#Asynchronous
public <T> void fire(Event<T> event, T parameter) {
event.fire(parameter);
}
}
Receiving the event is something else, which derives data from the signal, stores it, and fires another asynchronous event:
#Stateless
public class DerivedDataCreator {
#PersistenceContext
private EntityManager em;
#EJB
private EventDispatcher dispatcher;
#Inject
private Event<DerivedDataEntity> created;
#Asynchronous
public void onSignalEntityCreated(#Observes SignalEntity signalEntity) {
DerivedDataEntity entity = new DerivedDataEntity(signalEntity);
em.persist(entity);
dispatcher.fire(created, entity);
}
}
Reacting to that is even a third layer of entity creation.
To summarize, I have a REST call, which synchronously creates a SignalEntity, which asynchronously triggers the creation of a DerivedDataEntity, which asynchronously triggers the creation of a third type of entity. It all works perfectly, and the storage processes are beautifully decoupled.
Except for when I programmatically trigger a lot (f.e. 1000) of signals in a for-loop. Depending on my AsynchronousPool size, after processing signals (quite fast) in the amount of about half of that size, the application completely freezes for up to some minutes. Then it resumes, to process about the same amount of signals, quite fast, before freezing again.
I have been playing around with AsynchronousPool settings for the last half hour. Setting it to 2000, for instance, will easily make all my signals be processed at once, without any freezes. But the system isn't sane either, after that. Triggering another 1000 signals, resulted in them being created allright, but the entire creation of derived data never happened.
Now I am completely at a loss as to what to do. I can of course get rid of all those asynchronous events and implement some sort of queue myself, but I always thought the point of an EE container was to relieve me of such tedium. Asynchronous EJB events should already bring their own queue mechanism. One which should not freeze as soon as the queue is too full.
Any ideas?
UPDATE:
I have now tried it with 1.6.0-SNAPSHOT. It behaves a little bit differently: It still doesn't work, but I do get an exception:
Aug 01, 2013 3:12:31 PM org.apache.openejb.core.transaction.EjbTransactionUtil handleSystemException
SEVERE: EjbTransactionUtil.handleSystemException: fail to allocate internal resource to execute the target task
javax.ejb.EJBException: fail to allocate internal resource to execute the target task
at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:81)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:240)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
at <<... my code ...>>
...
Caused by: java.util.concurrent.RejectedExecutionException: Timeout waiting for executor slot: waited 30 seconds
at org.apache.openejb.util.executor.OfferRejectedExecutionHandler.rejectedExecution(OfferRejectedExecutionHandler.java:55)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:75)
... 38 more
It is as though TomEE would not do ANY queueing of operations. If no thread is free to process in the moment of the call, tough luck. Surely, this cannot be intended..?
UPDATE 2:
Okay, I seem to have stumbled upon a semi-solution: Setting the AsynchronousPool.QueueSize property to maxint solves the freeze. But questions remain: Why is the QueueSize so limited in the first place, and, more worryingly: Why would this block the entire application? If the queue is full, it blocks, but as soon as a task is taken from it, another should pop in, right? The queue appears to be blocked until it is completely empty again.
UPDATE 3:
For anyone who wants to have a go: http://github.com/JanDoerrenhaus/tomeefreezetestcase
UPDATE 4:
As it turns out, increasing the queue size does NOT solve the problem, it merely delays it. The problem remains the same: Too many asynchronous operations at once, and TomEE chockes so bad, that it cannot even undeploy the application on termination anymore.
So far, my diagnosis is that the task cleanup does not work properly. My tasks are all very small and fast (see the test case on github). I was already afraid that it would be OpenJPA or HSQLDB slowing down on too many concurrent calls, but I commented out all em.persist calls, and the problem remained the same. So if my tasks are quite small and fast, but still manage to block out TomEE so bad that it could not get any further task in after 30 seconds (javax.ejb.EJBException: fail to allocate internal resource to execute the target task), I would imagine that completed tasks linger, clogging up the pipe, so to speak.
How could I resolve this issue?
Basically BlockingQueues use locks to ensure the consistency of data and avoid data loss, so in too highly concurrent environment it will reject a lot of tasks (your case).
You can play on trunk with the RejectedExecutionHandler implementation to retry to offer the task. One implementation can be:
new RejectedExecutionHandler() {
#Override
public void rejectedExecution(final Runnable r, final ThreadPoolExecutor executor) {
for (int i = 0; i < 10; i++) {
if (executor.getQueue().offer(r)) {
return;
}
try {
Thread.sleep(50);
} catch (final InterruptedException e) {
// no-op
}
}
throw new RejectedExecutionException();
}
}
It even works better with random sleep (between min and max).
The idea is basically: if the queue is full, wait some short time to reduce the concurrency.
configurable through WEB-INF/application.properties https://issues.apache.org/jira/browse/TOMEE-1012
I have a couple of questions about Transactions in Spring if you may.
Let's suppose i have this DAO class :
public class MyDAO {
/**
* verifies if a certain record in DB contains 'True' in a certain Column named publishFlag
*/
#Transactional
public bloolean isBeingPublished(Long recordID){
...
}
/**
* sets the record's publishFlag column to true indicating that it's being published
*/
#Transactional
public boolean setBeingPublished(Long recordID){
...
}
}
And the following class using it :
public class MyClass {
#Autowired
MyDAO dao;
public void publishRecords(List<Long> ids){
for(Long id : ids){
if(!dao.isBeingPublished(id)){
dao.setBeingPublished(id);
//do something to publish the record
}
}
}
}
My questions are :
First of all, will the !dao.isBeingPublished(id) and dao.setBeingPublished(id) be executed in the same transaction or in separate ones?
Second question's about concurrency, Multiple MyClass instances can be created and concurrent calls to the publishRecord method can occur, so two concurrent calls to !dao.isBeingPublished(id) might both give the same result and thus making the record published twice!
I would consider making the publishRecords synchronized but the application may be deployed on multiple servers which renders the synchronized declaration useless, hence my question about transactions since the database is the only shared resource between the apps deployed on those servers.
What would be the solution to my problem exactly? I read about spring's transaction propagation and found out that REQUIRES_NEW would create a new transaction even if one is currently being executed, but still, I just can't see how that's going to be a solution to my problem.
Thank you in advance for your help.
Few things need consider, DAO is focus on operation on single entity, and service is focus on operation of one or more entities, so the transaction should put on service layer, so you can reuse DAO's operation without any transaction, but let service to decide when start and end transaction
It is not in single transaction, but two separate transaction.
That is the problem concurrency issue with your current design, see the following suggestion.
Interface
public interface MyClass {
public void publishRecords(List<Long> ids);
}
Implementation
#Service
#Transactional(readOnly = false)
class DefaultMyClass implements MyClass {
#Autowired
MyDAO dao;
// single transaction
#Override
public void publishRecords(List<Long> ids) {
for(Long id : ids){
if(!dao.isBeingPublished(id)){
dao.setBeingPublished(id);
//do something to publish the record
}
}
}
}
DAO
class MyDAO {
public bloolean isBeingPublished(Long recordID){
// bigbang
}
public boolean setBeingPublished(Long recordID){
// bigbang
}
}
Using the above design, both problems are being resolved.
First of all, will the !dao.isBeingPublished(id) and
dao.setBeingPublished(id) be executed in the same transaction or in
seperate ones?
Unless there's a method annotated with #Transactional further up the stack, they will be occurring in separate transactions, so yes you will have a potential for a race condition.
If I were you, I would toss the isBeingPublished and setBeingPublished in favor of a single #Transactional publishIfPossible method that returns a boolean value of whether it was able to acquire the database row lock and do the publish operation.