In my spring web app, I have a service method containing a block of code guarded by a lock.
Only a single thread can enter the code block at a time.
This works fine in a non clustered environment but fails in a clustered one. In a clustered environment, within a node, synchronization happens but among different nodes, code block is executed in parallel. Is this because in each node a separate Lock object is created ?
Can anyone advise me ?
Code Sample
//Service Class
#Service
class MyServiceClass {
private final Lock globalLock;
#Autowired
public MyServiceClass(#Qualifier("globalLock") final Lock globalLock){
this.globalLock = globalLock;
}
public void myServiceMethod(){
...
globalLock.lock();
try {
...
}
finally {
globalLock.unlock();
}
...
}
}//End of MyServiceClass
//Spring Configuration XML
<bean id="globalLock" class="java.util.concurrent.locks.ReentrantLock" scope="singleton" />
If you want to synchronize objects in a cluster environment, this meaning many VMs involved, your solution would involve some kind of communication between the different VMs involved.
In this case, it will require some imagination to get the thing done: you will need the mutual exclusion implemented on some object that is common to all the VMs involved, and that may escalate when you put additional machines into the cluster. Have you thought some solution based on JNDI? Here you have something on it, but I am afraid it looks rather an academic discussion:
http://jsr166-concurrency.10961.n7.nabble.com/Lock-implementation-td2180.html
There is always the chance to implement something based on DB mechanisms (always thinking that your DB is a central and common resource to all the nodes in the cluster). You could devise something based on some SELECT FOR UPDATE mechanism implemented in your database, over some table used only for synchronization...
You have an interesting problem! :) Good luck
You are right, the reason is that each node has it's own lock. To solve this, consider introducing in the database a table SERVICE_LOCKS, with the columns service class name, service Id, lock status and acquisition timestamp.
For service Id make each service generate a unique distributed Id using UUID.randomUUID().
To acquire the locks, issue an update to try to grab it, and then query it to see if you have the lock. But don't do select, check and then update. Locks older than a certain amount of time should be not taken into account.
This is an implementation of to the coarse grained lock design pattern, where an application level pessimistic lock is acquired to lock shared resources.
Depending on the business logic on the services and the type of transaction manager you use, increasing the isolation level of the service method to REPEATABLE_READ might be an option.
For a solution that does not involve the database, have a look at a framework for distributed concurrent processing based on the Actor concurrency model - The Akka Framework (click Remoting button).
Related
Let's imagine a problem:
I have a REST service, which is implemented using Java/MySQL/Spring and HTTP/JSON technologies.
The clients of the REST service are mobile applications.
So it's possible that someone will decompile the code and will get the API of the REST service.
( Yes, the code is obfuscated etc, but anyway ).
Problem: there is a POST method to send money to other user of the application.
I am worried, that someone can get the API, write a bot and make this POST request 500 or 5,000 or even 50,000 times per second.
As a result, he might send more money than he actually has, because if 1000 requests are processed simultaneously then the balance checking might be
successful for all 1000 request, however the real amount of money on an account might be enough only for, lets say, 50 requests.
So, basically, it's more like the standard "race" condition with multiple threads.
The problem is, that I have multiple servers and they are not related with each other anyhow.
So, 300 request can come to server A, 300 requests can come to server B and rest requests can come to server C.
The best idea what I have is to use something like "SELECT ... FOR UPDATE" and synchronize on database level.
However, I would like to consider another solutions.
Any ideas or suggestions?
You have a few options:
Rely on ACID implementation of the database (MySQL in your case). Assuming you are using InnoDB engine, you need to choose the right transaction isolation level (SET TRANSACTION syntax) in combination with the right locking reads mechanism (SELECT ... FOR UPDATE and SELECT ... LOCK IN SHARE MODE Locking Reads). You need to understand these concepts well in order to do the right choice. It might be possible that simply using the right isolation level will already prevent the race condition even without the locking reads. The cons are you are trading off consistency for scalability and tying your application to RDBMS database thus it will be more difficult for you to move to NoSQL for example.
Decompose your back end into web tier and service tier (option suggested by atk in the comments). This will allow you to scale the web tier instances independently while keeping a single service tier instance. Having a single service tier instance makes it possible to use Java synchronization mechanisms such as synchronised blocks or ReadWriteLock. Although this solution will work I wouldn't recommend it since it reduces the scalability of your service tier.
This is an enhancement of the previous option. You can use a Distributed lock manager instead of built-in java synchronization mechanisms. It will allow you to scale your web tier and service tier independently.
For mission critical applications, it's best to have multiple levels of locking mechanisms.
"SELECT ... FOR UPDATE" is a good way to do so, but they're pretty expensive, and when you try to bombard this with Charles, you'll see that your upper API stack will suffer, and that simple mechanism will cripple your infrastructure pretty easily, much like a DDoS event.
Implement it at the Load Balancer/Proxy layer first, to throttle N-number of requests per specified intervals from a single IP address.
Then apply a shared cache layer lock, where all your boxes go to synchronize on certain keys depending on which critical transaction you want to lockdown. For example, you can use Redis GETSET or INCR functionality to atomically set a flag, before entering the critical code path. Reject anything else quickly to avoid those bad actors from holding on to CPU/memory.
You can also implement things like an APC cache (before hitting your Redis / Memcache cluster) to do similar locking on a per box basis. This is quicker as there's no network latency involved.
The above are necessary on top of using "SELECT ... FOR UPDATE"
I have a method in which some database insert operations are happening using hibernate and i want them to be thread safe. The method is getting some data in parametres and its a possiblity that sometimes two calls are made with same data at same point of time.
I can't lock those tables because of performance degradation. Can anyone suggest making the method as synchronized will solve issue?
Synchronizing a method will ensure that it can only be accessed by one thread at a time. If this method is your only means of writing to the database, then yes, this will stop two threads from writing at the same time. However, you still have to deal with the fact that you have multiple insert operations with the same data.
You should let Hibernate handle the concurrency, that's what it is meant to do. Don't assume Hibernate will lock anything: it supports optimistic transactions for exactly this purpose. Quote from the above link:
The only approach that is consistent with high concurrency and high scalability, is optimistic concurrency control with versioning. Version checking uses version numbers, or timestamps, to detect conflicting updates and to prevent lost updates. Hibernate provides three possible approaches to writing application code that uses optimistic concurrency.
Database Concurrency is handled by transactions. Transactions have the Atomic Consistent Isolated Durable (ACID) properties. They provide isolation between programs accessing a database concurrently. In the Hibernate DAO template of spring framework there are single line methods for CRUD operations on the database. When used individually these don't need to be synchronized by method. Spring provides declarative (XML), programmatic and annotation meta-data driven transaction management if you need to declare "your method" as transactional with specific propagation settings, rollbackFor settings, isolation settings. So in "your method" you can do multiple save,update,deletes etc and the ORM will ensure that it is executed with the transaction settings you have given in the meta-data.
Another issue is that the thread has to have the lock on all the objects that are taking part in the transaction.Otherwise the transaction might fail or the ORM will persist stale data. In another situation it can result in a deadlock because of lock-ordering. I think this is what really answers your question.
Both objects a and b have an instance variable of the type Lock. A boolean flag can be used to indicate the success of the transaction. The client code can retry the same transaction if it fails.
if (a.lock.tryLock()) {
try {
if (b.lock.tryLock()) {
try {
// persist or update object a and b
} finally {
b.lock.unlock();
}
}
} finally {
a.lock.unlock();
}
}
The problem with using synchronized methods is that it locks up the entire Service or DAO class making other service methods unavailable to other threads. By using individual locks on objects we can gain the advantage of fine grained concurrency.
No. This method probably uses another methods and objects, which may be not thread safe. synchronized makes threads to use that's method's object monitor only once at a time, so it makes thread-safe a method with respect to the object.
If you are sure that all other threads use shared functionality only with this method, then making it synchronized may be sufficient.
Choosing the best strategy depends on the architecture, sometimes to increase performance seems to be easier to use the trick like method synchronization, but this is bad approach.
There's no doubts, you should use transactions, and if with that strategy you're facing performance issues you should optimize your db queries or db structure.
Please remember that "Synchronization" should be as much as possible atomic.
I have a tasks thread running in two separate instances of tomcat.
The Task threads concurrently reads (using select) TASKS table on certain where condition and then does some processing.
Issue is ,sometimes both the threads pick the same task , because of which the task is executed twice.
My question is how do i make both thread not to read the same set of data from the TASKS table
It is just because your code(which is accessing data base)DAO function is not synchronized.Make it synchronized,i think your problem will be solved.
If the TASKS table you mention is a database table then I would use Transaction isolation.
As a suggestion, within a trasaction, set an attribute of the TASK table to some unique identifiable value if not set. Commit the tracaction. If all is OK then the task has be selected by the thread.
I haven't come across this usecase so treat my suggestion with catuion.
I think you need to see some information how does work with any enterprise job scheduler, for example with Quartz
For your use case there is a better tool for the job - and that's messaging. You are persisting items that need to be worked on, and then attempting to synchronise access between workers. There are a number of issues that you would need to resolve in making this work - in general updating a table and selecting from it should not be mixed (it locks), so storing state there doesn't work; neither would synchronization in your Java code, as that wouldn't survive a server restart.
Using the JMS API with a message broker like ActiveMQ, you would publish a message to a queue. This message would contain the details of the task to be executed. The message broker would persist this somewhere (either in its own message store, or a database). Worker threads would then subscribe to the queue on the message broker, and each message would only be handed off to one of them. This is quite a powerful model, as you can have hundreds of message consumers all acting on tasks so it scales nicely. You can also make this as resilient as it needs to be, so tasks can survive both Tomcat and broker restarts.
Whether the database can provide graceful management of this will depend largely on whether it is using strict two-phase locking (S2PL) or multi-version concurrency control (MVCC) techniques to manage concurrency. Under MVCC reads don't block writes, and vice versa, so it is very possible to manage this with relatively simple logic. Under S2PL you would spend too much time blocking for the database to be a good mechanism for managing this, so you would probably want to look at external mechanisms. Of course, an external mechanism can work regardless of the database, it's just not really necessary with MVCC.
Databases using MVCC are PostgreSQL, Oracle, MS SQL Server (in certain configurations), InnoDB (except at the SERIALIZABLE isolation level), and probably many others. (These are the ones I know of off-hand.)
I didn't pick up any clues in the question as to which database product you are using, but if it is PostgreSQL you might want to consider using advisory locks. http://www.postgresql.org/docs/current/interactive/explicit-locking.html#ADVISORY-LOCKS I suspect many of the other products have some similar mechanism.
I think you need have some variable (column) where you keep last modified date of rows. Your threads can read same set of data with same modified date limitation.
Edit:
I did not see "not to read"
In this case you need have another table TaskExecutor (taskId , executorId) , and when some thread runs task you put data to TaskExecutor; and when you start another thread it just checks that task is already executing or not (Select ... from RanTask where taskId = ...).
Нou also need to take care of isolation level for transaсtions.
Suppose that I have a method called doSomething() and I want to use this method in a multithreaded application (each servlet inherits from HttpServlet).I'm wondering if it is possible that a race condition will occur in the following cases:
doSomething() is not staic method and it writes values to a database.
doSomething() is static method but it does not write values to a database.
what I have noticed that many methods in my application may lead to a race condition or dirty read/write. for example , I have a Poll System , and for each voting operation, a certain method will change a single cell value for that poll as the following:
[poll_id | poll_data ]
[1 | {choice_1 : 10, choice_2 : 20}]
will the JSP/Servlets app solve these issues by itself, or I have to solve all that by myself?
Thanks..
It depends on how doSomething() is implemented and what it actually does. I assume writing to the database uses JDBC connections, which are not threadsafe. The preferred way of doing that would be to create ThreadLocal JDBC connections.
As for the second case, it depends on what is going on in the method. If it doesn't access any shared, mutable state then there isn't a problem. If it does, you probably will need to lock appropriately, which may involve adding locks to every other access to those variables.
(Be aware that just marking these methods as synchronized does not fix any concurrency bugs. If doSomething() incremented a value on a shared object, then all accesses to that variable need to be synchronized since i++ is not an atomic operation. If it is something as simple as incrementing a counter, you could use AtomicInteger.incrementAndGet().)
The Servlet API certainly does not magically make concurrency a non-issue for you.
When writing to a database, it depends on the concurrency strategy in your persistence layer. Pessimistic locking, optimistic locking, last-in-wins? There's way more going on when you 'write to a database' that you need to decide how you're going to handle. What is it you want to have happen when two people click the button at the same time?
Making doSomething static doesn't seem to have too much bearing on the issue. What's happening in there is the relevant part. Is it modifying static variables? Then yes, there could be race conditions.
The servlet api will not do anything for you to make your concurrency problems disappear. Things like using the synchronized keyword on your servlets are a bad idea because you are basically forcing your threads to be processed one at a time and it ruins your ability to respond quickly to multiple users.
If you use Spring or EJB3, either one will provide threadlocal database connections and the ability to specify transactions. You should definitely check out one of those.
Case 1, your servlet uses some code that accesses a database. Databases have locking mechanisms that you should exploit. Two important reasons for this: the database itself might be used from other applications that read and write that data, it's not enough for your app to deal with contending with itself. And: your own application may be deployed to a scaled, clustered web container, where multiple copies of your code are executing on separate machines.
So, there are many standard patterns for dealing with locks in databases, you may need to read up on Pessimistic and Optimistic Locking.
The servlet API and JBC connection pooling gives you some helpful guarantees so that you can write your servlet code without using Java synchronisation provided your variables are in method scope, in concept you have
Start transaction (perhaps implicit, perhaps on entry to an ejb)
Get connection to DB ( Gets you a connection from pool, associated with your tran)
read/write/update code
Close connection (actually keeps it for your thread until your transaction commits)
Commit (again maybe implictly)
So your only real issue is dealing with any contentions in the DB. All of the above tends to be done rather more nicely using things such as JPA these days, but under the covers thats more or less what's happening.
Case 2: static method, this presumably implies that you now keep everything in a memory structure. This (barring remote invocation of some sort) impies a single JVM and you managing your own locking. Should your JVM or machine crash I guess you lose your data. If you care about your data then using a DB is probably better.
OR, how about a completely other approach: servlet simply records the "vote" by writing a message to a persistent JMS queue. Have some other processes pick up the votes from the queue and adds them up. You won't give immediate feedback to the voter this way, but you decouple the user's experience from the actual (in similar scenarios) quite complex processing .
I thing that the best solution for your problem is to use something like "synchronized" keyword and wait/notify!
I've been working on this for a few days now, and I've found several solutions but none of them incredibly simple or lightweight. The problem is basically this: We have a cluster of 10 machines, each of which is running the same software on a multithreaded ESB platform. I can deal with concurrency issues between threads on the same machine fairly easily, but what about concurrency on the same data on different machines?
Essentially the software receives requests to feed a customer's data from one business to another via web services. However, the customer may or may not exist yet on the other system. If it does not, we create it via a web service method. So it requires a sort of test-and-set, but I need a semaphore of some sort to lock out the other machines from causing race conditions. I've had situations before where a remote customer was created twice for a single local customer, which isn't really desirable.
Solutions I've toyed with conceptually are:
Using our fault-tolerant shared file system to create "lock" files which will be checked for by each machine depending on the customer
Using a special table in our database, and locking the whole table in order to do a "test-and-set" for a lock record.
Using Terracotta, an open source server software which assists in scaling, but uses a hub-and-spoke model.
Using EHCache for synchronous replication of my in-memory "locks."
I can't imagine that I'm the only person who's ever had this kind of problem. How did you solve it? Did you cook something up in-house or do you have a favorite 3rd-party product?
you might want to consider using Hazelcast distributed locks. Super lite and easy.
java.util.concurrent.locks.Lock lock = Hazelcast.getLock ("mymonitor");
lock.lock ();
try {
// do your stuff
}finally {
lock.unlock();
}
Hazelcast - Distributed Queue, Map, Set, List, Lock
We use Terracotta, so I would like to vote for that.
I've been following Hazelcast and it looks like another promising technology, but can't vote for it since I've not used it, and knowing that it uses a P2P based system at its heard, I really would not trust it for large scaling needs.
But I have also heard of Zookeeper, which came out of Yahoo, and is moving under the Hadoop umbrella. If you're adventurous trying out some new technology this really has lots of promise since it's very lean and mean, focusing on just coordination. I like the vision and promise, though it might be too green still.
http://www.terracotta.org
http://wiki.apache.org/hadoop/ZooKeeper
http://www.hazelcast.com
Terracotta is closer to a "tiered" model - all client applications talk to a Terracotta Server Array (and more importantly for scale they don't talk to one another). The Terracotta Server Array is capable of being clustered for both scale and availability (mirrored, for availability, and striped, for scale).
In any case as you probably know Terracotta gives you the ability to express concurrency across the cluster the same way you do in a single JVM by using POJO synchronized/wait/notify or by using any of the java.util.concurrent primitives such as ReentrantReadWriteLock, CyclicBarrier, AtomicLong, FutureTask and so on.
There are a lot of simple recipes demonstrating the use of these primitives in the Terracotta Cookbook.
As an example, I will post the ReentrantReadWriteLock example (note there is no "Terracotta" version of the lock - you just use normal Java ReentrantReadWriteLock)
import java.util.concurrent.locks.*;
public class Main
{
public static final Main instance = new Main();
private int counter = 0;
private ReentrantReadWriteLock rwl = new ReentrantReadWriteLock(true);
public void read()
{
while (true) {
rwl.readLock().lock();
try {
System.out.println("Counter is " + counter);
} finally {
rwl.readLock().unlock();
}
try { Thread.currentThread().sleep(1000); } catch (InterruptedException ie) { }
}
}
public void write()
{
while (true) {
rwl.writeLock().lock();
try {
counter++;
System.out.println("Incrementing counter. Counter is " + counter);
} finally {
rwl.writeLock().unlock();
}
try { Thread.currentThread().sleep(3000); } catch (InterruptedException ie) { }
}
}
public static void main(String[] args)
{
if (args.length > 0) {
// args --> Writer
instance.write();
} else {
// no args --> Reader
instance.read();
}
}
}
I recommend to use Redisson. It implements over 30 distributed data structures and services including java.util.Lock. Usage example:
Config config = new Config();
config.addAddress("some.server.com:8291");
Redisson redisson = Redisson.create(config);
Lock lock = redisson.getLock("anyLock");
lock.lock();
try {
...
} finally {
lock.unlock();
}
redisson.shutdown();
I was going to advice on using memcached as a very fast, distributed RAM storage for keeping logs; but it seems that EHCache is a similar project but more java-centric.
Either one is the way to go, as long as you're sure to use atomic updates (memcached supports them, don't know about EHCache). It's by far the most scalable solution.
As a related datapoint, Google uses 'Chubby', a fast, RAM-based distributed lock storage as the root of several systems, among them BigTable.
I have done a lot of work with Coherence, which allowed several approaches to implementing a distributed lock. The naive approach was to request to lock the same logical object on all participating nodes. In Coherence terms this was locking a key on a Replicated Cache. This approach doesn't scale that well because the network traffic increases linearly as you add nodes. A smarter way was to use a Distributed Cache, where each node in the cluster is naturally responsible for a portion of the key space, so locking a key in such a cache always involved communication with at most one node. You could roll your own approach based on this idea, or better still, get Coherence. It really is the scalability toolkit of your dreams.
I would add that any half decent multi-node network based locking mechanism would have to be reasonably sophisticated to act correctly in the event of any network failure.
Not sure if I understand the entire context but it sounds like you have 1 single database backing this? Why not make use of the database's locking: if creating the customer is a single INSERT then this statement alone can serve as a lock since the database will reject a second INSERT that would violate one of your constraints (e.g. the fact that the customer name is unique for example).
If the "inserting of a customer" operation is not atomic and is a batch of statements then I would introduce (or use) an initial INSERT that creates some simple basic record identifying your customer (with the necessary UNIQUEness constraints) and then do all the other inserts/updates in the same transaction. Again the database will take care of consistency and any concurrent modifications will result in one of them failing.
I made a simple RMI service with two methods: lock and release. both methods take a key (my data model used UUIDs as pk so that was also the locking key).
RMI is a good solution for this because it's centralized. you can't do this with EJBs (specialially in a cluster as you don't know on which machine your call will land). plus, it's easy.
it worked for me.
If you can set up your load balancing so that requests for a single customer always get mapped to the same server then you can handle this via local synchronization. For example, take your customer ID mod 10 to find which of the 10 nodes to use.
Even if you don't want to do this in the general case your nodes could proxy to each other for this specific type of request.
Assuming your users are uniform enough (i.e. if you have a ton of them) that you don't expect hot spots to pop up where one node gets overloaded, this should still scale pretty well.
You might also consider Cacheonix for distributed locks. Unlike anything else mentioned here Cacheonix support ReadWrite locks with lock escalation from read to write when needed:
ReadWriteLock rwLock = Cacheonix.getInstance().getCluster().getReadWriteLock();
Lock lock = rwLock.getWriteLock();
try {
...
} finally {
lock.unlock();
}
Full disclosure: I am a Cacheonix developer.
Since you are already connecting to a database, before adding another infra piece, take a look at JdbcSemaphore, it is simple to use:
JdbcSemaphore semaphore = new JdbcSemaphore(ds, semName, maxReservations);
boolean acq = semaphore.acquire(acquire, 1, TimeUnit.MINUTES);
if (acq) {
// do stuff
semaphore.release();
} else {
throw new TimeoutException();
}
It is part of spf4j library.
Back in the day, we'd use a specific "lock server" on the network to handle this. Bleh.
Your database server might have resources specifically for doing this kind of thing. MS-SQL Server has application locks usable through the sp_getapplock/sp_releaseapplock procedures.
We have been developing an open source, distributed synchronization framework, currently DistributedReentrantLock and DistributedReentrantReadWrite lock has been implemented, but still are in testing and refactoring phase. In our architecture lock keys are devided in buckets and each node is resonsible for certain number of buckets. So effectively for a successfull lock requests, there is only one network request. We are also using AbstractQueuedSynchronizer class as local lock state, so all the failed lock requests are handled locally, this drastically reduces network trafic.
We are using JGroups (http://jgroups.org) for group communication and Hessian for serialization.
for details, please check out http://code.google.com/p/vitrit/.
Please send me your valuable feedback.
Kamran