Sorry for that I am the newbie for Distributed Lock and Redis, I heard about Redis is a single-threaded server. So My question is Why we need distributed lock management for the Redis? For example ,The second thread(Client B initiated) will not interrupt the previous thread until the operation of the first thread(Client A initiated) is accomplished. Even both threads are working on the same data. I knew I must miss somthing . Please kindly help to correct me. Thanks.
I think that distributed lock is not about the Redis itself (you shouldn't really care whether its single-threaded or not), but rather about your application.
It's clear what a "regular" (not distributed) lock does, but it works with multi-threaded application in the single JVM.
The word "distributed" adds a way to synchronize the access to some resource across many JVMs so that only one JVM will execute a critical section.
Now, as stated in the article you refer, its possible to implement the lock with "SET" command but the fundamental issue with such an implementation is a single point of failure of the Redis itself, that's why they're talking about the Redlock algorithm which supports to acquire a lock based on the state of many independent Redis machines. Note that in any point of this we don't really care whether the Redis itself is single- or multi-threaded.
Related
I am working on a platfor that hosts small Java applications, all of which currently uses a single thread, living inside a Docker engine, consuming data from a Kafka server and logging to a central DB.
Now, I need to put another Java application to this platform. This app at hand uses multithreading relatively heavily, I already tested it inside a Docker container and it works perfectly there, so I'm ready to deploy it on the platform where it would be scaled manually, that is, some human would define the number of containers that would be started, each of them containing an instance of this app.
My Architect has an objection, saying that "In a distributed environment we never use multithreading". So now, I have to refactor my application eliminating any thread related logic from it, making it single threaded. I requested a more detailed reasoning from him, but he yells "If you are not aware of this principle, you have no place near Java".
Is it really a mistake to use a multithreaded Java application in a distributed system - a simple cluster with ten or twenty physical machines, each hosting a number of virtual machines, which then runs Docker containers, with Java applications inside them.
Honestly, I don't see the problem of multithreading inside a container.
Is it really a mistake or somehow "forbidden"?
Thanks.
When you write for example a web application that will run in a Java EE application server, then normally you should not start up your own threads in your web application. The application server will manage threads, and will allocate threads to process incoming requests on the server.
However, there is no hard rule or reason why it is never a good idea to use multi-threading in a distributed environment.
There are advantages to making applications single-threaded: the code will be simpler and you won't have to deal with difficult concurrency issues.
But "in a distributed environment we never use multithreading" is not necessarily always true and "if you are not aware of this principle, you have no place near Java" sounds arrogant and condescending.
I guess he only tells you this as using a single thread eliminates multi threading and data ordering issues.
There is nothing wrong with multithreading though.
Distributed systems usually have tasks that are heavily I/O bound.
If I/O calls are blocking in your system
The only way to achieve concurrency within the process is spawning new threads to do other useful work. (Multi-threading).
The caveat with this approach is that, if they are too many threads
in flight, the operating system will spend too much time context
switching between threads, which is wasteful work.
If I/O calls are Non-Blocking in your system
Then you can avoid the Multi-threading approach and use a single thread to service all your requests. (read about event-loops or Java's Netty Framework or NodeJS)
The upside for single thread approach
The OS does not any wasteful thread context switches.
You will NOT run into any concurrency problems like dead locks or race conditions.
The downside is that
It is often harder to code/think in a non-blocking fashion
You typically end up using more memory in the form of blocking queues.
What? We use RxJava and Spring Reactor pretty heavily in our application and it works pretty fine. You can't work with threads across two JVMs anyway. So just make sure that your logic is working as you expect on a single JVM.
I am trying to understand exactly what types of problems Apache ZooKeeper ("ZK") solves, and perhaps their Recipes page is the best place to start.
First off, I am making the following assumptions:
The ZooKeeper API (available in both Java and C) exposes these 7 simple methods which then allow you to build up your own usage patterns, known as "ZK Recipes"
It is then up to you to use these ZK Recipes to solve problems in distributed programming yourself
Or, instead of building up your own ZK Recipes, you could just use the ones that ship with Apache Curator
So either way, you're using ZK Recipes (again, homegrown or provided by Curator) to solve distributed computing problems
I believe Apache Kafka is an example of this, where Kafka uses ZK to create a distributed Queue (which is one of the listed ZK Recipes). So if my assumptions are correct, ZK exposes those API methods, and the creators of Apache Kafka either used ZK directly or used Curator to implement the "Queue" ZK Recipe.
If any of my above assumptions are wrong, please begin by correcting me! Assuming I'm more or less on track:
Looking at the list of ZK Recipes, I see the following (non-exhaustive):
Barriers
Locks
Leader Election
In order for me to appreciate these recipes and the solutions they present, I first need to appreciate the problem that they solve! I understand what a lock is from basic Java concurrency, but I'm just not seeing the use case for when a "distributed Lock" would ever be necessary. For leading election, all I can think of - as a use case for needing it in the first place - would be if you were building an application that you wanted to ship with a built-in master/slave or primary/secondary capability. Perhaps in that case you would use ZK to implement your own "Leader Election" recipe, or perhaps just use Curator's Leader Latch out of the box. As for Barriers, I don't see how those are any different than Locks. So I ask:
Is my master/slave or primary/secondary problem an accurate use case for ZK's Leader Election recipe?
What would be an example of a distributed Lock? What problem(s) does it solve?
Ditto for Barriers: and what's the difference between Locks and Barriers?
Yes. Your Zk's leader election recipe example is a correct one. In general, if a recipe already exists why rewrite it?
Quoting Zookeeper documentation:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Regarding distributed locks - Let's say you have a distributed system where all configuration are saved on Zookeeper, and more than one entity is responsible for updating a certain configuration - In such a case you would want the configuration updates to be synchronous.
Regarding the barrier, I personally never used them - but with a lock you need to aquire the lock to actually do something on the node, a barrier you wait until it's free but do not necessarily need to set the barrier once it's free.
I have a job processing analytic service working against RDBMS that, due to the need for complex caching and cache update logic needs to be a singleton in a high availability cluster. Jobs are coming as JMS messages (via ActiveMQ). It is part of the application hosted in HA Tomcat cluster with web front end.
The problem is, the service itself needs to be able to recover within seconds if a node where it is running fails. Failure could mean system down or just a slow CPU - i.e. if node recovers after CPU delay, but the processing is handed over, it cannot continue.
From experience, what would be the most suitable solution here:
database-based locks and lock checking before each job starts (I could not easily come up with bullet-proof solution here - any recommendations?)
some kind of Paxos algorithm? Do you know of any slim framework for that purpose as the algorithm itself takes time to get right and then QA?
anything else?
I don't mind if failure recovery is slow, but I would want to minimize an overhead for each job.
Some additional background: job does not involve anything more than reading data from the database, massaging it with various algorithms (somewhat resembling finding shortest routes) and putting back optimal solutions for different actors to move on. Actors interact with real world and put back some feedback, based on which consequent steps are optimized by the same job processor.
Solution Using Hazelcast
Hazelcast locking method proposed by Tomasz works. You need to read documentation carefully, use time leased locks and ensure monitoring of your singleton to renew leases. One thing to keep in mind is that Hazelcast was written to work in large clusters - as such its start up time is relatively slow, 1 to 5 seconds even for two nodes. But after that operations are qute performant and obtainng the lock takes milliseconds. Normally it all does not matter, but failure/recovery cycle takes time and it should be treated as exceptional situation.
There are limits to this solution being buletproof. If the cluster is split (network disruption between nodes) but each node is alive and has access to the database, there is no way of knowing deterministically how to proceed. Ultimately, you need to think about a contingency plan here. In real life this scenario is very unlikely for a typical failover HA setup.
At the end of the day, before resorting to a solution with distributed locking, think hard about making your process not-so-singleton. It might still be hard to run certain things in parallel, but it might not be so hard to ensure the cache is not stale or find other ways to prevent database corruption. In my case, there is a database transaction counter working like optimisitic lock. Code reads it before making all the decisions and update-where's it in both, db and cache in the transaction where the result is stored. In case of discrepancy cache is purged and operation repeated. It makes two nodes working in parallel impossibly slow, but it prevents data corruption. By storing additional data with the transaction counter you might be able to optimize cache refresh strategies and slowly move towards parallel processing.
Conclusion.
This is how I would proceed about such a request next time.
Try making your singletons survive working in parallel on different nodes
Try again, maybe there is a way to orchestrate them
Check if it is possible to use HASingleton or similar technology to
avoid boilerplate
Implement the Hazelcast solution as outlined above
It makes no sense to post the code here as the most time consuming part is to test and verify all failure scenarios and contingency plans. There is almost no boilerplate, the code itself will always be solution specific. It is possible to come up with well working PoC covering all the bases within couple of days.
I'm trying to accomplish something that in terms of concept is very simple to understand. I want to synchronize a block of java code between different machines. There are two instances of a programa running in different machines that cannot run at the same time.
I've heard of zookeeper, jgroups and akka too, but while reading the documentation it seemed to me a bit overkill for what I'm trying to do. Does anyone have any idea if there's anything more straight forward to use?
Thanks in advance,
Rui
I think Hazelcast's Distributed Lock ( http://docs.hazelcast.org/docs/3.6/manual/html-single/index.html#lock ) may be helpful. Hazelcast is relatively lightweight so should hopefully not be overkill.
If all the technologies you mentioned (also take a look at Terracotta) are too sophisticated for your needs, maybe simple database locking?
SELECT FOR UPDATE statement will lock given database record, making other clients running this query to block. Simple, yet safe and reliable.
A very very basic solution would be using RMI.
Decide to use one machine as master which has a method which uses a mutex lock to allow only one mthod caller passing.
This special method you have to call via RMI from all other slave instances before you run your special java code block.
this is a bit related to this question.
I'm using make to extract some information concerning some C programs. I'm wrapping the compilation using a bash script that runs my java program and then gcc. Basically, i'm doing:
make CC=~/my_script.sh
I would like to use several jobs (-j option with make). It's running several processes according to the dependency rules.
If i understood well, I would have as many instances of the jvm as jobs, right ?
The thing is that i'm using sqlite-jdb to collect some info. So the problem is how to avoid several processes trying to modify the db at the same time ?
It seems that the sqlite lock is jvm-dependant (i mean one lock can be "see" only inside the locking jvm), and that this is the same for RandomAccessFile.lock().
Do you have any idea how to do that ? (creating a tmp file and then looking if it exists or not seems to be one possibility but may be expensive. A locking table in the dB ? )
thanks
java.nio.channels.FileLock allows OS-level cross-process file locking.
However, using make to start a bash scripts that runs several JVMs in parallel before calling gcc sounds altogether too Rube-Goldbergian and brittle to me.
there are several solutions for this.
if your lock should be within the same machine, you can use a server socket to implement it (The process that manages to bind to the port first owns the lock, other processes waits for the port to become available).
if you need a lock that span across multiple machines you can use a memcached lock. this will require a memcached server running. I can paste some code if you are interested in this solution.
you can get Java library to connect to memcached here.
You may try Terracotta for sharing objects between various JVM instances. It may appear as a too heavy solution for your needs, but at least worth considering.