I am trying to understand exactly what types of problems Apache ZooKeeper ("ZK") solves, and perhaps their Recipes page is the best place to start.
First off, I am making the following assumptions:
The ZooKeeper API (available in both Java and C) exposes these 7 simple methods which then allow you to build up your own usage patterns, known as "ZK Recipes"
It is then up to you to use these ZK Recipes to solve problems in distributed programming yourself
Or, instead of building up your own ZK Recipes, you could just use the ones that ship with Apache Curator
So either way, you're using ZK Recipes (again, homegrown or provided by Curator) to solve distributed computing problems
I believe Apache Kafka is an example of this, where Kafka uses ZK to create a distributed Queue (which is one of the listed ZK Recipes). So if my assumptions are correct, ZK exposes those API methods, and the creators of Apache Kafka either used ZK directly or used Curator to implement the "Queue" ZK Recipe.
If any of my above assumptions are wrong, please begin by correcting me! Assuming I'm more or less on track:
Looking at the list of ZK Recipes, I see the following (non-exhaustive):
Barriers
Locks
Leader Election
In order for me to appreciate these recipes and the solutions they present, I first need to appreciate the problem that they solve! I understand what a lock is from basic Java concurrency, but I'm just not seeing the use case for when a "distributed Lock" would ever be necessary. For leading election, all I can think of - as a use case for needing it in the first place - would be if you were building an application that you wanted to ship with a built-in master/slave or primary/secondary capability. Perhaps in that case you would use ZK to implement your own "Leader Election" recipe, or perhaps just use Curator's Leader Latch out of the box. As for Barriers, I don't see how those are any different than Locks. So I ask:
Is my master/slave or primary/secondary problem an accurate use case for ZK's Leader Election recipe?
What would be an example of a distributed Lock? What problem(s) does it solve?
Ditto for Barriers: and what's the difference between Locks and Barriers?
Yes. Your Zk's leader election recipe example is a correct one. In general, if a recipe already exists why rewrite it?
Quoting Zookeeper documentation:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Regarding distributed locks - Let's say you have a distributed system where all configuration are saved on Zookeeper, and more than one entity is responsible for updating a certain configuration - In such a case you would want the configuration updates to be synchronous.
Regarding the barrier, I personally never used them - but with a lock you need to aquire the lock to actually do something on the node, a barrier you wait until it's free but do not necessarily need to set the barrier once it's free.
Related
My application uses a custom key-value store as data persistence layer. This key-value store is developed in-house and has some APIs to work with, however, it does not provide anything around transaction management or locking (especially distributed locking).
Now, we as the user of this key-value store need to develop such a locking/transaction management system. Could someone help showing how such a distributed locking can be implemented? Is Apache Zookeeper is worth looking at?
We use Java 7.
Thanks,
NN
Plenty of options out there, many of them will even tell you how to do it:
Zookeeper which you mentioned, they even say how to do it in their
cookbook. You can check the Zookeeper whitepaper they say a little bit about implementing distributed locks.
Hazelcast which is a in memory distributed data grid has a
distributed lock implementation
Redis has one, too!
Anything that's distributed and in-memory could be a good candidate (and it actually works) is a good candidate. Personally at work we are using ZK (to be more precise Curator to make it even easier) and it works well. Some of the libraries we are using do it with Hazelcast and it doesn't seem to be any worse
I am trying to reconstruct a flow of information from multiple parts handled by different Java processes. Please note that i don't generate the flows, i just read some information about them.
I've tried using MySQL (MyISAM/InnoDB tables) with INSERT ON DUPLICATE KEY UPDATE using an id for each flow. I've also tried storing all the pieces of information and running a query at the end to get the full information. Neither of these approaches yielded the performance needed.
I'm looking for a solution that will allow me to have a set of shared objects between multiple Java processes. The objects should be persistent between runs and fast to lookup/update concurrently (>100k lookups/updates per second).
I've thought of a few solutions including:
NoSQL: something like MongoDB, HBase etc.
a caching solution like EhCache, Memcached etc.
The problem is i don't have any experience with any of these solutions. So, what would you recommend that fits the following criteria:
very fast on a single system. Most of the applications i mentioned were built for distributed systems, but it's not the case here.
easy to learn/use (i want to be able to prototype it in a day)
mature technology
free to use even for commercial purposes
preferably open-source
You could try a seperate java process that co-ordinates between the others. This process would hold the information to pass over to the main processes. You could wire them up with RMI.
If you want to do only exchange of objects withing java applications, you could also looki into tuple spaces. There are specific implementations of spaces for java, JavaSpaces, which should be able to do what you need. Not sure if they can keep up with the performance though. Also I’m not sure how widely this technology is still being used, since it only supports Java and isn’t as flexible as NoSQL stores would be these days.
Wikipedia has a more detailed description and list of different implementations, many of which are open source.
The other option is to go with Redis, you have notifications there and it can for sure scale to the requirements you are looking for.
The old (legacy?) solution is JavaSpaces. However, from an software architects point of view I would say distributed caches are the replacements for that nowadays. Especially take a look at hazelcast and infinispan.
From the performance viewpoint I am not happy with the performance of the "big" distributed caching solutions, when only a single in-memory cache is needed, see my writeup on the cache2k benchmarks page (hazelcast needs to be added here).
Anyways, please clarify your problem statement first, because your question falls into the XyProblem category. You are not describing the actual problem, and your question just boils down to "fast reliable distributed objects" solution. What kind of data comes in? What is the rate? Who is it accessed? What consistency guarantees need to be met, considering the fact that writing and reading is in parallel?
By the term "flow of information" it sounds more like a complex event processing problem to me.
I'm developing high-scalable application, so I decided to use Hazelcast for it. I have one frontend server, which puts messages for nodes. Every node in cluster change it's workload in background thread in distributed map, so, frontend server choose queue (every node has it's own message queue) to put message in. My question is: Is Hazelcast suitable for such design (we need workload distribution and load balancing) or may be some alternatives? I like Hazelcast for it's simplicity and nice design.
Hazelcast is great, it's very lightweight and easy to use, however, it's still in development and there are a few issues when using it.
If you look here: http://code.google.com/p/hazelcast/issues/list you can see that there are some bugs with the queue data structure while using transactions. Overall, it's provides what it advertises and basically gives a distributed cache for free.
I have first hand experience with hazelcast. The version we went to production with is version 1.9.4. We recently upgraded to 2.2, and now 2.3 is the latest. I am quite pleased with it. What you are describing is a pretty good use case for hazelcast. I had a similar use case where each node has its own queue and messages are pushed to the appropriate queue based on which node the client was connected to. It worked great and the business loved it.
I have a task to build a prototype for a massively scalable distributed shared memory (DSM) app. The prototype would only serve as a proof-of-concept, but I want to spend my time most effectively by picking the components which would be used in the real solution later on.
The aim of this solution is to take data input from an external source, churn it and make the result available for a number of frontends. Those "frontends" would just take the data from the cache and serve it without extra processing. The amount of frontend hits on this data can literally be millions per second.
The data itself is very volatile; it can (and does) change quite rapidly. However the frontends should see "old" data until the newest has been processed and cached. The processing and writing is done by a single (redundant) node while other nodes only read the data. In other words: no read-through behaviour.
I was looking into solutions like memcached however this particular one doesn't fulfil all our requirements which are listed below:
The solution must at least have Java client API which is reasonably well maintained as the rest of app is written in Java and we are seasoned Java developers;
The solution must be totally elastic: it should be possible to add new nodes without restarting other nodes in the cluster;
The solution must be able to handle failover. Yes, I realize this means some overhead, but the overall served data size isn't big (1G max) so this shouldn't be a problem. By "failover" I mean seamless execution without hardcoding/changing server IP address(es) like in memcached clients when a node goes down;
Ideally it should be possible to specify the degree of data overlapping (e.g. how many copies of the same data should be stored in the DSM cluster);
There is no need to permanently store all the data but there might be a need of post-processing of some of the data (e.g. serialization to the DB).
Price. Obviously we prefer free/open source but we're happy to pay a reasonable amount if a solution is worth it. In any way, paid 24hr/day support contract is a must.
The whole thing has to be hosted in our data centers so SaaS offerings like Amazon SimpleDB are out of scope. We would only consider this if no other options would be available.
Ideally the solution would be strictly consistent (as in CAP); however, eventual consistence can be considered as an option.
Thanks in advance for any ideas.
Have a look at Hazelcast. It is pure Java, open source (Apache license) highly scalable in-memory data grid product. It does offer 7X24 support. And it does solve all of your problems I tried to explain each of them below:
It has a native Java Client.
It is 100% dynamic. Add and remove nodes dynamically. No need to change anything.
Again everything is dynamic.
You can configure number of backup nodes.
Hazelcast support persistency.
Everything that Hazelcast offers is free(open source) and it does offer enterprise level support.
Hazelcast is single jar file. super easy to use. Just add jar to your classpath. Have a look at screen cast in main page.
Hazelcast is strictly consistent. You can never read stale data.
I suggest you to use Redisson - Redis based In-memory Data Grid for Java. Implements (BitSet, BloomFilter, Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, BlockingQueue, BlockingDeque, ReadWriteLock, Semaphore, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, RemoteService, ExecutorService, LiveObjectService, SchedulerService) on top of Redis server! It supports master/slave, sentinel and cluster server modes. Automatic cluster/sentinel servers topology discovery supported also. This lib is free and open-source.
Perfectly works in cloud thanks to AWS Elasticache support
Depending of what you prefer, i would surely follow the others by suggesting Hazelcast if you're towards AP from the CAP Theorem but if you need CP, i would choose Redis
Have a look at Terracotta's JVM clustering, it's OpenSource ;)
It has no API while it works efficent at JVM level, when you store the value in a replicated object it is sent to all other nodes.
Even locking and all those things work transparent and without adding any new code.
You may want to checkout Java-specific solutions like Coherence: http://www.oracle.com/global/ru/products/middleware/coherence/index.html
However, I consider such solutions to be too complex and prefer to use solutions like memcached. Big disadvantage of memcached for your purpose is lack of record lock it seems and there is no built in way to replicate data for failover. That is why I would look into the key-value data stores. Many of them would satisfy your need completely.
Here is a list of key-value data stores that may help you with your task:
http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores
Just pick one that you fill comfortable with.
I am doing a similar project, but instead targeting the .NET platform. Apart from the already mentioned solutions, I think you should take a look at ScaleOut StateServer and Alachisoft NCache. I am afraid neither of these alternatives are cheap, but they are a safer bet than open source for commercial solutions according to my judgement.
Both provide Java client APIs, even though I have only played around with the .NET APIs.
StateServer features self-discovery of new cache nodes, and NCache has a management console where new cache nodes can be added.
Both should be able to handle failovers seamlessly.
StateServer can have 1 or 2 passive copies of the data. NCache features more caching topologies to choose between.
If you mean write-through/write-behind to a database that is available in both.
I have no idea how many cache servers you plan to use, but here are the full price specs:
ScaleOut StateServer
Alachisoft NCache
Both are installed and configured locally on your server and they both have GUI Management.
I am not sure exactly what strictly consistent involves, so I'll leave that for you to investigate..
Overall, StateServer is the best option if you want to skip configuring every little detail in the cache cluster, while NCache features very many features and caching topologies to choose from.
Depending on the behaviour of data towards the clients (if the data is read many times from the same client) it might be a good idea to mix local caching on the clients with the distributed caching in the cluster (available for both NCache and StateServer), just a thought.
The specified use case seems to fit into Netflix's Hollow. This is a read-only replicated cache with a single producer and multiple consumers.
Have you tought about using a standard messaging solution like rabbitmq ?
RabbitMQ is an open source implementation of the AMQP protocol.
Your application seems more or less like a Publish/subscribe system.
The Publisher node is the one that does the processing and puts messages (processed data) in a queue in the servers.
Subscribers can get messages from the server in various ways. AMQP decouples the producer and the consumer of messages and is very flexible in how you can combine the two sides.
I am looking for a library (or a combination of libraries) to build a java distributed system, made of several applications exchanging data through several pairwise connections (no mapreduce). For the moment I did an expolration of existing libraries and I could only discard what I'v found. Here are my requirements:
Easy discovery of systems at runtime (possibly through a central server/directory)
Lightweight and low latency messages (no CORBA, RMI, SOAP,. etc.)
Decentralized communications (no LINDA like)
Easy enough to use and learn (no JXTA)
Compatible with GPL license (so GPL, BSD, etc.)
Do you have any suggestion ? Thanks in advance
Are you familiar with JGroups? You could use it to design your own architecture. They provide easy-to-use multicast abstraction.
I'm a big fan of JGroups, but I recently discovered hazelcast and will probably give it a try. It might be what you're looking for.
You might want to take a peek at Terracotta ( http://www.terracotta.org/ )
You could take a look at Jade if you like multi-agents paradigm http://jade.tilab.com/
I think Apache River (formerly Jini) should at least be mentioned. It never received too much attention, probably also because it had (don't know if it still has) a rather steep learning curve. Anyhow, it is under active development:
http://river.apache.org/
JBoss, ok, ok, it is not a framework but they have a number of projects that sound just like what you want.
You may use Redisson - distributed and scalable Java data structures (BitSet, BloomFilter, Set, SortedSet, Map, ConcurrentMap, List, Queue, Deque, BlockingQueue, BlockingDeque, ReadWriteLock, Semaphore, Lock, AtomicLong, CountDownLatch, Publish / Subscribe, RemoteService, ExecutorService, LiveObjectService, ScheduledExecutorService) on top of high performance Redis server.