Let's assume we have 3 geographically distributed data centers A,B,C. In each of these, a Cassandra cluster is up and running. Now assume DC A can no longer gossip with B and C.
Writes to A with LOCAL_QUORUM, would still be satisfied - but they would no longer be propagated to B and C; and vice-versa.
This situation could have some very disastrous consequences...
What I'm looking for are some tips on how to rapidly ascertain that DC A has become 'isolated' from the other data centers (using the Native Java driver).
I remember reading about push notifications, but I seem to recall they referred only to the status of the local cluster. Does anybody have any ideas? Thanks.
First thing to note is that in the event that A can no longer connect to B and C, Hints will be stored and delivered upon the restoration of the network connection. So for outages that do not last for a long period of time there is already a safety mechanism and you don't need to do anything.
For longer outages it has been best practice to use the repair command following such an outage to synchronize the replicas.
That said, if you are looking for way to determine when inter DC communication has been disrupted you have several options.
1) Use a tool like Datastax Opscenter to monitor your cluster state, this tool will automatically discover when these sorts of events happen and log them. I also believe you are able to set up triggered events but i'm not an expert in how Opscenter works.
2) Use the Java driver's public Cluster register(Host.StateListener listener) to register a function to be called on node down events, you can then determine when entire DC's go down.
3) Track via JMX on each of the DCs the current state of gossip, this will allow you to see what each Datacenter thinks about the current availability of all the machines. You could do this directly or via nodetool status.
#RussS .. I dont think point (2) works when all three host are not reachable ..
For example ..I Implemented state listener and i am poining to my cluster from my local machine .. I can see that listener gets invoked when nodes go up/down .. But i dont see this listener being invoked when I unplug my ether
Related
I have a project which I watch several nodes in several different threads. Now, I noticed that when I watch a node, and it changed, and an event is raised, the watch on a certain node (called A for example) blocks all the other watchers. So only after the watcher on A is finished , the other watcher will return to watch the nodes changes. Meaning, if a node is changed (called B for example) while its watcher is blocked, only after the watcher on A is finished, the watcher on node B will raise the event.
This problem causes the application be slower.
So, in order to fix this, I wanted to use a different client connnection for each thread, (using curator), but I have read that one connection should be enough, and if I need more than one, there is something worng with my implementation.
1) I dont understand what is the problem with multiple connection to zookeeper server
2) Is there another solution for my problem?
Edit - more specific about the problem
I have a master which gets requests from clients ( and each clients can save files on my server and we do some process on this file, it is more complex than it sounds, i wont elaborate it), and the master creates a node in /tasks/ for a worker to process the file (without its data of course, the data is in a db). When the worker watches his node, he processes the file, and when he finishes, he creates a node in /status (which has all the files that their process were finished).
The master watches the node /status , and when something was changed , it gets the children, and creates a thread (in order to make all faster, because zookeeper watchers and callbacks are single threaded ), which will release those files (remove some meta from db, return a response to client, remove some variable etc.).
That is one of the main flows, but I have another important parts of the code which listen on nodes, and process their children when there are changes.
Because this thing is in a thread, I created a list of the nodes that were finished already, so I wont do the final process more than once, but it was more complex than that, and that solution caused other problems, some concurrency bugs.
So as i asked
1) What is the problem with multiple connection , for each important flow, so i wont have to create threades inside watches and callback?
2) Is there another solution i can use here?
It's not well documented, but ZooKeeper has a single thread for handling watchers and async callbacks. We wrote a tech note for Curator about it. https://cwiki.apache.org/confluence/display/CURATOR/TN1
I have a cluster with 3 nodes (in different machines) and I have a "business logic" that use a distributed lock at startup.
Sometimes when there is more latency every node acquires the exclusive lock with success because the cluster isn't already "startup" so each node does not yet see the other.
Subsequently the nodes see each other and the cluster is correctly configured with 3 nodes. I know there is a "MemberShipListener" to capture the event "Member added" so I could execute again the "business logic", but I would to know if there is a method to ensure when the cluster startup is properly finished in order to wait to execute the "business logic" until the cluster is on.
I tried to use hazelcast.initial.wait.seconds but configure the right seconds isn't deterministic and I don't know if this also delay the member join operations.
Afaik, there is no such thing in Hazelcast. As the cluster is dynamic, a node can go and leave at any time, so the cluster is never "complete" or not.
You can, however :
Configure an initial wait, like you described, in order to help with initial latencies
use hazelcast.initial.min.cluster.size to define the minimum number of members hazelcast is waiting for at start
Define a minimal quorum : the minimal number of nodes for the cluster to be considered as useable/healthy (see cluster quorum)
Use the PartitionService to check is the cluster is safe, or if there are pending migrations
I have a jvm process that wakes a thread every X minutes.
If a condition is true -> it starts a job (JobA).
Another jvm process does almost the same but if the condition is true -
it throws a message to a message broker which triggers the job in another server (JobB).
Now, to avoid SPOF problem I want to add another instance of this machine in my cloud.
But than I want ensure I run a single instance of a JobA each time.
What are my options?
There are a number of patterns to solve this common problem. You need to choose based on your exact situation and depending on which factor has more weight in your case (performance, correctness, fail-tolerance, misfires allowed or not, etc). The two solution-groups are:
The "Quartz" way: you can use a JDBCStore from the Quartz library which (partially) was designed for this very reason. It allows multiple nodes to communicate, and share state and workload between each other. This solution gives you a probably perfect solution at the cost of some extra coding and setting up a shared DB (9 tables I think) between the nodes.
Alternatively your nodes can take care of the distribution itself: locking on a resource (single record in a DB for example) can be enough to decide who is in charge for that iteration of the execution. Sharing previous states however will require a bit more work.
At our company we have a server which is distributed into few instances. Server handles users requests. Requests from different users can be processed in parallel. Requests from same users should be executed strongly sequentionally. But they can arrive to different instances due to balancing. Currently we use Redis-based distributed locks but this is error-prone and requires more work around concurrency than business logic.
What I want is something like this (more like a concept):
Distinct queue for each user
Queue is named after user id
Each requests identified by request id
Imagine two requests from the same user arriving at two different instances concurrently:
Each instance put their request id into this user queue.
Additionaly, they both store their request ids locally.
Then some broker takes request id from the top of "some_user_queue" and moves it into "some_user_queue_processing"
Both instances listen for "some_user_queue_processing". They peek into it and see if this is request id they stored locally. If yes, then do processing. If not, then ignore and wait.
When work is done server deletes this id from "some_user_queue_processing".
Then step 3 again.
And all of this happens concurrently for a lot (thousands of them) of different users (and their queues).
Now, I know this sounds a lot like actors, but:
We need solution requiring as small changes as possible to make fast transition from locks. Akka will force us to rewrite almost everything from scratch.
We need production ready solution. Quasar sounds good, but is not production ready yet (more correctly, their Galaxy cluster).
Tops at my work are very conservative, they simply don't want another dependency which we'll need to support. But we already use Redis (for distributed locks), so I thought maybe it could help with this too.
Thanks
The best solution that matches the description of your problem is Redis Cluster.
Basically, the cluster solves your concurrency problem, in the following way:
Two (or more) requests from the same user, will always go to the same instance, assuming that you use the user-id as a key and the request as a value. The value must be actually a list of requests. When you receive one, you will append it to that list. In other words, that is your queue of requests (a single one for every user).
That matching is being possible by the design of the cluster implementation. It is based on a range of hash-slots spread over all the instances.
When a set command is executed, the cluster performs a hashing operation, which results in a value (the hash-slot that we are going to write on), which is located on a specific instance. The cluster finds the instance that contains the right range, and then performs the writing procedure.
Also, when a get is performed, the cluster does the same procedure: it finds the instance that contains the key, and then it gets the value.
The transition from locks is very easy to perform because you only need to have the instances ready (with the cluster-enabled directive set on "yes") and then to run the cluster-create command from redis-trib.rb script.
I've worked last summer with the cluster in a production environment and it behaved very well.
I'm a game server developer. I'm using hazelcast for my game server (SmartfoxServer 2x) to avoid request db directly.
But I wonder how to use hazelcast to get the best performance:
Always create new hazelcast client instance when access cache and then shutdown it. Or
Create a hazelcast client instance pool, and reuse. Never shutdown until application end. Or
Create only one hazelcast client instance, never shutdown until application end. Or
Make my realtime server as a hazelcast member.
What's the right way? My system serve for about 5000 CCU.
My game is a kind of card game - turn based. Each game occurs in about 2 minutes, with maximum 4 players. When It end, I have to log every transactions (money change), and new money value for user. With 5000 ccu, in worst case, at the same time there're (5000/4) * (4 + 4) = 10000 entries have to be logged. It cannot be done with mysql queries directly (slow), but with hazelcast, it's possible, right?
But I'm a newbie in Hazelcast technique, so I don't know what is the right way to solve my issue.
Thank in advance,
The best answer is 3 or 4. While I was working at the gaming industry I used Hazelcast not as a cache but to distribute / cluster the gameservers itself. Therefor my gameservers itself were part of the cluster. If you want to have a dedicated db-cache cluster just use one client, start it together with the gameserver and keep it alive while the gameserver itself is alive. It is pooled internally and the client is smart enough to route requests to the correct cluster node in almost all cases. If you want to use it for more than just caching I would go for making the gameserver a node of the cluster itself. Remember that every node holds a part of the data, that might not be what you want for a gameserver to act as a db-cache.