I'm a game server developer. I'm using hazelcast for my game server (SmartfoxServer 2x) to avoid request db directly.
But I wonder how to use hazelcast to get the best performance:
Always create new hazelcast client instance when access cache and then shutdown it. Or
Create a hazelcast client instance pool, and reuse. Never shutdown until application end. Or
Create only one hazelcast client instance, never shutdown until application end. Or
Make my realtime server as a hazelcast member.
What's the right way? My system serve for about 5000 CCU.
My game is a kind of card game - turn based. Each game occurs in about 2 minutes, with maximum 4 players. When It end, I have to log every transactions (money change), and new money value for user. With 5000 ccu, in worst case, at the same time there're (5000/4) * (4 + 4) = 10000 entries have to be logged. It cannot be done with mysql queries directly (slow), but with hazelcast, it's possible, right?
But I'm a newbie in Hazelcast technique, so I don't know what is the right way to solve my issue.
Thank in advance,
The best answer is 3 or 4. While I was working at the gaming industry I used Hazelcast not as a cache but to distribute / cluster the gameservers itself. Therefor my gameservers itself were part of the cluster. If you want to have a dedicated db-cache cluster just use one client, start it together with the gameserver and keep it alive while the gameserver itself is alive. It is pooled internally and the client is smart enough to route requests to the correct cluster node in almost all cases. If you want to use it for more than just caching I would go for making the gameserver a node of the cluster itself. Remember that every node holds a part of the data, that might not be what you want for a gameserver to act as a db-cache.
Related
I am new to redis, and redisson but kind of up to date with what is available now.
Mostly from here: https://github.com/redisson/redisson/wiki/9.-distributed-services#91-remote-service
The case here involves a worker, on only one server out of say many. The worker gets to images which can be downloaded later on. They can be pushed to an executor to download later, however, that is not persistable and so we will loose out.
Redis offers executorservice. But I was wondering, does all redis nodes share or ship in to peform the work by default? Is there a way to control that only one gets to do the work? The stuff in the runnable / callable that is being accessed, I am guessing there has to be restrictions on what could be used since it is a closure with access to environment? No access?
Redis also offers something called distributed remote services. How are they different from an executorservice in this regard?
Another option is to push these to reddis list / queue / dequeu, and work of the "messages" albeit the executor service I think would allow me to have all that logic in the same place.
What is the best approach?
What are the rules for objects inside the closure supplied in a runnable / callable ? Fully serialiazeble everything ?
How do I manage if a worker is working, and suddenly dies (nuclear). Can I ensure that someone else gets to work?
At our company we have a server which is distributed into few instances. Server handles users requests. Requests from different users can be processed in parallel. Requests from same users should be executed strongly sequentionally. But they can arrive to different instances due to balancing. Currently we use Redis-based distributed locks but this is error-prone and requires more work around concurrency than business logic.
What I want is something like this (more like a concept):
Distinct queue for each user
Queue is named after user id
Each requests identified by request id
Imagine two requests from the same user arriving at two different instances concurrently:
Each instance put their request id into this user queue.
Additionaly, they both store their request ids locally.
Then some broker takes request id from the top of "some_user_queue" and moves it into "some_user_queue_processing"
Both instances listen for "some_user_queue_processing". They peek into it and see if this is request id they stored locally. If yes, then do processing. If not, then ignore and wait.
When work is done server deletes this id from "some_user_queue_processing".
Then step 3 again.
And all of this happens concurrently for a lot (thousands of them) of different users (and their queues).
Now, I know this sounds a lot like actors, but:
We need solution requiring as small changes as possible to make fast transition from locks. Akka will force us to rewrite almost everything from scratch.
We need production ready solution. Quasar sounds good, but is not production ready yet (more correctly, their Galaxy cluster).
Tops at my work are very conservative, they simply don't want another dependency which we'll need to support. But we already use Redis (for distributed locks), so I thought maybe it could help with this too.
Thanks
The best solution that matches the description of your problem is Redis Cluster.
Basically, the cluster solves your concurrency problem, in the following way:
Two (or more) requests from the same user, will always go to the same instance, assuming that you use the user-id as a key and the request as a value. The value must be actually a list of requests. When you receive one, you will append it to that list. In other words, that is your queue of requests (a single one for every user).
That matching is being possible by the design of the cluster implementation. It is based on a range of hash-slots spread over all the instances.
When a set command is executed, the cluster performs a hashing operation, which results in a value (the hash-slot that we are going to write on), which is located on a specific instance. The cluster finds the instance that contains the right range, and then performs the writing procedure.
Also, when a get is performed, the cluster does the same procedure: it finds the instance that contains the key, and then it gets the value.
The transition from locks is very easy to perform because you only need to have the instances ready (with the cluster-enabled directive set on "yes") and then to run the cluster-create command from redis-trib.rb script.
I've worked last summer with the cluster in a production environment and it behaved very well.
I have a web service for which I need to limit the number of transactions a client can perform. A transaction is hitting the URL with correct parameters. Every client will have different number of transactions it can perform per second. Client will be identified based on the IP address or a parameter in the URL.
The maximum TPS a client can perform will be kept in database or any other configurable manner. I understand that it would be possible to write servlet filter to do this. The filter would calculate requests per second and make database connection to get maximum TPS of client and reject the request when TPS reached as it will further slow down the application response. But that will not be helpful during a DOS attack. Is there a better way?
I had to do the same thing. This is how I did it.
1) I had a data model for tracking an IP's requests. It mainly tracked the rate of requests by using some math that allowed me to add a new request and the new rate of requests for that IP would quickly be recalculated. Lets call this class IpRequestRate.
2) For each unique IP that made a request an instance of IpRequestRate was instantiated. Only one instance was required per IP. They were put into a HashMap for fast retrieval. If a new IP came in, then a new instance of IpRequestRate was created for it.
3) When a request came in, if there was already an instance of IpRequestRate in the HashMap, then I would add the new request to that instance and get the new rate. If the rate was above a certain threshold, then the request would not be processed.
4) If the requester accidentally went above that threshold, then the rate would quickly dip below the threshold. But if it was a real DOS, or in my case too many attempts to access an account (due to hackers), then it would take much longer for their rate to dip below the threshold; which is what I wanted.
5) I do not recall if I had a cleanup thread to remove old IP's but that's something that would be needed at some point. You can use EhCache as your HashMap so it could do this for you automatically.
6) It worked really well and I thought about open sourcing it. But it was really simple and easily reproducible. You just have to get that math down right. The math for getting the rate is easy to get it accurate, but a little tricky if you want it to be fast, so that not a lot fo CPU's are spent calculating the new rate when a new request is added to the IpRequestRate.
Does that answer your question or would you need more info on how to setup the Filter in your server?
Edit: WRT DOS, during a DOS attack we want to waste as little resources as possible. If it all possible DOS detection should be done in a load balancer or reverse proxy or gateway or firewall.
If we want to do per IP max transmission rate, which is stored in a database then I would just cache the max transmission rates. This can be done without doing DB lookup for a request. I would instead load the table into a HashMap.
1) At start of application, say in the init() method, I would load the table into a HashMap that maps IP to maxTransmissionRate.
2) When request comes in, try to get the maxTransmissionRate from the HashMap. If its not there then use a default maxTransmissionRate.
3) During the init(), kickoff a ScheduleExecutorService to update the HashMap at some desired interval, to keep the HashMap fresh. Here is the link to ScheduleExecutorService, its not that hard. http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
4) Which HashMap implementation should we use? If we use a regular HashMap then we will have problems when it gets updated by the ScheduledExecutorService. We can use a synchronized HashMap, but this locks the HashMap and hurts performance during concurrent requests. So I would go with ConcurrentHashMap, which was designed with speed and multithreaded environment. You can safely update a ConcurrentHashMap on separate thread without worry.
If you apply this technique then its still a viable solution for DOS prevention and while supporting per client maxTransmissionRate.
I have a List which contains a lot of objects.
The problem is that i have to process these objects (process includes cloning, deep copy, and making DB calls, running business logic etc etc.
Doing this in a normal fashion, first come first serve is really time consuming and in a web application , this generally results in transaction timeouts at the server side (as this processing is anync from client perspective).
How do i process those objects so as to take minimal time and not overload the DB.
I'm using java 7 on server environment.
I'm already using a messaging solution , rabbitmq, which gets me the item and its quantity. problem occurs when i try to deep copy items to mimic real items (business logic every item should be uniquely processed) and save them to DB.
After some discussions, the viable solution is using a ABQ (array blocking queues) which is processed by a pool of threads.
Following are the thought out benefits:
1) we wont have to manage the 3rd party queues created e.g. rabbitmq
2) At any point in time the blocking queue wont have all the items to be processed as the consumer threads will be simultaneously processing them, so it will leave lesser memory footprint.
#cody123 i'm using spring batch for retry mechanisms in this case.
After another round of profiling i found that the bottle neck was the DB connection pool having low number of max connections.
I deduced this by running the same transaction without db thread pool and it went perfectly well and completed without any exception.
However combining the previous approach i.e. managing an ABQ and light commits with HA DB will be the best solution.
I'm trying to understand the connection pooling in Datastax Cassandra Driver, so I can better use it in my web service.
I have version 1.0 of the documentation. It says:
The Java driver uses connections asynchronously, so multiple requests can be submitted on the same connection at the same time.
What do they understand by connection? When connecting to a cluster, we have: a Builder, a Cluster and a Session. Which one of them is the connection?
For example, there is this parameter:
maxSimultaneousRequestsPerConnection - number of simultaneous requests on all connections
to a host after which more connections are created.
So, these connections are automatically created, in the case of connection pooling (which is what I would expect). But what exactly are the connections? Cluster objects? Sessions?
I'm trying to decide what to keep 'static' in my web service. For the moment, I decided to keep the Builder static, so for every call I create a new Cluster and a new Session. Is this ok? If the Cluster is the Connection, then it should be ok. But is it? Now, the logger says, for every call:
2013:12:06 12:05:50 DEBUG Cluster:742 - Starting new cluster with contact points
2013:12:06 12:05:50 DEBUG ControlConnection:216 - [Control connection] Refreshing node list and token map
2013:12:06 12:05:50 DEBUG ControlConnection:219 - [Control connection] Refreshing schema
2013:12:06 12:05:50 DEBUG ControlConnection:147 - [Control connection] Successfully connected to...
So, it connects to the Cluster every time? It's not what I want, I want to reuse connections.
So, the connection is actually the Session? If this is the case, I should keep the Cluster static, not the Builder.
What method should I call, to be sure I reuse connections, whenever possible?
The accepted answer (at the time of this writing) is giving the correct advice:
As long as you use the same Session object, you [will] be reusing connections.
However, some parts were originally oversimplified. I hope the following provides insight into the scope of each object type and their respective purposes.
Builder ≠ Cluster ≠ Session ≠ Connection ≠ Statement
A Cluster.Builder is used to configure and create a Cluster
A Cluster represents the entire Cassandra ring
A ring consists of multiple nodes (hosts), and the ring can support one or more keyspaces. You can query a Cluster object about cluster- (ring)-level properties.
I also think of it as the object that represents the calling application to the ring. You communicated your application's needs (e.g. encryption, compression, etc.) to the builder, but it is this object that first implements/communicates with the actual C* ring. If your application uses more than one authentication credential for different users/purposes, you likely have different Cluster objects even if they connect to the same ring.
A Session itself is not a connection, but it manages them
A session may need to talk to all nodes in the ring, which cannot be done with a single TCP connection except in the special case of rings that contain exactly one(1) node. The Session manages a connection pool, and that pool will generally have at least one connection for each node in the ring.
This is why you should re-use Session objects as much as possible. An application does not directly manage or access connections.
A Session is accessed from the Cluster object; it is usually "bound" to a single keyspace at a time, which becomes the default keyspace for the statements executed from that session. A statement can use a fully-qualified table name (e.g. keyspacename.tablename) to access tables in other keyspaces, so it's not required to use multiple sessions to access data across keyspaces. Using multiple sessions to talk to the same ring increases the total number of TCP connections required.
A Statement executes within a Session
Statements can be prepared or not, and each one either mutates data or queries it (and in some cases, both). The fastest, most efficient statements need to communicate with at most one node, and a Session from a topology-aware Cluster should contact only that node (or one of its peers) on a single TCP connection. The least efficient statements must touch all replicas (a majority of nodes), but that will be handled by the coordinator node on the ring itself, so even for these statements the Session will only use a single connection from the application.
Also, versions 2 and 3 of the Cassandra binary protocol used by the driver use multiplexing on the connections. So while a single statement requires at least one TCP connection, that single connection can potentially service up to 128 or 32k+ asynchronous requests simultaneously, depending on the protocol version (respectively).
You are right, the connection is actually in the Session, and the Session is the object you should give to your DAOs to write into Cassandra.
As long as you use the same Session object, you should be reusing connections (you can see the Session as being your connection pool).
Edit (2017/4/10) : I precised this answer following #William Price one.
Please be aware that this answer is 4 years old, and Cassandra have changed a fair bit in the meantime !
Just an update for the community. You can set connection pool in the following way
private static Cluster cluster;
cluster.getConfiguration().getPoolingOptions().setMaxConnectionsPerHost(HostDistance.LOCAL,100);