In a simple situation with 3 servers with 1 master and 2 slaves with no sharding. Is there a proven solution with java and Jedis that has no single point of failure and will automatically deal with a single server going down be that master or slave(automated failover). e.g. promoting masters and reseting after the failure without any lost data.
It seems to me like it should be a solved problem but I can't find any code on it just high level descriptions of possible ways to do it.
Who actually has this covered and working in production?
You may want to give a try to Redis Sentinel to achieve that:
Redis Sentinel is a system designed to help managing Redis instances.
It performs the following three tasks:
Monitoring. Sentinel constantly check if your master and slave instances are working as expected.
Notification. Sentinel can notify the system administrator, or another computer program, via an API, that something is wrong with one
of the monitored Redis instances.
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the
other additional slaves are reconfigured to use the new master, and
the applications using the Redis server informed about the new address
to use when connecting.
... or to use an external solution like Zookeeper and Jedis_failover:
JedisPool pool = new JedisPoolBuilder()
.withFailoverConfiguration(
"localhost:2838", // ZooKeeper cluster URL
Arrays.asList( // List of redis servers
new HostConfiguration("localhost", 7000),
new HostConfiguration("localhost", 7001)))
.build();
pool.withJedis(new JedisFunction() {
#Override
public void execute(final JedisActions jedis) throws Exception {
jedis.ping();
}
});
See this presentation of Zookeeper + Redis.
[Update] ... or a pure Java solution with Jedis + Sentinel is to use a wrapper that handles Redis Sentinel events, see SentinelBasedJedisPoolWrapper.
Currently using Jedis 2.4.2 ( from git ), I didn't find a way to do a failover based only on Redis or Sentinel. I hope there will be a way. I am thinking to explore the zookeeper option right now. Redis cluster works well in terms of performance and even stability but its still on beta stage.
If anyone has better insight let us know.
Related
We want to use Ignite as a cache layer on top of Postgres.
We have implemented a Custom Cache Store.
And we experiment some troubles in some situation where Ignite is not stable and we have such error :
javax.cache.CacheException: class org.apache.ignite.IgniteClientDisconnectedException: Client node disconnected: null at
org.apache.ignite.internal.processors.cache.GridCacheGateway.checkState(GridCacheGateway.java:97) at
org.apache.ignite.internal.processors.cache.GridCacheGateway.isStopped(GridCacheGateway.java:269) at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.checkProxyIsValid(GatewayProtectedCacheProxy.java:1597) at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onEnter(GatewayProtectedCacheProxy.java:1621) at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:673)
Ignite is launched apart of our application, and when we launch our app, we loadcache and disable WAL.
When we relaunch our app, without relaunch Ignite, we have these issues.
I am wonder why. Is there any link with the fact that WAL must not be disable? And how to know that cache is already initialised and do not need to loadCache? DO you have recommendation for several apps with custom cache store, connected with one ignite cluster?
Thanks
Please check out https://ignite.apache.org/docs/latest/clustering/connect-client-nodes:
While a client is in a disconnected state and an attempt to reconnect is in progress, the Ignite API throws a IgniteClientDisconnectedException. The exception contains a future that represents a re-connection operation. You can use the future to wait until the operation is complete.
Also, WAL enable-disable is known to have issues, and it's only safe to do on stable topology. Please share logs if you want to investigate.
I have searched solutions for my usecase but did not get right one, so expecting some nice ideas to explore further.
I have two gemfire (version 8.2) clusters (private and public) each stores 110+ GB data without persisting to diskstore. Private cluster gets data from DB and transmits entries to public through WAN gateway until both clusters are online. I have a usecase where I restart only public cluster but it looses data after that and to populate data back I have to restart private cluster and loading data from DB to private cluster that in turn transmits data through WAN.
I can't populate public cluster from DB as it puts load onto my master DB that will affect other applications.
There are multiple solutions I tried.
First: Exporting dataset from private cluster and then importing to public; but this disconnects private cluster gemfire nodes as it stores large volume of data in each region, also I have limitation on disk space for downloading large volumes of data.
Second: There is a possibility that I will expose a JMX bean from public cluster. I then can run a client program that invokes gemfire function in private cluster which iterates through entries and drops entries into public cluster through JMX, but my organizational infrastrucure doesn't let me expose JMX beans in gemfire nodes.
Third: As like second one, gemfire function can transmits data to public cluster through queue which seems to be working but has its own limitations. Queue can only transfer text message of 1MB due to which I need to specially handle large objects and also data transfer includes unnecessary serialization and deserialization (JSON text message).
Is there anyway that I can ask private cluster to re-transmit all data through WAN gateway or any other solution someone can propose me to explore.
You can try "gemtouch" in this open source project gemfire-toolkit.
It sounds very similar to idea 2 but it doesn't require exposing a JMX bean. It does use JMX the same way gfsh does. If that's a problem you could easily remove the use of JMX as it only uses JMX for retrieving the list of regions.
I have the same problem but working with 3 Geode clusters (each in a different location).
When something weird happens in one the clusters, we would need to recover it using one of the existing 2 remaining clusters:
If we "touch" one of the clusters, that means that all that info will replicate to cluster that needs recovery, but also to the other cluster that is actually OK. Probably that is OK is not causing any damage, but I would appreciate any opinion.
If we keep running traffic on the remaining 2 clusters while in one of them we are running GemTouch I guess some consistency problems between cluster could pop-up, but not sure.
Last topic it is about LICENSE of gemfire-toolkit. Actually, there is no LICENSE file, so I am not 100% sure if the tool can be used.
We are connecting to Hazelcast cluster using Java clients from multiple nodes.
HazelcastClient.newHazelcastClient(cfg)
We need our EntryEvictedListener to be executed only once per cluster.
By default it is executed on all connected clients.
Found how to reach this goal with Hazelcast embedded (Time Based Eviction in Hazelcast), but looks like
map.addLocalEntryListener(...)
is not allowed for client.
So is there any way to execute eviction listener only once per cluster using client?
Unfortunately not. You're listener would need to run on a cluster node, since the local is directly connected to the underlying partitioning scheme. What do you want to do on the evict event, maybe you can achieve it differently.
I have an application incorporating a stretched Hazelcast cluster deployed on 2 data centres simultaneously. The 2 data centres are usually both fully functional, but, at times, one of them is taken completely out of the network for SDN upgrades.
What I intend to achieve is to configure the cluster in such a way that each main partition from a DC will have at least 2 backups - one in the other cluster and one in the current one.
For this purpose, checking the documentation pointed me toward the direction of partition groups(http://docs.hazelcast.org/docs/2.3/manual/html/ch12s03.html). Enterprise WAN Replication seemed exactly like the thing we wanted, but, unfortunately, this feature is not available for the free version of Hazelcast.
My configuration is as follows:
NetworkConfig network = config.getNetworkConfig();
network.setPort(hzClusterConfigs.getPort());
JoinConfig join = network.getJoin();
join.getMulticastConfig().setEnabled(hzClusterConfigs.isMulticastEnabled());
join.getTcpIpConfig()
.setMembers(hzClusterConfigs.getClusterMembers())
.setEnabled(hzClusterConfigs.isTcpIpEnabled());
config.setNetworkConfig(network);
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig()
.setEnabled(true).setGroupType(PartitionGroupConfig.MemberGroupType.CUSTOM)
.addMemberGroupConfig(new MemberGroupConfig().addInterface(hzClusterConfigs.getClusterDc1Interface()))
.addMemberGroupConfig(new MemberGroupConfig().addInterface(hzClusterConfigs.getClusterDc2Interface()));
config.setPartitionGroupConfig(partitionGroupConfig);
The configs used initially were:
clusterMembers=host1,host2,host3,host4
clusterDc1Interface=10.10.1.*
clusterDc2Interface=10.10.1.*
However, with this set of configs at any event triggered when changing the components of the cluster, a random node in the cluster started logging "No member group is available to assign partition ownership" every other second (as here: https://github.com/hazelcast/hazelcast/issues/5666). What is more, checking the state exposed by the PartitionService in JMX revealed that no partitions were actually getting populated, despite the apparently successful cluster state.
As such, I proceeded to replacing hostnames with the corresponding IPs and the configuration worked. The partitions were getting created successfully and no nodes were acting out.
The problem here is that the boxes are created as part of an A/B deployment process and get their IPs automatically from a range of 244 IPs. Adding all those 244 IPs seems like a bit much, even if it would be done programatically from Chef and not manually, because of all the network noise it would entail. Checking at every deployment using a telnet-based client which machines are listening on the hazelcast port also seems a bit problematic, since the IPs will be different from a deployment to another and we would get ourselves into a situation in which a part of the nodes in the cluster will have a certain member list and another part will have a different member list at the same time.
Using hostnames would be the best solution, in my opinion, because we would rely on DNS resolution and wouldn't need to wrap our heads around IP resolution at provisioning time.
Does anyone know of a workaround for the group config issue? Or, perhaps, an alternative to achieve the same behavior?
This is not possible at the moment. Backup groups cannot be designed the way to have a backup of themselves. As a workaround you might be able to design 4 groups but in this case there is no guarantee that one backup will be on each datacenter, at least not without using 3 backups.
Anyhow in general we do not recommend to spread a Hazelcast cluster over multiple datacenters, except for the very specific situation where the DCs are interconnected in a way that it is similar to a LAN network and redundancy is set up.
I'm trying to test out MariaDB, Galera, and Corosync/Pacemaker to understand clustering with high-availability using CentOS 7 servers. The cluster size I am using for testing is 3 servers to prevent quorum issues for the most part. My tests and application are written in Java.
I have the clustering part down; it works in an active-active configuration and runs just fine. As well, I have HA set up using Pacemaker and corosync. I've done many tests with it in failing the slaves or bringing them up during the run. As well, I've written to all three at run-time regardless of if it was the master connection or one of the child connections. When I try to test the Master going down during run-time (to simulate a power outage, server crash, or whatever in the data center), the application immediately stops running. I get a java.net.SocketException error and the application closes with the other two connections being shut down successfully. I've used both the kill and stop commands in the terminal to test and see if it'll work (just in the off chance it would).
JDBC URL String
Below is the part of the code that connects the application to the cluster. It connects correctly and does work until I cause the first master to go down; the other two going down does not affect this.
public void connections() {
try {
bigConnec = DriverManager.getConnection(
"jdbc:mariadb:sequential:failover:loadbalance://"
+ "10.32.18.90,10.32.18.91,10.32.18.92/"+DB+"?autoReconnect=true&failOverReadOnly=false"
+ "&retriesAllDown=120",
"root", "PASS");
bigConnec.setAutoCommit(false);
} catch(SQLException e) {
System.err.println("Unable to connect to any one of the three servers! \n" + e);
System.exit(1);
}
...
}
There are three other connections made to each individual server so I can pull information more easily from them; that is what the ellipses indicate. The servers will exchange which is the "primary" node but the application will not connect to the next node in the list.
I feel like the issue is in the way I have my URL set up because everything works outside of the cluster. As well, nothing happens during testing when I shut down the child nodes. The most that happens is I'll get a warning that the URL lost connection to either or both of them. Is there a way to configure the URL in such a way to allow automatic failover to the next available node in the string or do I have to go about it some other way using individual connection URLs and Objects (or an array of Connection objects) or black magic and pixie dust that really only SysAdmins know some other way I have yet to try?
What I Have Tried
How to make MaxScale High Available with Corosync/Pacemaker (MariaDB Article)
Failover and High availability with MariaDB Connector/J (MariaDB Documentation)
What is the right MariaDB Galera jdbc URL properties for loadbalance (Stack Overflow)
HA Proxy Configuration with MariaDB Galera cluster (Stack Overflow)
Configuring Server Failover (MySQL Documentation)
Advanced Load-balancing and Failover Configuration (MySQL Documentation)
TL;DR
Problem: Java application is not automatically failing over despite being flagged for failover and sequential support in the JDBC URL (see JDBC URL String). Everything works with corosync and Pacemaker but I cannot get the Java application to transfer to the next available node to act as the primary connection when the current one goes down.
Question: Is the issue in the URL? A follow-up to that is, if so, would it be better to use three separate connections and use the first valid one or is there something I can do to allow the application to automatically rollover to the next available connection in the current URL?
Software/Equipment
MariaDB 10.1.24
corosync 2.4.0
Pacemaker 1.1.15
CentOS 7
Java 8 / Eclipse Neon.3 / Eclipse 4.6.3
MariaDB Connector/J 2.0.1
If there is any more information you need, please do tell me in the comments and I'll update this as soon as I can!