We Keep Coding

Using Apache Ignite some expired data remains in memory with TTL enabled and after thread execution

Issue
Create an ignite client (in client mode false) and put some data (10k entries/values) to it with very small expiration time (~20s) and TTL enabled.
Each time the thread is running it'll remove all the entries that expired, but after few attempts this thread is not removing all the expired entries, some of them are staying in memory and are not removed by this thread execution.
That means we got some expired data in memory, and it's something we want to avoid.
Please can you confirm that is a real issue or just misuse/configuration of my setup?
Thanks for your feedback.
Test
I've tried in three different setups: full local mode (embedded server) on MacOS, remote server using one node in Docker, and also remote cluster using 3 nodes in kubernetes.
To reproduce
Git repo: https://github.com/panes/ignite-sample
Run MyIgniteLoadRunnerTest.run() to reproduce the issue described on top.
(Global setup: Writing 10k entries of 64octets each with TTL 10s)

It seems to be a known issue. Here's the link to track it https://issues.apache.org/jira/browse/IGNITE-11438. It's to be included into Ignite 2.8 release. As far as I know it has already been released as a part of GridGain Community Edition.

Hazelcast: Execute EntryEvictedListener only once per cluster in client mode

We are connecting to Hazelcast cluster using Java clients from multiple nodes.
HazelcastClient.newHazelcastClient(cfg)
We need our EntryEvictedListener to be executed only once per cluster.
By default it is executed on all connected clients.
Found how to reach this goal with Hazelcast embedded (Time Based Eviction in Hazelcast), but looks like
map.addLocalEntryListener(...)
is not allowed for client.
So is there any way to execute eviction listener only once per cluster using client?

Unfortunately not. You're listener would need to run on a cluster node, since the local is directly connected to the underlying partitioning scheme. What do you want to do on the evict event, maybe you can achieve it differently.

Zookeeper in-memory log

Does ZooKeeper have an "in-memory log"? I've some experience with ZooKeeper but I never seen anything like it before (from the client side), and after searching I haven't found anything related to an in-memory log. As far as I know, every operation (create, setData, delete) is made on disk.
However, in the paper Ravana: Controller Fault-Tolerance in Software-Defined Networking the authors
Event logging: The master saves each event in ZooKeeper’s distributed
in-memory log. Slaves monitor the log by registering a
trigger for it. When a new event is propagated to a slave’s log, the
trigger is activated so that the slave can read the newly arrived event
locally.
So, assuming that there is an in-memory log, how would a (java) client app use it? Or is it server side only?

OutOfMemoryError due to a huge number of ActiveMQ XATransactionId objects

We have a Weblogic server running several apps. Some of those apps use an ActiveMQ instance which is configured to use the Weblogic XA transaction manager.
Now after about 3 minutes after startup, the JVM triggers an OutOfMemoryError. A heap dump shows that about 85% of all memory is occupied by a LinkedList that contains org.apache.activemq.command.XATransactionId instances. The list is a root object and we are not sure who needs it.
What could cause this?

We had exactly the same issue on Weblogic 12c and activemq-ra. XATransactionId object instances were created continuously causing server overload.
After more than 2 weeks of debugging, we found that the problem was caused by WebLogic Transaction Manager trying to recover some pending activemq transactions by calling the method recover() which returns the ids of transaction that seems to be not completed and have to be recovered. The call to this method by Weblogic returned always a not null number n (always the same) and that causes the creation of n instance of XATransactionId object.
After some investigations, we found that Weblogic stores by default its Transaction logs TLOG in filesystem and this can be changed to be persisted in DB. We thought that there was a problem in TLOGs being in file system and we tried to change it to DB and it worked ! Now our server runs for more that 2 weeks without any restart and memory is stable because no XATransactionId are created a part from the necessary amount of it ;)
I hope this will help you and keep us informed if it worked for you.
Good luck !

To be honest it sounds like you're getting a ton of JMS messages and either not consuming them or, if you are, your consumer is not acknowledging the messages if they are not in auto acknowledge mode.

Check your JMS queue backlog. There may be a queue with high backlog, which server is trying to read. These messages may have been corrupted, due to some crash
The best option is to delete the backlog in JMS queue or take a back up in some other queue

Redis/Jedis no single point of failure and automated failover

In a simple situation with 3 servers with 1 master and 2 slaves with no sharding. Is there a proven solution with java and Jedis that has no single point of failure and will automatically deal with a single server going down be that master or slave(automated failover). e.g. promoting masters and reseting after the failure without any lost data.
It seems to me like it should be a solved problem but I can't find any code on it just high level descriptions of possible ways to do it.
Who actually has this covered and working in production?

You may want to give a try to Redis Sentinel to achieve that:
Redis Sentinel is a system designed to help managing Redis instances.
It performs the following three tasks:
Monitoring. Sentinel constantly check if your master and slave instances are working as expected.
Notification. Sentinel can notify the system administrator, or another computer program, via an API, that something is wrong with one
of the monitored Redis instances.
Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the
other additional slaves are reconfigured to use the new master, and
the applications using the Redis server informed about the new address
to use when connecting.
... or to use an external solution like Zookeeper and Jedis_failover:
JedisPool pool = new JedisPoolBuilder()
.withFailoverConfiguration(
"localhost:2838", // ZooKeeper cluster URL
Arrays.asList( // List of redis servers
new HostConfiguration("localhost", 7000),
new HostConfiguration("localhost", 7001)))
.build();
pool.withJedis(new JedisFunction() {
#Override
public void execute(final JedisActions jedis) throws Exception {
jedis.ping();
}
});
See this presentation of Zookeeper + Redis.
[Update] ... or a pure Java solution with Jedis + Sentinel is to use a wrapper that handles Redis Sentinel events, see SentinelBasedJedisPoolWrapper.

Currently using Jedis 2.4.2 ( from git ), I didn't find a way to do a failover based only on Redis or Sentinel. I hope there will be a way. I am thinking to explore the zookeeper option right now. Redis cluster works well in terms of performance and even stability but its still on beta stage.
If anyone has better insight let us know.