We Keep Coding

Concurrent blocking map with entries eviction

What I need is a fairly complex data structure with the following requirements:
It should support concurrent reads/writes without any excessive locking (like java.util.concurrent.ConcurrentHashMap)
It should have capacity limit and block once the limit is reached (just like BlockingQueue implementations)
It should have efficient search mechanism, like Map/HashSet do: given an ID of an object, I need to be able to find it without sequential scan.
It should be possible to evict elements on timeout, for instance: if an entry is put in this structure more than X minutes ago, it should be automatically removed.
Of course, there's always a chance to implement it on my own, but I'd prefer to find something existing, optimized and well-tested.
The only thing that's near is Guava's cache, but it seems to be missing #2. Any ideas on known implementations of this?

You could write a simple BlockingCache, which wraps an existing Guava Cache and checks capacity on put operations, so the put would look something like this:
public V put(K key, V value)
{
while (size() >= capacity) Thread.sleep(100);
return innerCache.put(key, value);
}

A cache which knows about reachability

I'd like a cache with some maximum retaining capacity of N. I'm allowing it to hold up to N objects which would otherwise be eligible for GC. Now, if my application itself currently holds N+1 strong references to objects which it's previously added to the cache, I want the cache to hold N+1 too. Why? Because the cache won't be keeping this N+1th object from being collected any longer than it would be otherwise, and I'm fine trading a bigger hash table for more cache hits.
Another way of putting it, I'd like an object cache which retains all objects added to it while they remain strongly reachable, and also retains enough non-strongly reachable objects to keep its size == N.
Example
We have a cache created with N=100. Size starts at 0. 150 objects are added, size is 150. 100 of those objects become non-strongly reachable (weakly, softly, whatever). Cache evicts 50 of those and keeps 50, size is 100. 49 more strongly reachable objects are added. Size is still 100 but now 99 of them are strongly reachable and only one is non-strongly reachable. What happened is 49 older, non-strongly reachable objects were replaced with the new 49 because the new ones were strongly reachable.
Motivation
I suspect it's actually an intuitive thing to want for a number of use cases. Typically the cache's capacity trades off cache hit probability for a guarantee for maximum memory usage. Knowing about the reachability of the objects it holds, a cache could deliver higher cache hit probability without changing its maximum memory usage guarantee.
The Trouble
I'm worried it's not possible on the JVM. I'm hoping to be told otherwise, but if you know for a fact it's not possible I'll accept that answer too if there's rationale.

You can add the entries to a LinkedHashMap configured as an LRU or FIFO cache. You can have a WeakHashMap as well. If you add the key to both maps, the LHM will prevent cleanup even though its in the WHM. Once the LHM discards the key, it may or may not be in the WHM.
e.g
private final int retainedSize;
private final Map<K,V> lruMap = new LinkedHashMap<K, V>(16, 0.7f, true) {
#Override
protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
return size() > retainedSize;
}
};
private final Map<K,V> weakMap = new WeakHashMap<K, V>();
public void put(K k, V v) {
lruMap.put(k, v);
weakMap.put(k,v);
}
public V get(K k) {
V v = lruMap.get(k);
return v == null ? weakMap.get(k) : v;
}
One of the reason to do this is that a WeakHashMap is like to be clearer all at once, so you hit rate can drop very sharply. This approach ensures that after you have been hit with a Full GC, your performance won't drop too much as you try to catch up. ;)

Check out WeakHashMap. Stale references will be removed automatically. Before putting you could check if size exceeds your threshold and skip putting in a new value.
Alternativley you could override put and discard the value if the size is above your threshold.
This method would work as you propose, since you do not need a cache eviction policy you could just skip putting in new elements if the size is greater than your threshold.

I think what you want makes sense, but maybe not that much. Let's assume that the values are quite big (some kilobytes), otherwise the caching of the elsewhere strongly hold values may get expensive too. Ignore this overhead, your cache indeed has constant memory costs. However, I'm not sure if this goal is worth pursuing -- I'm rather interested how to use about constant amount of memory for the whole program (I don't want to leave too much memory unused and in no case I want to start swapping).
Idea: The cache should use registered weak (or soft) references.1 You use another thread calling ReferenceQueue.remove() in loop and checking some condition2. Depending on it, you either remove the corresponding entry from the cache (as Guava does) or you resurrect the value via reference.get() and thus protect it temporarily from being garbage collected.3. This should work, but it costs some time during each GC run.
1Overriding finalize() would do as well. Actually, it looks like this is the only way as reference.get() when enqueued always returns null so it can't be used for resurrection.
2The condition should be sort of "do it 100 times per GC run".
3I'm not sure the GC really works this way, but I suppose it does. If not, then you could use a copy of the value instead. I'm also unsure what happens when the value loses strong reachability the next time, but again, this is surely solvable (e.g., create a new Reference).

reliably forcing Guava map eviction to take place

EDIT: I've reorganized this question to reflect the new information that since became available.
This question is based on the responses to a question by Viliam concerning Guava Maps' use of lazy eviction: Laziness of eviction in Guava's maps
Please read this question and its response first, but essentially the conclusion is that Guava maps do not asynchronously calculate and enforce eviction. Given the following map:
ConcurrentMap<String, MyObject> cache = new MapMaker()
.expireAfterAccess(10, TimeUnit.MINUTES)
.makeMap();
Once ten minutes has passed following access to an entry, it will still not be evicted until the map is "touched" again. Known ways to do this include the usual accessors - get() and put() and containsKey().
The first part of my question [solved]: what other calls cause the map to be "touched"? Specifically, does anyone know if size() falls into this category?
The reason for wondering this is that I've implemented a scheduled task to occasionally nudge the Guava map I'm using for caching, using this simple method:
public static void nudgeEviction() {
cache.containsKey("");
}
However I'm also using cache.size() to programmatically report the number of objects contained in the map, as a way to confirm this strategy is working. But I haven't been able to see a difference from these reports, and now I'm wondering if size() also causes eviction to take place.
Answer: So Mark has pointed out that in release 9, eviction is invoked only by the get(), put(), and replace() methods, which would explain why I wasn't seeing an effect for containsKey(). This will apparently change with the next version of guava which is set for release soon, but unfortunately my project's release is set sooner.
This puts me in an interesting predicament. Normally I could still touch the map by calling get(""), but I'm actually using a computing map:
ConcurrentMap<String, MyObject> cache = new MapMaker()
.expireAfterAccess(10, TimeUnit.MINUTES)
.makeComputingMap(loadFunction);
where loadFunction loads the MyObject corresponding to the key from a database. It's starting to look like I have no easy way of forcing eviction until r10. But even being able to reliably force eviction is put into doubt by the second part of my question:
The second part of my question [solved]: In reaction to one of the responses to the linked question, does touching the map reliably evict all expired entries? In the linked answer, Niraj Tolia indicates otherwise, saying eviction is potentially only processed in batches, which would mean multiple calls to touch the map might be needed to ensure all expired objects were evicted. He did not elaborate, however this seems related to the map being split into segments based on concurrency level. Assuming I used r10, in which a containsKey("") does invoke eviction, would this then be for the entire map, or only for one of the segments?
Answer: maaartinus has addressed this part of the question:
Beware that containsKey and other reading methods only run postReadCleanup, which does nothing but on each 64th invocation (see DRAIN_THRESHOLD). Moreover, it looks like all cleanup methods work with single Segment only.
So it looks like calling containsKey("") wouldn't be a viable fix, even in r10. This reduces my question to the title: How can I reliably force eviction to occur?
Note: Part of the reason my web app is noticeably affected by this issue is that when I implemented caching I decided to use multiple maps - one for each class of my data objects. So with this issue there is the possibility that one area of code is executed, causing a bunch of Foo objects to be cached, and then the Foo cache isn't touched again for a long time so it doesn't evict anything. Meanwhile Bar and Baz objects are being cached from other areas of code, and memory is being eaten. I'm setting a maximum size on these maps, but this is a flimsy safeguard at best (I'm assuming its effect is immediate - still need to confirm this).
UPDATE 1: Thanks to Darren for linking the relevant issues - they now have my votes. So it looks like a resolution is in the pipeline, but seems unlikely to be in r10. In the meantime, my question remains.
UPDATE 2: At this point I'm just waiting for a Guava team member to give feedback on the hack maaartinus and I put together (see answers below).
LAST UPDATE: feedback received!

I just added the method Cache.cleanUp() to Guava. Once you migrate from MapMaker to CacheBuilder you can use that to force eviction.

I was wondering the about the same issue you described in the first part of your question. From what I can tell from looking at the source code for Guava's CustomConcurrentHashMap (release 9), it appears that entries are evicted on the get(), put(), and replace() methods. The containsKey() method does not appear to invoke eviction. I'm not 100% sure because I took a quick pass at the code.
Update:
I also found a more recent version of the CustomConcurrentHashmap in Guava's git repository and it looks like containsKey() has been updated to invoke eviction.
Both release 9 and the latest version I just found do not invoke eviction when size() is called.
Update 2:
I recently noticed that Guava r10 (yet to be released) has a new class called CacheBuilder. Basically this class is a forked version of the MapMaker but with caching in mind. The documentation suggests that it will support some of the eviction requirements you are looking for.
I reviewed the updated code in r10's version of the CustomConcurrentHashMap and found what looks like a scheduled map cleaner. Unfortunately, that code appears unfinished at this point but r10 looks more and more promising each day.

Beware that containsKey and other reading methods only run postReadCleanup, which does nothing but on each 64th invocation (see DRAIN_THRESHOLD). Moreover, it looks like all cleanup methods work with single Segment only.
The easiest way to enforce eviction seems to be to put some dummy object into each segment. For this to work, you'd need to analyze CustomConcurrentHashMap.hash(Object), which is surely no good idea, as this method may change anytime. Moreover, depending on the key class it may be hard to find a key with a hashCode ensuring it lands in a given segment.
You could use reads instead, but would have to repeat them 64 times per segment. Here, it'd easy to find a key with an appropriate hashCode, since here any object is allowed as an argument.
Maybe you could hack into the CustomConcurrentHashMap source code instead, it could be as trivial as
public void runCleanup() {
final Segment<K, V>[] segments = this.segments;
for (int i = 0; i < segments.length; ++i) {
segments[i].runCleanup();
}
}
but I wouldn't do it without a lot of testing and/or an OK by a guava team member.

Yep, we've gone back and forth a few times on whether these cleanup tasks should be done on a background thread (or pool), or should be done on user threads. If they were done on a background thread, this would eventually happen automatically; as it is, it'll only happen as each segment gets used. We're still trying to come up with the right approach here - I wouldn't be surprised to see this change in some future release, but I also can't promise anything or even make a credible guess as to how it will change. Still, you've presented a reasonable use case for some kind of background or user-triggered cleanup.
Your hack is reasonable, as long as you keep in mind that it's a hack, and liable to break (possibly in subtle ways) in future releases. As you can see in the source, Segment.runCleanup() calls runLockedCleanup and runUnlockedCleanup: runLockedCleanup() will have no effect if it can't lock the segment, but if it can't lock the segment it's because some other thread has the segment locked, and that other thread can be expected to call runLockedCleanup as part of its operation.
Also, in r10, there's CacheBuilder/Cache, analogous to MapMaker/Map. Cache is the preferred approach for many current users of makeComputingMap. It uses a separate CustomConcurrentHashMap, in the common.cache package; depending on your needs, you may want your GuavaEvictionHacker to work with both. (The mechanism is the same, but they're different Classes and therefore different Methods.)

I'm not a big fan of hacking into or forking external code until absolutely necessary. This problem occurs in part due to an early decision for MapMaker to fork ConcurrentHashMap, thereby dragging in a lot of complexity that could have been deferred until after the algorithms were worked out. By patching above MapMaker, the code is robust to library changes so that you can remove your workaround on your own schedule.
An easy approach is to use a priority queue of weak reference tasks and a dedicated thread. This has the drawback of creating many stale no-op tasks, which can become excessive in due to the O(lg n) insertion penalty. It works reasonably well for small, less frequently used caches. It was the original approach taken by MapMaker and its simple to write your own decorator.
A more robust choice is to mirror the lock amortization model with a single expiration queue. The head of the queue can be volatile so that a read can always peek to determine if it has expired. This allows all reads to trigger an expiration and an optional clean-up thread to check regularly.
By far the simplest is to use #concurrencyLevel(1) to force MapMaker to use a single segment. This reduces the write concurrency, but most caches are read heavy so the loss is minimal. The original hack to nudge the map with a dummy key would then work fine. This would be my preferred approach, but the other two options are okay if you have high write loads.

I don't know if it is appropriate for your use case, but your main concern about the lack of background cache eviction seems to be memory consumption, so I would have thought that using softValues() on the MapMaker to allow the Garbage Collector to reclaim entries from the cache when a low memory situation occurs. Could easily be the solution for you. I have used this on a subscription-server (ATOM) where entries are served through a Guava cache using SoftReferences for values.

Based on maaartinus's answer, I came up with the following code which uses reflection rather than directly modifying the source (If you find this useful please upvote his answer!). While it will come at a performance penalty for using reflection, the difference should be negligible since I'll run it about once every 20 minutes for each caching Map (I'm also caching the dynamic lookups in the static block which will help). I have done some initial testing and it appears to work as intended:
public class GuavaEvictionHacker {
//Class objects necessary for reflection on Guava classes - see Guava docs for info
private static final Class<?> computingMapAdapterClass;
private static final Class<?> nullConcurrentMapClass;
private static final Class<?> nullComputingConcurrentMapClass;
private static final Class<?> customConcurrentHashMapClass;
private static final Class<?> computingConcurrentHashMapClass;
private static final Class<?> segmentClass;
//MapMaker$ComputingMapAdapter#cache points to the wrapped CustomConcurrentHashMap
private static final Field cacheField;
//CustomConcurrentHashMap#segments points to the array of Segments (map partitions)
private static final Field segmentsField;
//CustomConcurrentHashMap$Segment#runCleanup() enforces eviction on the calling Segment
private static final Method runCleanupMethod;
static {
try {
//look up Classes
computingMapAdapterClass = Class.forName("com.google.common.collect.MapMaker$ComputingMapAdapter");
nullConcurrentMapClass = Class.forName("com.google.common.collect.MapMaker$NullConcurrentMap");
nullComputingConcurrentMapClass = Class.forName("com.google.common.collect.MapMaker$NullComputingConcurrentMap");
customConcurrentHashMapClass = Class.forName("com.google.common.collect.CustomConcurrentHashMap");
computingConcurrentHashMapClass = Class.forName("com.google.common.collect.ComputingConcurrentHashMap");
segmentClass = Class.forName("com.google.common.collect.CustomConcurrentHashMap$Segment");
//look up Fields and set accessible
cacheField = computingMapAdapterClass.getDeclaredField("cache");
segmentsField = customConcurrentHashMapClass.getDeclaredField("segments");
cacheField.setAccessible(true);
segmentsField.setAccessible(true);
//look up the cleanup Method and set accessible
runCleanupMethod = segmentClass.getDeclaredMethod("runCleanup");
runCleanupMethod.setAccessible(true);
}
catch (ClassNotFoundException cnfe) {
throw new RuntimeException("ClassNotFoundException thrown in GuavaEvictionHacker static initialization block.", cnfe);
}
catch (NoSuchFieldException nsfe) {
throw new RuntimeException("NoSuchFieldException thrown in GuavaEvictionHacker static initialization block.", nsfe);
}
catch (NoSuchMethodException nsme) {
throw new RuntimeException("NoSuchMethodException thrown in GuavaEvictionHacker static initialization block.", nsme);
}
}
/**
* Forces eviction to take place on the provided Guava Map. The Map must be an instance
* of either {#code CustomConcurrentHashMap} or {#code MapMaker$ComputingMapAdapter}.
*
* #param guavaMap the Guava Map to force eviction on.
*/
public static void forceEvictionOnGuavaMap(ConcurrentMap<?, ?> guavaMap) {
try {
//we need to get the CustomConcurrentHashMap instance
Object customConcurrentHashMap;
//get the type of what was passed in
Class<?> guavaMapClass = guavaMap.getClass();
//if it's a CustomConcurrentHashMap we have what we need
if (guavaMapClass == customConcurrentHashMapClass) {
customConcurrentHashMap = guavaMap;
}
//if it's a NullConcurrentMap (auto-evictor), return early
else if (guavaMapClass == nullConcurrentMapClass) {
return;
}
//if it's a computing map we need to pull the instance from the adapter's "cache" field
else if (guavaMapClass == computingMapAdapterClass) {
customConcurrentHashMap = cacheField.get(guavaMap);
//get the type of what we pulled out
Class<?> innerCacheClass = customConcurrentHashMap.getClass();
//if it's a NullComputingConcurrentMap (auto-evictor), return early
if (innerCacheClass == nullComputingConcurrentMapClass) {
return;
}
//otherwise make sure it's a ComputingConcurrentHashMap - error if it isn't
else if (innerCacheClass != computingConcurrentHashMapClass) {
throw new IllegalArgumentException("Provided ComputingMapAdapter's inner cache was an unexpected type: " + innerCacheClass);
}
}
//error for anything else passed in
else {
throw new IllegalArgumentException("Provided ConcurrentMap was not an expected Guava Map: " + guavaMapClass);
}
//pull the array of Segments out of the CustomConcurrentHashMap instance
Object[] segments = (Object[])segmentsField.get(customConcurrentHashMap);
//loop over them and invoke the cleanup method on each one
for (Object segment : segments) {
runCleanupMethod.invoke(segment);
}
}
catch (IllegalAccessException iae) {
throw new RuntimeException(iae);
}
catch (InvocationTargetException ite) {
throw new RuntimeException(ite.getCause());
}
}
}
I'm looking for feedback on whether this approach is advisable as a stopgap until the issue is resolved in a Guava release, particularly from members of the Guava team when they get a minute.
EDIT: updated the solution to allow for auto-evicting maps (NullConcurrentMap or NullComputingConcurrentMap residing in a ComputingMapAdapter). This turned out to be necessary in my case, since I'm calling this method on all of my maps and a few of them are auto-evictors.

Use PermGen space or roll-my-own intern method?

I am writing a Codec to process messages sent over TCP using a bespoke wire protocol. During the decode process I create a number of Strings, BigDecimals and dates. The client-server access patterns mean that it is common for the client to issue a request and then decode thousands of response messages, which results in a large number of duplicate Strings, BigDecimals, etc.
Therefore I have created an InternPool<T> class allowing me to intern each class of object. Internally, the pool uses a WeakHashMap<T, WeakReference<T>>. For example:
InternPool<BigDecimal> pool = new InternPool<BigDecimal>();
...
// Read BigDecimal from in buffer and then intern.
BigDecimal quantity = pool.intern(readBigDecimal(in));
My question: I am using InternPool for BigDecimal but should I consider also using it for String instead of String's intern() method, which I believe uses PermGen space? What is the advantage of using PermGen space?

If you already have such a InternPool class, it think it is better to use that than to choose a different interning method for Strings. Especially since String.intern() seems to give a much stronger guarantee than you actually need. Your goal is to reduce memory usage, so perfect interning for the lifetime of the JVM is not actually necessary.
Also, I'd use the Google Collections MapMaker to create a InternPool to avoid re-creating the wheel:
Map<BigDecimal,BigDecimal> bigDecimalPool = new MapMaker()
.weakKeys()
.weakValues()
.expiration(1, TimeUnits.MINUTES)
.makeComputingMap(
new Function<BigDecimal, BigDecimal>() {
public BigDecimal apply(BigDecimal value) {
return value;
}
});
This would give you (correctly implemented) weak keys and values, thread safety, automatic purging of old entries and a very simple interface (a simple, well-known Map). To be sure you could also wrap it using Collections.immutableMap() to avoid bad code messing with it.

It is likely that the JVM's String.intern() pool will be faster. AFAIK, it is implemented in native code, so it should be faster and use less space than a pool implemented using WeakHashMap and WeakReference. You would need to do some careful benchmarking to confirm this.
However, unless you have huge numbers of long-lived duplicate objects, I doubt that interning (either in permGen or with your own pools) will make much difference. And if the ratio of unique to duplicate objects is too low, then interning will just increase the number of live objects (making the GC take longer) and reduce performance due the overheads of interning, and so on. So I would also advocate benchmarking the "intern" versus "no intern" approaches.

Using SoftReference for static data to prevent memory shortage in Java

I have a class with a static member like this:
class C
{
static Map m=new HashMap();
{
... initialize the map with some values ...
}
}
AFAIK, this would consume memory practically to the end of the program. I was wondering, if I could solve it with soft references, like this:
class C
{
static volatile SoftReference<Map> m=null;
static Map getM() {
Map ret;
if(m == null || (ret = m.get()) == null) {
ret=new HashMap();
... initialize the map ...
m=new SoftReference(ret);
}
return ret;
}
}
The question is
is this approach (and the implementation) right?
if it is, does it pay off in real situations?

First, the code above is not threadsafe.
Second, while it works in theory, I doubt there is a realistic scenario where it pays off. Think about it: In order for this to be useful, the map's contents would have to be:
Big enough so that their memory usage is relevant
Able to be recreated on the fly without unacceptable delays
Used only at times when other parts of the program require less memory - otherwise the maximum memory required would be the same, only the average would be less, and you probably wouldn't even see this outside the JVM since it give back heap memory to the OS very reluctantly.
Here, 1. and 2. are sort of contradictory - large objects also take longer to create.

This is okay if your access to getM is single threaded and it only acts as a cache.
A better alternative is to have a fixed size cache as this provides a consistent benefit.

getM() should be synchronized, to avoid m being initialized at the same time by different threads.

How big is this map going to be ? Is it worth the effort to handle it this way ? Have you measured the memory consumption of this (for what it's worth, I believe the above is generally ok, but my first question with optimisations is "what does it really save me").
You're returning the reference to the map, so you need to ensure that your clients don't hold onto this reference (and prevent garbage collection). Perhaps your class can hold the reference, and provide a getKey() method to access the content of the map on behalf of clients ? That way you'll maintain control of the reference to the map in one place.
I would synchronise the above, in case the map gets garbage collected and two threads hit getMap() at the same time. Otherwise you're going to create two maps simultaneously!

Maybe you are looking for WeakHashMap? Then entries in the map can be garbage collected separately.
Though in my experience it didn't help much, so I instead built an LRU cache using LinkedHashMap. The advantage is that I can control the size so that it isn't too big and still useful.

I was wondering, if I could solve it with soft references
What is it that you are trying to solve? Are you running into memory problems, or are you prematurely optimizing?
In any case,
The implementation should be altered a bit if you were to use it. As has been noted, it isnt thread-safe. Multiple threads could access the method at the same time, allowing multiple copies of your collection to be created. If these collections were then strongly referenced for the remainder of your program you would end up with more memory consumption, not less
A reason to use SoftReferences is to avoid running out of memory, as there is no contract other than that they will be cleared before the VM throws an OutOfMemoryError. Therefore there is no guaranteed benefit of this approach, other than not creating the cache until it is first used.

The first thing I notice about the code is that it mixes generic with raw types. That is just going to lead to a mess. javac in JDK7 has -Xlint:rawtypes to quickly spot that kind of mistake before trouble starts.
The code is not thread-safe but uses statics so is published across all threads. You probably don' want it to be synchronized because the cause problems if contended on multithreaded machines.
A problem with use a SoftReference for the entire cache is that you will cause spikes when the reference is cleared. In some circumstances it might work out better to have ThreadLocal<SoftReference<Map<K,V>>> which would spread the spikes and help-thread safety at the expense of not sharing between threads.
However, creating a smarter cache is more difficult. Often you end up with values referencing keys. There are ways around this bit it is a mess. I don't think ephemerons (essentially a pair of linked References) are going to make JDK7. You might find the Google Collections worth looking at (although I haven't).
java.util.LinkedHashMap gives an easy way to limit the number of cached entries, but is not much use if you can't be sure how big the entries are, and can cause problems if it stops collection of large object systems such as ClassLoaders. Some people have said you shouldn't leave cache eviction up to the whims of the garbage collector, but then some people say you shouldn't use GC.