Hi I am running a thread service, the job of this thread is to check the age of a list items in a HashMap. When an item is older than say 5 seconds, I will have to delete the item from the HashMap. The below is the simplified code. But when the code attempts to delete the item from the HashMap, I get a java.util.ConcurrentModificationException.
I am populating the HashMap in the main() method in the original program.
Can somebody please help me out with this ? PS: The deleteFromTrackList() is being called by different clients across a network through RMI.
import java.util.*;
public class NotifierThread extends Thread {
private HashMap<Integer, ArrayList> NotificationTrackList = new HashMap<Integer, ArrayList>();
#Override
public void run() {
while (true) { // this process should run continuously
checkNotifierList(getNotificationTrackList());
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public HashMap<Integer, ArrayList> getNotificationTrackList() {
return NotificationTrackList;
}
public void deleteFromTrackList(Integer messageID) {
NotificationTrackList.remove(messageID);
}
public synchronized void checkNotifierList(HashMap list) {
Set entries = list.entrySet();
for (Iterator iterator = entries.iterator(); iterator.hasNext();) {
Map.Entry<Integer, ArrayList> entry = (Map.Entry) iterator.next();
ArrayList messageInfo = entry.getValue();
Integer messageID = entry.getKey();
messageInfo = new ArrayList((ArrayList) list.get(messageID));
Long curTime = new Date().getTime();
Long refTime = (Long) messageInfo.get(1);
Long timeDiff = curTime - refTime;
if (timeDiff > 5000) {
// delete the entry if its older than 5 milliseconds and update
// internal entry list
deleteFromTrackList(messageID);
}
}
}
public static void main(String[] args) {
new NotifierThread().start();
}
}
This is the stacktrace I am getting at the console
Exception in thread "tracker" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
at java.util.HashMap$EntryIterator.next(Unknown Source)
at java.util.HashMap$EntryIterator.next(Unknown Source)
at NotifierThread.checkNotifierList(NotifierThread.java:32)
at NotifierThread.run(NotifierThread.java:10)
The only way to remove an entry from a map while iterating over it is to remove it using the iterator. Use
iterator.remove();
instead of
deleteFromTrackList(messageID);
Note that the same applies to all the collections (List, Set, etc.)
Also, note that your design is not thread-safe, because you let other threads access the map in an unsynchronized way.
Your code isn't complectly synchronized. Try to change
public void deleteFromTrackList(Integer messageID) {
to
public synchronized void deleteFromTrackList(Integer messageID) {
Correct. You cannot modify a Map while iterating over it without using the iterator directly. There are a couple main options.
Create a List of elements that should be removed. Add each expired element to the List in the loop. After the loop, remove the elements in the List from the Map.
Use Guava's filter capability.
Maps.filterEntries This creates a new Map however which may not work for what you are trying to do.
Since you have a multi-threaded system. You may want to consider immutability as your friend. Rather than blocking threads over your entire check for stale loop, you could use an ImmutableMap which would be more thread-safe with better performance.
Thanks for your answers guys... I have found the solution for my question, instead of using HashMap, I am using ConcurrentHashMap. This solved my problem. Thanks again !
In reality you do not even need concurrent access to an hash map to get a concurrency exception.
In fact, a single thread is quite enough.
For example,
You may create a loop based on the hash map map.keySet().iterator(),
And, while you are within this loop, your (single) thread decides to remove an element from the hash map. (Not a good idea while the iterator is open.)
In the next request to the iterator().next() you will get your concurrency exception.
So careful with that.
Related
I have a java restful webservice program thats hosted on tomcat. In one of my web service methods, I load a big arraylist of objects (about 25,000 entries) from redis. This arraylist is updated once every 30 mins. There are multiple threads reading from this arraylist all the time. When, I update the arraylist I want to cause minimum disruption/delays since there could be other threads reading from it.
I was wondering what is the best way to do this? One way is to use synchronized keyword to the method that updates the list. But, the synchronized method has an overhead, since no threads can read while the update is going on. The update method itself could take few hundred millisecs since it involves reading from redis + deserialization.
class WebService {
ArrayList<Entry> list = new ArrayList<Entry>();
//need to call this every 30 mins.
void syncrhonized updateArrayList(){
//read from redis & add elements to list
}
void readFromList(){
for(Entry e: list) {
//do some processing
}
}
}
Updated the final solution:
I ended up using no explicit synchronization primitives.
Does it have to be the same List instance getting updated? Can you build a new list every 30 minutes and replace a volatile reference?
Something along these lines:
class WebService {
private volatile List<Entry> theList;
void updateList() {
List<Entry> newList = getEntriesFromRedis();
theList = Collections.unmodifiableList(newList);
}
public List<Entry> getList() {
return theList;
}
}
The advantage of this approach is that you don't have to do any other synchronization anywhere else.
A reader-writer lock (or ReadWriteLock in Java) is what you need.
A reader-writer lock will allow concurrent access for read operations, but mutually exclusive access for write.
It will look something like
class WebService {
final ReentrantReadWriteLock listRwLock = new ReentrantReadWriteLock();
ArrayList<Entry> list = new ArrayList<Entry>();
//need to call this every 30 mins.
void updateArrayList(){
listRwLock.writeLock().lock();
try {
//read from redis & add elements to list
} finally {
listRwLock.writeLock().unlock()
}
}
void readFromList(){
listRwLock.readLock().lock();
try {
for(Entry e: list) {
//do some processing
}
} finally {
listRwLock.readLock().unlock()
}
}
}
Here is the solution I finally ended up with,
class WebService {
// key = timeWindow (for ex:10:00 or 10:30 or 11:00), value = <List of entries for that timewindow>
ConcurrentHashMap<String, List<Entry>> map= new ConcurrentHashMap<String, List<Entry>>();
//have setup a timer to call this every 10 mins.
void updateArrayList(){
// populate the map for the next time window with the corresponding entries. So that its ready before we start using it. Also, clean up the expired entries for older time windows.
}
void readFromList(){
list = map.get(currentTimeWindow)
for(Entry e: list) {
//do some processing
}
}
}
ArrayList is not thread safe.. You must use vector List to make it thread safe.
You can also use Thread safe Array list by using Collections Api but I would recommend vector list since it already provides you what you want.
//Use Collecions.synzhonizedList method
List list = Collections.synchronizedList(new ArrayList());
...
//If you wanna use iterator on the synchronized list, use it
//like this. It should be in synchronized block.
synchronized (list) {
Iterator iterator = list.iterator();
while (iterator.hasNext())
...
iterator.next();
...
}
I would recommend you to through this:
http://beginnersbook.com/2013/12/difference-between-arraylist-and-vector-in-java/
I am trying to utilize LoadingCache from the Guava library to cache a LinkedList.
LoadingCache<Integer, LinkedList<String>> cache;
I've setup a CacheLoader to handle misses, which is working fine. However there is another system that needs to submit updates to existing cache entries. Each update needs to be appended to the LinkedList and will arrive at a fairly quick rate (thousands per minute). Finally, it needs to be thread safe.
Here is a naive approach that illustrates the logic but is not thread safe:
public void add(Integer key, String value) {
LinkedList<String> list = cache.get(key);
list.add(value);
cache.put(key, list);
}
Any advice on how to make this work? I can look at other libraries but Guava 14 is already a dependency of this codebase and would be very convenient.
The last line in
public void add(Integer key, String value) {
LinkedList<String> list = cache.get(key);
list.add(value);
cache.put(key, list);
}
is not needed as you already modify the object obtained from the cache. Maybe all you need is
public void add(Integer key, String value) {
LinkedList<String> list = cache.get(key);
synchronized (list) {
list.add(value);
}
}
It depends on what eviction happens. If there's no eviction at all, then it will work. If an entry can get evicted before the updating method finishes, then you're out of luck.
Nonetheless, there's a simple solution: Using a global lock would work, but obviously inefficiently. So use a list of locks:
private static final CONCURRENCY_LEVEL = 64; // must be power of two
List<Object> locks = Lists.newArrayList(); // an array would do as well
for (int i=0; i<CONCURRENCY_LEVEL; ++i) locks.add(new Object());
public void add(Integer key, String value) {
synchronized (locks.get(hash(key))) {
cache.get(key).add(value);
}
}
where hash - depending on the distribution of your keys - can be as simple as key.intValue() & (CONCURRENCY_LEVEL-1) or something like here what sort of randomizes the distribution.
While my above list of locks should work, there's Striped.lock(int) in Guava, which makes it a bit simpler and takes care of padding (see false sharing for what it's good for) and whatever.
Most probably you should not use LinkedList as it's nearly always slower than ArrayList.
This question already has answers here:
Map.clear() vs new Map : Which one will be better? [duplicate]
(7 answers)
Fastest way to recreate the ArrayList in a for loop
(4 answers)
Closed 8 years ago.
Using Java, I have a Map interface which has some items. I want to clear all the data in it to use it again. Which method is more efficient?
params.clear()
or
params = new HashMap();
I would prefer clear() because you can have the Map as final member.
class Foo {
private final Map<String, String> map = new HashMap<String, String>();
void add(String string) {
map.put(string, "a value");
}
void clear() {
map.clear();
}
}
If you assign a new Map every time you can run into multithreading issues.
Below is an almost threadsafe example for using a Map wrapped in Collections.synchronizedMap but it assigns a new map every time you clear it.
class MapPrinter {
private static Map<String, String> createNewMap() {
return Collections.synchronizedMap(new HashMap<String, String>());
}
private Map<String, String> map = createNewMap();
void add(String key, String value) {
// put is atomic due to synchronizedMap
map.put(key, value);
}
void printKeys() {
// to iterate, we need to synchronize on the map
synchronized (map) {
for (String key : map.values()) {
System.out.println("Key:" + key);
}
}
}
void clear() {
// hmmm.. this does not look right
synchronized(map) {
map = createNewMap();
}
}
}
The clear method is responsible for a big problem: synchonized(map) will no longer work as intended since the map object can change and now two threads can simultanously be within those synchronized blocks since they don't lock the same object. To make that actually threadsafe we would either have to synchronize completely externally (and .synchronizedMap would be useless) or we could simply make it final and use Map.clear().
void clear() {
// atomic via synchronizedMap
map.clear();
}
Other advantages of a final Map (or anything final)
No extra logic to check for null or to create a new one. The overhead in code you may have to write to change the map can be quite a lot.
No accidential forgetting to assign a Map
"Effective Java #13: Favor Immutability" - while the map is mutable, our reference is not.
In general:
if you don't know how clear() is implemented, you can't guess which one would be more performant. I can come up with synthetic use-cases where one or another would definitely win.
If your map does not hold millions and millions or records you can go either way. Performance would be the same.
Specifically:
HashMap clears by wiping content of the inner array. Making old map content available for GC immediately. When you create a new Hashmap it also makes old map content available for GC + the HashMap object itself. You are trading a few CPU cycles for slightly less memory to GC
You need to consider other issue:
Do you pass this reference to some other code/component? You might want to use clear() so that this other code sees your changes, reverse is also true
Do you want no-hassle, no side-effect new map? I'd go with creating a new one.
etc
I want to remove elements from HashTable,
I use hashTable.remove() for this but not getting
Hashtable<Integer,String> players = new Hashtable<Integer,String>();
players.put(1, "Sachin Tendulkar");
players.put(2, "Rahul Dravid");
players.put(3, "Virat Kohli");
players.put(4, "Rohit Sharma");
Enumeration<String> enumration = players.elements();
while(enumration.hasMoreElements()){
String elmnt = enumration.nextElement();
System.out.println(elmnt);
if(elmnt.startsWith("R")){
players.remove(elmnt);
System.out.println(elmnt+"::Contains R");
}
}
System.out.println(players);
The output that i get is:
Rohit Sharma
Rohit Sharma::Contains R
Virat Kohli
Rahul Dravid
Rahul Dravid::Contains R
Sachin Tendulkar
{4=Rohit Sharma, 3=Virat Kohli, 2=Rahul Dravid, 1=Sachin Tendulkar}
That's because you're trying to remove the value, but the remove() method expects a key.
When you're calling player.elements() you get an enumeration of the values, not the keys. The remove() method works on a key, not a value.
From HashTable remove() documentation
public V remove(Object key):
Removes the key (and its corresponding value) from this hashtable. This method does nothing if the key is not in the hashtable.
Also, from the same documentation, here's a brief explanation on why it is not a good idea to use Enumeration to remove elements:
Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future. The Enumerations returned by Hashtable's keys and elements methods are not fail-fast.
What I would suggest you to do is to iterate over the entries, not the values.
for (Entry<Integer, String> entry: players.entrySet()){
String value = entry.getValue();
System.out.println(value);
if(value.startsWith("R")){
players.remove(entry.getKey());
System.out.println(value+"::Contains R");
}
}
This way you have both the key and the value in each iteration. You can examine the value (which is the player's name) and then use the key in order to manipulate your table.
for(Iterator<Map.Entry<Integer, String>> iter = players.entrySet().iterator(); iter.hasNext(); ) {
Map.Entry<Integer, String> e = iter.next();
if(e.getValue().startsWith("R")) {
iter.remove();
}
}
You need to pass key as argument in hashTable.remove() to remove from HashTable.
Reference : Remove from HashTable
Best way to remove an entry from a hash table.
This may be helpful to you, have a look at it.
Don't remove from enumerations.
Javadocs:
"Thus, in the face of concurrent modification, the iterator fails
quickly and cleanly, rather than risking arbitrary, non-deterministic
behavior at an undetermined time in the future. The Enumerations
returned by Hashtable's keys and elements methods are not fail-fast."
You should using iterator:
import java.util.Hashtable;
import java.util.Iterator;
public class Test {
public static void main(String[] args) {
Hashtable<Integer,String> players = new Hashtable<Integer,String>();
players.put(1, "Sachin Tendulkar");
players.put(2, "Rahul Dravid");
players.put(3, "Virat Kohli");
players.put(4, "Rohit Sharma");
Iterator<Integer> iterators = players.keySet().iterator();
while(iterators.hasNext()) {
int key = iterators.next();
if(players.get(key).startsWith("R")) {
iterators.remove();
}
}
System.out.println(players);
}
}
please add key to remove() method rather than value.
Hashtable<String, Integer> h = new Hashtable<String, Integer>();
h.put("n1", 30);
h.put("n2", 30);
h.remove("n1"); //here we are giving "n1" which is key of the hashTable
players.remove(elmnt); method try to remove the entry of elmt key. But you passed value of Hashtable instead of key.
Use players.keySet() to get the keys in place of players.elements().
Here is your syntax for coding for removing an element from hash tables:-
public virtual void Remove(
Object key
)
A mutual hash table can be manipulated with hash ref and hash remove. Concurrently by multiple threads and the operations are protected by a table-specific semaphore as needed.
Whereas, Deleting a record must not hinder later searches. In other words, the search process must still pass through the newly emptied slot to reach records whose probe sequence passed through this slot.
I have a Android Application whose core component is a HashMap<String,float[]>.
The System is having high concurrency. e.g
here are the following three situations I have which occur frequently and they are highly overlapping in nature
Iterate through all the keys in the hashmap and do some operation on
its value(read only operations).
Add new key,value pairs in the Hashmap.
Remove Certain keys from the Hashmap.
I do all these operations in different threads and thus am using a ConcurrentHashMap since some inconsistency in retrievals doesnt matter. e.g While iterating the map,if new entries are added then it doesnt matter to not read in those new values immediately as I ensure that next time they are read .
Also while removing the entries I am recreating the iterator everytime to avoid "ConcurrentModificationException"
Suppose , there is a following hashmap(i.e ConcurrentHashmap)
ConcurrentHashMap<String,float[]> test=new ConcurrentHashMap<String, float[]>(200);
Now for Retrieval I do the following
Iterator<String> reader=test.keySet().iterator();
while(reader.hasNext())
{
String s=reader.next();
float[] temp=test.get(s);
//do some operation with float[] temp here(read only operation)
}
and for removal I do the following
boolean temp = true;
while (temp) {
for (String key : test.keySet()) {
temp = false;
if (key.contains("abc")) {
test.remove(key);
temp = true;
break;
}
}
}
and when inserting in new values I simply do
test.put("temp value", new float[10]);
I am not sure if its a very efficient utilisation. Also it does matter not to read in removed values(however I need efficiency ,and since the iterator is again created during the function call,its guaranteed that in the next time I don't get the removed values)so that much inconsistency can be tolerated?
Could someone please tell me an efficient way to do it?
PS. I forgot to mention that why I am doing the removal operation in such a way.
I have now changes the condition on which its deleted from equal to contains(there might be multiple stings having the prefix "abc" followed by different suffixes. so I need to delete all those then.
Iterate through all the keys in the hashmap and do some operation on its value(read only operations).
Don't iterate over the key set to then retrieve the values too - iterate over the entry set directly:
for (Map.Entry<String, float[]> e : map.entrySet() {
String key = e.getKey();
float[] value = e.getValue();
//do something with them
}
This is more efficient in general (even for "normal" HashMaps), but it will also reduce contention in your case (half as many accesses to the map).
Add new key,value pairs in the Hashmap.
Yes it is simply: map.put(s, f);
Remove Certain keys from the Hashmap.
If you need to check that the key contains a given substring then you do need to iterate over the keys like you are doing, although I'm not sure why you have a while+for+break instead of a simple for.
Because of the way you are using the ConcurrentHashMap, you are precisely removing its Concurrent characteristics. Your attempts at (re-)synchronization will work very frequently, but not always.
Have you considered to leave the keys in the HashMap? I'm thinking of something like:
public static final float[] DELETED= new float[0] ;
/* delete */
test.put(key,DELETED);
/* insert */
test.put(key,value);
/* iterate */
for(Map.Entry<String,float[]> e: test.entrySet ) {
if( e.getValue() != DELETED ) {
operateOn(e);
}
}
If keys are too volatile (i.e. after a while you would have too many DELETED items), then you can create a cleanup Thread.
According to ConcurrentHashMap API its iterators never throw ConcurrentModificationException so you dont need to break after removal. But in any case the correct way to iterate and remove is this
for (Iterator<String> i = test.keySet().iterator(); i.hasNext();) {
String next = i.next();
if (next.equals("abc")) {
i.remove();
}
}
this way it will work even with fail-fast iterators without ConcurrentModificationException