I am unsure how to do this, I'd like to iterate the ConcurrentLinkedQueue (all of it), removing the i-th item and performing some code on it.
This is what I was used to do:
public static class Input {
public static final ConcurrentLinkedQueue<TreeNode> treeNodes = new ConcurrentLinkedQueue<>();
}
public static class Current {
public static final ConcurrentHashMap<Integer, TreeNode> treeNodes = new ConcurrentHashMap<>();
}
TreeNode is a simple class
TreeNode treeNode = Input.treeNodes.poll();
while (treeNode != null) {
treeNode.init(gl3);
Current.treeNodes.put(treeNode.getId(), treeNode);
treeNode = Input.treeNodes.poll();
}
This is how I am trying to do using stream:
Input.treeNodes.stream()
.forEach(treeNode -> {
Input.treeNodes.remove(treeNode);
treeNode.init(gl3);
Current.treeNodes.put(treeNode.getId(), treeNode);
});
I am afraid that something may be error prone having to remove the item inside the forEach action.
So my question is:
Is this safe and/or are there any better ways to do it?
Just as you've assumed, you should not modify the backing collection while processing the stream because you might get a ConcurrentModificationException (just as with for(Object o:objectArray){} loops)
On the other hand it is not very clear which TreeNode you are trying to remove, as in the current case, seemingly you wish to remove all elements from the List, perform some actions on them and put them in a Map.
You may safely achieve your current logic via:
Input.treeNodes.stream()
.map(treeNode -> {
treeNode.init(gl3);
Current.treeNodes.put(treeNode.getId(), treeNode);
});
Input.treeNodes.clear();
This behavior is determine by the Spliterator used to construct the Stream. The documentation of ConcurrentLinkedQueue.spliterator() says:
Returns a Spliterator over the elements in this queue.
The returned spliterator is weakly consistent.
“weakly consistent” implies:
Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators and Spliterators provide weakly consistent rather than fast-fail traversal:
they may proceed concurrently with other operations
they will never throw ConcurrentModificationException
they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
This implies that removing the encountered elements should not have any impact.
On the other hand, when other threads add or remove elements, the outcome of your Stream operation regarding these elements is unpredictable.
However, you should consider that remove(Object) is not the intended use case for a queue.
Related
I am relying on list iterators to move through a list of characters. This is a single-threaded program and I use listIterator objects sequentially in 4 different methods. Each method has the same setup:
private void myMethod(ArrayList<Integer> input) {
ListIterator<Integer> i = input.listIterator();
while (i.hasNext()) {
Integer in = i.next();
if (in < 10)
i.remove();
else
i.set(in*in); // because its lucky
}
}
With this pattern, on the second iterator the following Exception is thrown:
java.util.ConcurrentModificationException
However, looking at the javadocs I don't see this Exception in the Exceptions thrown nor do I see a method to close the iterator after I am done. Am I using the listIterator incorrectly? I have to iterate over the same ArrayList multiple times, each time conditionally removing or mutating each element. Maybe there is a better way to iterate over the ArrayList and this use-case is not best solved by a ListIterator.
java docs for ListIterator
This is explained in the ArrayList javadoc, you are modifying the list with remove() and set() while using an Iterator:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
It’s hard to give diagnostic for a problem when the shown code clearly isn’t the code that produced the exception, as it doesn’t even compile. The remove method of Iterator doesn’t take arguments and the set method is defined on ListIterator, but your code declares the variable i only as Iterator.
A fixed version
private void myMethod(ArrayList<Integer> input) {
ListIterator<Integer> i = input.listIterator();
while (i.hasNext()) {
Integer in = i.next();
if (in < 10)
i.remove();
else
i.set(in*in);
}
}
would run without problems. The answer to your general question is that each modification invalidates all existing iterators, except the one used to make the modification when you did use an iterator for the modification and not the collection interface directly.
But in your code, there is only one iterator, which is only created and used for this one operation. As long as there is no overlapping use of iterators to the same collection, there is no problem with the invalidation. Iterators existing from previous operations are abandoned anyway and the iterators used in subsequent operations do not exist yet.
Still, it’s easier to use
private void myMethod(ArrayList<Integer> input) {
input.removeIf(in -> in < 10);
input.replaceAll(in -> in*in);
}
instead. Unlike the original code, this does two iterations, but as explained in this answer, removeIf will be actually faster than iterator based removal in those cases, where performance really matters.
But still, the problem persists. The shown code can’t cause a ConcurrentModificationException, so your actual problem is somewhere else and may still be present, regardless of how this one method has been implemented.
I am not knowledgable enough about Java ListIterators to answer the question but it appears I have run into the XY problem here. The problem seems to be better solved with Java Streams to remove the element or map the element into a new ArrayList by exercising a function on each element in the original ArrayList.
private ArrayList<Integer> myMethod(ArrayList<Integer> input) {
ArrayList<Integer> results = input.stream().filter(
in -> (in < 10)).collect(Collectors.toCollection(ArrayList::new));
results = input.stream().map(
in -> in*in).collect(Collectors.toCollection(ArrayList::new));
return results;
}
having a map holds list of eventlisteners for same event by type as key,
func_1() will start to get the listenerlist of one type from the map and iterate the list to handle the event with every listener.
When one listener finishes its handling, it will ask to remove it from the listenerlist in the map.
since the listeners are in an iterator, and removing it from the original list will cause java.util.ConcurrentModificationException in the iterator.previous() for getting next listener.
question is if using CopyOnWriteArrayList to copy the listener list then iterator on it, since it is a copy of the list, will it still throw when the listener is removed from other thread?
does it make any difference just simply making a copy of normal list instead of CopyOnWriteArrayList to iterator on?
func_1(Event event) {
List<WeakReference<EventListener<Event>>> listenerlist = mEventMap.get(event.eventType);
/* instead of directly iterator on the listenerlist
ListIterator<WeakReference<EventListener<Event>>> listenerIterator =
listenerlist.listIterator(listenerlist.size());
but making a CopyOnWriteArrayList first:
*/
List<WeakReference<EventListener<Event>>> listeners =
new CopyOnWriteArrayList<>(listenerlist);
ListIterator<WeakReference<EventListener<Event>>> listenerIterator =
listeners.listIterator(listeners.size());
while(listenerIterator.hasPrevious()){
WeakReference<EventListener<Event>> listenerItem =
listenerIterator.previous();
//doing something
listenerItem.func_2(event);
}
}
EventListener::func_2(Event event){
//do something
//remove the type in the map
funct_3(EventListener.this);
}
funct_3(EventListener listener) {
List<WeakReference<EventListener<Event>>> listeners =
mEventMap.get(listener.eventType);
if (listeners != null) {
Iterator<WeakReference<EventListener<Event>>> listenerIterator =
listeners.iterator();
while (listenerIterator.hasNext()) {
WeakReference<EventListener<Event>> listenerItem = listenerIterator.next();
if (listenerItem.get() != null && listenerItem.get() == listener) {
listenerIterator.remove();
break;
}
}
}
}
Did the test and it does not throw because it is iterating on a copy of the list, while the removing happens on the original list.
The draw back is it might be costly if the event comes too often.
-https://www.ibm.com/developerworks/library/j-5things4/
"2. CopyOnWriteArrayList
Making a fresh copy of an array is too expensive an operation, in terms of both time and memory overhead, to consider for ordinary use; developers often resort to using a synchronized ArrayList instead. That's also a costly option, however, because every time you iterate across the contents of the collection, you have to synchronize all operations, including read and write, to ensure consistency.
This puts the cost structure backward for scenarios where numerous readers are reading the ArrayList but few are modifying it.
CopyOnWriteArrayList is the amazing little jewel that solves this problem. Its Javadoc defines CopyOnWriteArrayList as a "thread-safe variant of ArrayList in which all mutative operations (add, set, and so on) are implemented by making a fresh copy of the array."
The collection internally copies its contents over to a new array upon any modification, so readers accessing the contents of the array incur no synchronization costs (because they're never operating on mutable data).
Essentially, CopyOnWriteArrayList is ideal for the exact scenario where ArrayList fails us: read-often, write-rarely collections such as the Listeners for a JavaBean event."
I have this code:
private ConcurrentMap<String, Integer> myMap = new ConcurrentHashMap<>();
#Scheduled(fixedDelay = 600_000)
public void foo(){
myMap.values().stream().
filter(predicate()).
forEach(this::remove);
}
public void insert(String str, Integer value){
myMap.put(str, value);
}
What would happen if while iterating over this map - someone will put a new value in it or remove an existing value from it?
The documentation for ConcurrentHashMap has some details about the behavior. First we look at what ConcurrentHashMap.values() does:
Returns a Collection view of the values contained in this map...
The view's iterators and spliterators are weakly consistent.
The view's spliterator reports Spliterator.CONCURRENT and Spliterator.NONNULL.
What's interesting are the terms "weakly consistent" and Spliterator.CONCURRENT, where the former is described as:
Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators and Spliterators provide weakly consistent rather than fast-fail traversal:
they may proceed concurrently with other operations
they will never throw ConcurrentModificationException
they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
and Spliterator.CONCURRENT is described as:
Characteristic value signifying that the element source may be safely concurrently modified (allowing additions, replacements, and/or removals) by multiple threads without external synchronization. If so, the Spliterator is expected to have a documented policy concerning the impact of modifications during traversal.
From all these documentations, and being consistent with the concurrency model of the ConcurrentHashMap, it means that the stream pipeline is completely thread-safe and will traverse the elements as they existed upon the creation of the iterator.
I want to return an unmodifiable view of the class (that maintain a collection of items ) to outside clients .
So to protect concurrent access, I need to wrap the collection in a synchronized wrapper first, then put an unmodifiable wrapper around the version I return to outside threads.
So I wrote the following code and unfortunately it is throwing a ConcurrentModificationException.
.
import java.util.*;
public class Test {
public static void main(String[] args) {
// assume c1 is private, nicely encapsulated in some class
final Collection col1 = Collections.synchronizedCollection(new ArrayList());
// this unmodifiable version is public
final Collection unmodcol1 = Collections.unmodifiableCollection(col1);
col1.add("a");
col1.add("b");
new Thread(new Runnable() {
public void run() {
while (true) {
// no way to synchronize on c1!
for (Iterator it = unmodcol1 .iterator(); it.hasNext(); it.next())
;
}
}
}).start();
while (true) {
col1 .add("c");
col1 .remove("c");
}
}
}
So my question is How to synchronize unmodifiable collections ?
To add more
When a client who received the collection wants to iterate over its elements
1) it doesn't necessarily know that it's a synchronized collection and
2) even if it does, it can't correctly synchronize on the synchronization wrapper mutex to iterate over its elements. The penalty, as described in
Collections.synchronizedCollection, is non-deterministic behaviour.
From my understanding Putting an unmodifiable wrapper on a synchronized collection leaves no access
to the mutex that must be held to iterate correctly.
If you can ensure that read-only clients of the collection synchronize on the collection, synchronize on that same view in your producer:
/* In the producer... */
Collection<Object> collection = new ArrayList<>();
Collection<Object> tmp = Collections.unmodifiableCollection(collection);
Collection<Object> view = Collections.synchronizedCollection(tmp);
synchronized (view) {
collection.add("a");
collection.add("b");
}
/* Give clients access only to "view" ... */
/* Meanwhile, in the client: */
synchronized (view) {
for (Object o : view) {
/* Do something with o */
}
}
You need to decide on a few things first.
A. Are users of the returned collection supposed to automatically see updates to it, and when? If so you would need to take care not to (or decide if this is ok) accidently locking it for updates for periods of time. If using synchronized and synchronizing on the returned collection you are effectively allowing the user of the returned collection to lock it for updates for example.
B. Or should they need to call again to get a fresh collection?
Besides, using Collections.synchronizedX won't give you any protection against iterating over it, just individual read and writes. So would require the client to guarantee that it locks during all explicit and implicit iterations. Sounds bad in general, but depends I guess.
Possible solutions:
Return a copy, don't need to wrap it in unmodifiable even. Just lock it while creating it. synchronized (collection) { return new ArrayList(collection); } No further synchronization needed. An example implementation of Option B above.
Like 1 but automatically by the data structure itself, use CopyOnWriteArrayList and return it (wrapped in unmodifiable). Note: This means writes to the collection are expensive. Reads are not. On the other hand even iterating on it is thread safe. No synchronization whatsoever needed. Supports option A above.
Depending on the properties of the data structure you need you could go for a non RandomAccess list like ConcurrentLinkedQueue or ConcurrentLinkedDeque, both allow iterating etc over the data structure without any extra synchronization. Again, wrapped in unmodifiable. Supports option A above.
I would go for option B-1 for the general case and to get started. But it depends as usual.
You're asking "How to synchronize unmodifiable collections", but actually that's not what you did in your code. You made a syncronized collection unmodifiable. If you 1st make your collection unmodifiable, and then syncronize it, then you'll get what you want.
// you'll need to create the list
final ArrayList list = new ArrayList();
// and add items to it while it's still modifiable:
list.add("a");
list.add("b");
final Collection unmodcol1 = Collections.unmodifiableCollection(list);
final Collection col1 = Collections.synchronizedCollection(unmodcol1);
However the add, remove inside the while will still fail for the same reason.
On the other hand if you created your list and made it unmodifiable, then you might not need to syncronize it at all.
You may want to use one of concurrent collections. That will give you what you need, if I read your question correctly. Just accept to pay on cost.
List<T> myCollection = new CopyOnWriteArrayList<T>();
List<T> clientsCollection = Collections.unmodifiableList(myCollection);
this way, you will not get CME, as client will always get unmodifiable collection and not interfere with your writes. However, price is rather high.
I'm looking for an implementation of java.util.Set with the following features:
Should be concurrent by no means of synchronized locking; So it's obvious that I don't want to use Collections.synchronizedSet().
Should keep insertion order. So ConcurrentSkipListSet is not preferable, because it uses compareTo() as equals(), and requires to provide either an implementation of Comparable or Comparator. There is also a ConcurrentLinkedHashMap which in spite of LinkedHashMap, doesn't keep insertion order.
Should be unbounded.
Recommended be a FIFO linked list, as my operations are done only to the first element of the queue.
As far as I could find the only proper impl is CopyOnWriteArraySet, but it states in the documentation that:
Mutative operations (add, set, remove,
etc.) are expensive since they usually entail copying the entire
underlying array.
In my case, I have lots of insertions to the end of queue (set) and lots Any deletions (and read) from head of the queue. So, any recommendation?
The following solution has a race condition on removal. It also behaves somewhat differently from standard JDK Set implementations.
However, it uses standard JDK objects, and is a simple implementation. Only you can decide whether this race condition is acceptable, or whether you're willing to invest the timee to find/implement a solution without races.
public class FifoSet<T>
{
private ConcurrentHashMap<T,T> _map;
private ConcurrentLinkedQueue<T> _queue;
public void add(T obj)
{
if (_map.put(obj,obj) != null)
return;
_queue.add(obj);
}
public T removeFirst()
{
T obj = _queue.remove();
_map.remove(obj);
return obj;
}
}
Some more explanation: the ConcurrentHashMap exists solely as a guard on the ConcurrentLinkedList; its put() method is essentially a compare-and-swap. So you ensure that you don't have anything in the map before adding to the queue, and you don't remove from the map until you remove from the queue.
The race condition on remove is that there's a space of time between removing the item from the queue and removing it from the map. In that space of time, add will fail, because it still thinks the item is in the queue.
This is imo a relatively minor race condition. One that's far less important than the gap in time between removing the item from the queue and actually doing something with that item.