We Keep Coding

thread-safe CopyOnWriteArrayList reverse iteration

Consider the following code snippet:
private List<Listener<E>> listenerList = new CopyOnWriteArrayList<Listener<E>>();
public void addListener(Listener<E> listener) {
if (listener != null) {
listenerList.add(listener);
}
}
public void removeListener(Listener<E> listener) {
if (listener != null) {
listenerList.remove(listener);
}
}
protected final void fireChangedForward(Event<E> event) {
for (Listener<E> listener : listenerList) {
listener.changed(event);
}
}
protected final void fireChangedReversed(Event<E> event) {
final ListIterator<Listener<E>> li = listenerList.listIterator(listenerList.size());
while (li.hasPrevious()) {
li.previous().changed(event);
}
}
There is a listener list that can be modified and iterated.
I think the forward iteration (see method #fireChangedForward)
should be safe.
The question is: is the reverse iteration (see method #fireChangedReversed) also safe in a multi-threaded environment?
I doubt that, because there are two calls involved: #size and #listIterator.
If it's not thread-safe, what is the most efficient way to implement #fireChangedReversed under the following circumstances:
optimize for traversal
avoid usage of locking if possible
avoid usage of javax.swing.event.EventListenerList
prefer solution without usage of third-party lib, e.g. implementation in own code possible

Indeed, listenerList.listIterator(listenerList.size()) is not thread-safe, for exactly the reason you suggested: the list could change size between the calls to size() and listIterator(), resulting in either the omission of an element from the iteration, or IndexOutOfBoundsException being thrown.
The best way to deal with this is to clone the CopyOnWriteArrayList before getting the iterator:
CopyOnWriteArrayList<Listener<E>> listenerList = ... ;
#SuppressWarnings("unchecked")
List<Listener<E>> copy = (List<Listener<E>>)listenerList.clone();
ListIterator<Listener<E>> li = copy.listIterator(copy.size());
The clone makes a shallow copy of the list. In particular, the clone shares the internal array with the original. This isn't entirely obvious from the specification, which says merely
Returns a shallow copy of this list. (The elements themselves are not copied.)
(When I read this, I thought "Of course the elements aren't copied; this is a shallow copy!" What this really means is that neither the elements nor the array that contains them are copied.)
This is fairly inconvenient, including the lack of a covariant override of clone(), requiring an unchecked cast.
Some potential enhancements are discussed in JDK-6821196 and JDK-8149509. The former bug also links to a discussion of this issue on the concurrency-interest mailing list.

One simple way to do that is to call #toArray method and iterate over the array in reverse order.

You could always just get a ListIterator and "fast-forward" to the end of the list as such:
final ListIterator<Listener<E>> li = listenerList.listIterator();
if (li.hasNext()) {
do{
li.next();
} while (li.hasNext());
}
while (li.hasPrevious()) {
li.previous().changed(event);
}
EDIT I switched the quirky exception-handling of my previous answer for a do/while loop that places the cursor of the ListIterator after the last element, in order to be ready for the next previous call.
RE-EDIT As pointed out by #MikeFHay, a do/while loop on an iterator will throw a NoSuchElementException on an empty list. To prevent this from happening, I wrapped the do/while loop with if (li.hasNext()).

Exception creating TreeSet from concurrently-modified ConcurrentSkipListSet

Generally, concurrent collections are safe to iterate; according to Javadoc: 'Iterators are weakly consistent, returning elements reflecting the state of the set at some point at or since the creation of the iterator. They do not throw ConcurrentModificationException, and may proceed concurrently with other operations.'
However, consider this:
import java.util.Random;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentSkipListSet;
public class ConcurrencyProblem {
private static volatile boolean modifierIsAlive = true;
public static void main(String[] args) {
final ConcurrentSkipListSet<Integer> concurrentSet = new ConcurrentSkipListSet<>();
Thread modifier = new Thread() {
private final Random randomGenerator = new Random();
public void run() {
while (modifierIsAlive) {
concurrentSet.add(randomGenerator.nextInt(1000));
concurrentSet.remove(randomGenerator.nextInt(1000));
}
};
};
modifier.start();
int sum = 0;
while (modifierIsAlive) {
try {
TreeSet<Integer> sortedCopy = new TreeSet<>(concurrentSet);
// make sure the copy operation is not eliminated by the compiler
sum += sortedCopy.size();
} catch (RuntimeException rte) {
modifierIsAlive = false;
rte.printStackTrace();
}
}
System.out.println("Dummy output: " + sum);
}
}
The output is
java.util.NoSuchElementException
at java.util.concurrent.ConcurrentSkipListMap$Iter.advance(ConcurrentSkipListMap.java:2299)
at java.util.concurrent.ConcurrentSkipListMap$KeyIterator.next(ConcurrentSkipListMap.java:2334)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2559)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2547)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2579)
at java.util.TreeMap.buildFromSorted(TreeMap.java:2504)
at java.util.TreeMap.addAllForTreeSet(TreeMap.java:2462)
at java.util.TreeSet.addAll(TreeSet.java:308)
at java.util.TreeSet.<init>(TreeSet.java:172)
at mtbug.ConcurrencyProblem.main(ConcurrencyProblem.java:27)
Dummy output: 44910
I'm wondering if this is a bug or a feature; we did not get a ConcurrentModificationException, but still, having to care about iteration (falling back to synchronized blocks or otherwise) kind of defeats the purpose of ConcurrentSkipListSet/Map. I've been able to reproduce this both with Java 7 and 8 (currently, 8u72 on my Linux box).

As far as I can understand from browsing the sources, the problem with TreeSet is that it calls size() before iterating and then uses it instead of calling hasNext(). This may be a bug, but I think it's just a consequence of red-black trees being complicated structures requiring careful balancing, and therefore knowing size in advance is needed to properly balance it in linear time during creation.
You may circumvent this by iterating manually and adding elements to the TreeSet, but this will lead to n log n complexity, which could be the reason why TreeSet's constructor doesn't do it that way (its API spec guarantees linear time). Of course it could still call hasNext() as it builds the tree, but then some additional actions may be required after the construction is finished to rebalance the tree, which could lead to amortized linear complexity. But red-black trees are a mess as they are, and that kind of hack would make the implementation even messier.
Still, I think it's very confusing and should probably be documented somewhere in the API docs, but I'm not sure where exactly. Probably in the part where they explain what weakly consistent iterators are. Specifically, it should be mentioned that some library classes rely on the returned size and therefore may throw NoSuchElementException. Mentioning specific classes would also help.

I'm actually starting to lean towards this being a bug in TreeSet/TreeMap (update, it is). The issue, as Sergey alludes, is that TreeMap caches the result of ConcurrentSkipListSet.size() before reading its elements.
TreeSet.addAll() calls
TreeMap.addAllForTreeSet() and passes the collection's current size and potentially concurrent Iterator to
TreeMap.buildFromSorted() which ultimately calls Iterator.next() size-times.
In other words, it assumes the Collection it is passed will not be modified during construction, which is an erroneous assumption.
Note that even if buildFromSorted() did call Iterator.hasNext() its only option at that point would be to fail, since the backing data structure was modified mid-construction.
Looking at other collections that could potentially have some issue copying concurrent structures, including ArrayList, LinkedList, and CopyOnWriteArrayList (most other collections I looked at simply for-each over the elements), explicitly copy the provided collection to an array before doing any actual work in order to avoid this exact issue. I think TreeSet and TreeMap should be doing the same thing.
We actually don't have to accept O(n log n) performance due to this bug, but it's going to be a hack. We can't simply copy the values into an array or other data structure, because then inserting into the TreeSet won't be linear time. But we can lie to TreeSet by claiming the copy is a SortedSet.
public static class IterateOnlySortedSet<E>
extends AbstractSet<E> implements SortedSet<E> {
private final ArrayList<E> elements;
private final Comparator<? super E> comparator;
public IterateOnlySortedSet(SortedSet<E> source) {
elements = new ArrayList<>(source);
comparator = source.comparator();
}
#Override
public Iterator<E> iterator() {
return elements.iterator();
}
#Override
public int size() {
return elements.size();
}
#Override
public Comparator<? super E> comparator() {
return comparator;
}
// remaining methods simply throw UnsupportedOperationException
}
Changing your TreeSet construction line to:
TreeSet<Integer> sortedCopy = new TreeSet<>(new IterateOnlySortedSet<>(concurrentSet));
Now succeeds.
Nice find :)

ConcurrentModificationException, help me understand something

I am not quite sure why i am getting an exception in this method to begin with and why at this location? (No modification happens there)
Iterator<EventGroup> groupIterator = eventGroups.iterator();
while (groupIterator.hasNext()) {
EventGroup eventGroup = groupIterator.next();
Iterator<ScheduledEvent> eventIterator = eventGroup.getEvents().iterator();
while (eventIterator.hasNext()) {
ScheduledEvent event = eventIterator.next(); // <------------- ConcurrentModificationException
boolean found = ((SomeObject) event).getSomeId() == someId;
if (found) {
unschedule++;
unscheduleEvent(event);
eventGroup.remove(event);
if (eventGroup.isEmpty()) {
eventGroups.remove(eventGroup);
}
}
}
}
It seems that the only thing being modified here is eventGroup, but i am using an iterator to walk over it.
Please help me understand what's happening here?

eventIterator is an iterator on eventGroup and you call eventGroup.remove(event); while iterating. You should use the iterator instead to remove the item:
eventIterator.remove();
This behaviour is explained in the javadoc of ArrayList:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException.
Note: you have the same issue with eventsGroup and you should replace eventGroups.remove(eventGroup); with groupIterator.remove();.

Iterator retrieve first value and place it back on the same iterator

I have the following scenario: I have an existing iterator Iterator<String> it and I iterate over its head (say first k elements, which are flagged elements, i.e. they start with '*' ). The only way to know that the flagged elements are over, is by noticing that the (k+1)th element is not flagged.
The problem is that if I do that, the iterator it will not provide me the first value anymore on the next call to next().
I want to pass this iterator to a method as it's only argument and I would like to avoid changing its signarture and it implementation. I know I could do this:
public void methodAcceptingIterator(Iterator<String> it) //current signature
//change it to
public void methodAcceptingIterator(String firstElement, Iterator<String> it)
But this looks like a workarround/hack decreasing the elegance and generality of the code, so I don't want to this.
Any ideas how I could solve this problem ?

You could use Guava's PeekingIterator (link contains the javadoc for a static method which, given an Iterator, will return a wrapping PeekingIterator). That includes a method T peek() which shows you the next element without advancing to it.

The solution is to create your own Iterator implementation which stores the firstElement and uses the existing iterator as an underlying Iterator to delegate the requests for the rest of the elements to.
Something like:
public class IteratorMissingFirst<E> implements Iterator<E>{
private Iterator<E> underlyingIterator;
private E firstElement;
private boolean firstElOffered;
public IteratorMissingFirst(E firstElement, Iterator<E> it){
//initialize all the instance vars
}
public boolean hasNext(){
if(!firstElOffered && firstElement != null){
return true;
}
else{
return underlyingIterator.hasNext();
}
}
public E next(){
if(!firstElOffered){
firstElOffered = true;
return firstElement;
}
else return underlyingIterator.next();
}
public void remove(){
}
}

Why don't you just have methodAcceptingIterator store the first element it gets out of the iterator in a variable? Or -- in a pinch -- just copy the contents of the Iterator into an ArrayList at the beginning of your method; now you can revisit elements as often as you like.

With Guava, you can implement Razvan's solution in an easier way by using some methods from the Iterables class:
Iterators.concat(Iterators.singletonIterator(firstElement), it)
This gives you an iterator working similar to IteratorMissingFirst, and it's easy to extend if you need to look at more than one element in front (but it creates two objects instead of only one).

Updating a PriorityQueue when iterating it

I need to update some fixed-priority elements in a PriorityQueue based on their ID. I think it's quite a common scenario, here's an example snippet (Android 2.2):
for (Entry e : mEntries) {
if (e.getId().equals(someId)) {
e.setData(newData);
}
}
I've then made Entry "immutable" (no setter methods) so that a new Entry instance is created and returned by setData(). I modified my method into this:
for (Entry e : mEntries) {
if (e.getId().equals(someId)) {
Entry newEntry = e.setData(newData);
mEntries.remove(e);
mEntries.add(newEntry);
}
}
The code seems to work fine, but someone pointed out that modifying a queue while iterating over it is a bad idea: it may throw a ConcurrentModificationException and I'd need to add the elements I want to remove to an ArrayList and remove it later. He didn't explain why, and it looks quite an overhead to me, but I couldn't find any specific explanation on internet.
(This post is similar, but there priorities can change, which is not my case)
Can anyone clarify what's wrong with my code, how should I change it and - most of all - why?
Thanks,
Rippel
PS: Some implementation details...
PriorityQueue<Entry> mEntries = new PriorityQueue<Entry>(1, Entry.EntryComparator());
with:
public static class EntryComparator implements Comparator<Entry> {
public int compare(Entry my, Entry their) {
if (my.mPriority < their.mPriority) {
return 1;
}
else if (my.mPriority > their.mPriority) {
return -1;
}
return 0;
}
}

This code is in the Java 6 implementation of PriorityQueue:
private class Itr implements Iterator<E> {
/**
* The modCount value that the iterator believes that the backing
* Queue should have. If this expectation is violated, the iterator
* has detected concurrent modification.
*/
private int expectedModCount = modCount;
public E next() {
if(expectedModCount != modCount) {
throw new ConcurrentModificationException();
}
}
}
Now, why is this code here? If you look at the Javadoc for ConcurrentModificationException you will find that the behaviour of an iterator is undefined if modification occurs to the underlying collection before iteration completes. As such, many of the collections implement this modCount mechanism.
To fix your code
You need to ensure that you don't modify the code mid-loop. If your code is single threaded (as it appears to be) then you can simply do as your coworker suggested and copy it into a list for later inclusion. Also, the use of the Iterator.remove() method is documented to prevent ConcurrentModificationExceptions. An example:
List<Entry> toAdd = new ArrayList<Entry>();
Iterator it = mEntries.iterator();
while(it.hasNext()) {
Entry e = it.next();
if(e.getId().equals(someId)) {
Entry newEntry = e.setData(newData);
it.remove();
toAdd.add(newEntry);
}
}
mEntries.addAll(toAdd);

The Javadoc for PriorityQueue says explicitly:
"Note that this implementation is not synchronized. Multiple threads should not access a PriorityQueue instance concurrently if any of the threads modifies the list structurally. Instead, use the thread-safe PriorityBlockingQueue class."
This seems to be your case.

What's wrong in your code was already explained -- implementing iterator, which can consistently iterate through collection with intersected modification is rather hard task to do. You need to specify how to deal with removed items (will it be seen through iterator?), added items, modified items... Even if you can do it consistently it will be rather complex and unefficient implementation -- and, mostly, not very usefull, since use case "iterate without modifications" is much more common. So, java architects choose to deny modification while iterate, and most collections from Java collection API follow this, and throw ConcurrentModificationException if such modification detected.
As for your code -- for me, your just should not make items immutable. Immutability is great thing, but it should not be overused. If Entry object you use here is some kind of domain object, and you really want them to be immutable -- you can just create some kind of temporary data holder (MutableEntry) object, use it inside your algorithm, and copy data to Entry before return. From my point of view it will be best solution.

a slightly better implementation is
List<Entry> toAdd = new ArrayList<Entry>();
for (Iterator<Entry> it= mEntries.iterator();it.hasNext();) {
Entry e = it.next();
if (e.getId().equals(someId)) {
Entry newEntry = e.setData(newData);
it.remove();
toAdd.add(newEntry);
}
}
mEntries.addAll(toAdd);
this uses the remove of the iterator and a bulk add afterwards