We Keep Coding

How Concurrent modification exception is handled internally by CopyOnWriteArrayList/ConcurrentHashMap?

I want to understand internally how concurrent modification exception is handled in concurrent collections like ConcurrentHashMap and CopyOnWriteArrayList.
There are so many blogs available in internet which suggest to use these two data structures to avoid concurrent modification exception. But nothing explains , how this exception is internally handled by concurrent collection.
Can someone give more insights on this? I need some detailed explanation.

The literal answer to your question is not very interesting. ConcurrentHashMap and CopyOnWriteArrayList don't throw ConcurrentModificationException because they don't include code to throw it.
It's not like ConcurrentModificationException is some low-level intrinsic thing. ArrayList and HashMap, among other collection classes, throw ConcurrentModificationException to help you. They have to include extra code to try to detect concurrent modifications, and extra code to throw an exception. ConcurrentModificationException is thrown when one of those classes detect that there is a bug somewhere that is causing an unsafe modification to your collection.
Classes that support safe concurrent modification don't throw ConcurrentModificationException because they don't need to.
If you're trying to debug a ConcurrentModificationException, there are plenty of other questions that help answer that:
Why is a ConcurrentModificationException thrown and how to debug it
Why doesn't this code throw a ConcurrentModificationException?

Here is the add() method definition of ArrayList and CopyOnWriteArrayList.
ArrayList:
public boolean add(E e) {
ensureCapacityInternal(size + 1); // Increments modCount!!
elementData[size++] = e;
return true;
}
CopyOnWriteArrayList:
public boolean add(E e) {
final ReentrantLock lock = this.lock;
lock.lock();
try {
Object[] elements = getArray();
int len = elements.length;
Object[] newElements = Arrays.copyOf(elements, len + 1);
newElements[len] = e;
setArray(newElements);
return true;
} finally {
lock.unlock();
}
}
From the above code, it is clear that CopyOnWriteArrayList takes lock before modifying the map. Here I have just posted the code of the add method. If you look on the code of remove() / addAll() or any method which modifies the List structurally you can see that it takes lock before modifying the collection. Also ArrayList's iterator's method such as next()/remove() check for modification but for CopyOnWriteArrayList's iterator's method does not check for the modification. For example :
ArrayList iterator next() method:
#SuppressWarnings("unchecked")
public E next() {
checkForComodification();
int i = cursor;
if (i >= size)
throw new NoSuchElementException();
Object[] elementData = ArrayList.this.elementData;
if (i >= elementData.length)
throw new ConcurrentModificationException();
cursor = i + 1;
return (E) elementData[lastRet = i];
}
CopyOnWriteArrayList iterator next() method:
#SuppressWarnings("unchecked")
public E next() {
if (! hasNext())
throw new NoSuchElementException();
return (E) snapshot[cursor++];
}

This will, right now, answer how CopyOnWriteArrayList avoids the need for a ConcurrentModificationException.
When you modify the collection the CopyOnWriteArrayList does two things
It prevents other threads from modifying the collection via locking
Copies all the elements in the current CopyOnWriteArrayList into a new array and then assigns that new array to the class's array instance
So how does that prevent a CME? A CME in standard collections will only be thrown as a result of iterating. The exception gets thrown if, while iterating over the collection, an add or remove is executed on the same collection instance.
The CopyOnWriteArrayList's iterator assigns the current array as a final field snapshot of the collection and uses that for iteration. If another thread (or even the same thread) tries to add to the CopyOnWriteArrayList then updates will be applied to a new copy and not the snapshot one we are currently iterating.
For instance, we know the add method looks like
public boolean add(E e) {
final ReentrantLock lock = this.lock;
lock.lock();
try {
Object[] elements = getArray();
int len = elements.length;
Object[] newElements = Arrays.copyOf(elements, len + 1);
newElements[len] = e;
setArray(newElements);
return true;
} finally {
lock.unlock();
}
}
Notice the thread local newElements assignment being made, when that is completed it will set to the class instance volatile array.
Then comes the iterator, it's defined as
static final class COWIterator<E> implements ListIterator<E> {
/** Snapshot of the array */
private final Object[] snapshot;
/** Index of element to be returned by subsequent call to next. */
private int cursor;
So when iterating, we are reading whatever was the array prior to any modifications, and since no other thread can modify the snapshot we are looking at a ConcurrentModificationException cannot happen.

ConcurrentModificationException in iterator.next()

I have next code in background thread
private List<IStartAction> mActions = Collections.synchronizedList(new ArrayList<IStartAction>());
protected void removeNonApplicableActions() {
Iterator<IStartAction> iterator = mActions.iterator();
while (iterator.hasNext()) {
IStartAction action = iterator.next();
if (!action.isApplicable()) {
iterator.remove();
}
}
}
When i run this in main thread got ConcurrentModificationException into iterator.next().
Why is this happening? I use thread-safe collection and remove items through iterator. Collection used in only this thread.

Thread safety for a synchronized collection only applies to one method call. Between method calls the lock is released and another thread can lock the collection. If you perform two operations, anything could happen in the meantime, unless you lock it your self. e.g.
// to add two elements in a row, you must hold the lock.
synchronized(mAction) {
mAction.add(x);
// without holding the lock, anything could happen in between
mAction.add(y);
}
Similarly to iterate over a synchronized collection, you have to hold the lock, otherwise anything could happen between method calls to the iterator.
synchronized (mAction) {
for(Iterator<IStartAction> iter = mActions.iterator(); iter.hashNext();) {
IStartAction action = iter.next();
if (!action.isApplicable()) {
iter.remove();
}
}
}

Getting java.util.LinkedList$ListItr.checkForComodification exception while using iterators?

I am using the listIterator() for accessing and removing items from a LinkedList in a class that implementsRunnable I am also modifying values of this list in some other part of the program concurrently.
Where I am using listIterator() in this part of the code I am getting ConcurrentModificationException from this function call:
java.util.LinkedList$ListItr.checkForComodification
Why do I get this and how do I prevent it?
#Override
public void run()
{
while(true)
{
itr = nodeAttributes.listIterator();
while (itr.hasNext())
{
System.out.println("enterred");
nodeAttribute nA = (nodeAttribute) itr.next();
//System.out.println("for");
if(!nA.isMoving && !nA.moveQueue.isEmpty())
{
if(animator != null)
animator.stop();
animator = (Animator) nA.moveQueue.poll();
//itr.remove();
animator.start();
nA.isMoving = true;
System.out.print( "animator");
}
}
System.out.println( "looping");
}
}

Your post doesn't have a question, just a statement. However what you describe is the expected behaviour. From the docs:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the Iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. 
So to prevent this you need to prevent modification by a writer at the same time as a reader is iterating. Use the Collections.synchronizedList method. All access (readers and writers) should follow this pattern:
// store a single synchronized list reference for all access to use
nodeAttributes = Collections.synchronizedList(theLinkedList);
and then all readers and writers should use a synchronized (list) block.
// Readers might do:
synchronized (list) {
itr = nodeAttributes.listIterator();
while (i.hasNext())
... do stuff ...
}
Those threads that operate without iterating can just use the "atomic" methods on the return object from Collections.synchronizedList, such as add. These methods use a sync block under the covers, so they are just a shorthand and they will still block the thread while another is in a sync block.
There are many, many ways to deal with concurrent reader and writers.
One is the above, but it may lock out other threads for a long time while each iterator does it's stuff.
Another is to copy the list to an array (inside a synchronized section) and then read the array outside the lock.
Yet another would be to use a ReadWriteLock.
and there are more options, all dependent on your exact use case.

ConcurrentModificationException on android's Apache BasicCookieStore

We've been getting crashes in the wild in our android app on ConcurrentModificationException.
Basically, in one of our library, it calls org.apache.http.impl.client.BasicCookieStore's addCookie method and it throws the ConcurrentModificationException. Here is the relevant stack trace:
ArrayList.java line 569: java.util.ArrayList$ArrayListIterator.next
Collections.java line 960: java.util.Collections$UnmodifiableCollection$1.next
....
It looks like ConcurrentModificationException is getting thrown because there are 2 (or more) thread trying to access the array list internal to the BasicCookieStore class. Now, given that BasicCookieStore class is marked as ThreadSafe and all the array list access method seems to be synchronized. What can cause this? Hints?
Here's the source code for BasicCookieStore for reference: source

It is not necessarily caused by a weak thread-safety. It can also happen when you call Iterator.next() for the structure which is modified before that iterator reaches his terminal state. Even in a single thread.
For example this code will throw ConcurrentModificationException:
ArrayList<Object> arrayList = new ArrayList<Object>();
arrayList.add(new Object());
arrayList.add(new Object());
arrayList.add(new Object());
//...
for (Object o : arrayList) { //iterating with iterator
arrayList.remove(0); // perform some modification while
//iterating over the structure
}
If you investigate the ArrayList source code you will see that every modification increments the int modCount field. When you create an iterator via ArrayList.iterator() it takes a snapshot of modCount and compares it to current list's modCount on every iteration to fail if they are not equal.
Update : I made an investigation and found some problem code of BasicCookieStore. I managed to find only one possibility for ConcurrentModificationException to happen : you call BasicCookieStore.toString() in one thread while some modifications (for example addCookie()) happening in another.
This class is almost safe for fail-fast iterators : all methods are synchronized except the toString().
Let's see it code :
#Override
public String toString() {
return cookies.toString();
}
It invokes the ArrayList.toString():
public String toString() {
Iterator<E> it = iterator();
if (! it.hasNext())
return "[]";
StringBuilder sb = new StringBuilder();
sb.append('[');
for (;;) {
E e = it.next();
sb.append(e == this ? "(this Collection)" : e);
if (! it.hasNext())
return sb.append(']').toString();
sb.append(',').append(' ');
}
}
You can see that it uses an iterator. So consider that while toString() is executed we do some modification (and it is really possible because of lack of synchronization), for example addCookie :
public synchronized void addCookie(Cookie cookie) {
if (cookie != null) {
// first remove any old cookie that is equivalent
for (Iterator<Cookie> it = cookies.iterator(); it.hasNext();) {
if (cookieComparator.compare(cookie, it.next()) == 0) {
it.remove();
break;
}
}
if (!cookie.isExpired(new Date())) {
cookies.add(cookie);
}
}
}
This method performs modifications to the list, generally, not very often, but it does.
You can see for your own that there is a possibility to increment modCount while toString's iterator isn't at his terminal state. So when it happens - toString's iterator.next() will throw a ConcurrentModificationException.

From the javadoc of ConcurrentModificationException :
Note that this exception does not always indicate that an object has
been concurrently modified by a different thread. If a single thread
issues a sequence of method invocations that violates the contract of
an object, the object may throw this exception. For example, if a
thread modifies a collection directly while it is iterating over the
collection with a fail-fast iterator, the iterator will throw this
exception.
here is the link :
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/ConcurrentModificationException.html

Concurrent modification exception from a static method that being called asynctask

I have a static method which I am calling from an Asynctask in doInBackGround()
In the method there is this part of code:
ArrayList<Message> messagesList = new ArrayList<Message>();
if (!clearList) {
messagesList.addAll(messages.getMessagesList());
for (Message msg : messagesList) {
if (msg.getId().length() == 0) {
messagesList.remove(msg);
}
}
}
It is sometimes throwing 'Concurrent modification exception', I have tried declaring the method as 'synchronized' but it still didn't help, and I cannot declare the block synchronized, since it is a static method and there is no 'this' reference.
I have also tried to stop a running asynctask if I need to start another one, but it didn't help as well.
Help appreciated.

This has nothing to do with synchronization. You're using an iterator to loop over messagesList, but then using remove to modify it during the iteration. You can't do that, because ArrayList's iterators fail when the list if modified during iteration. From the docs:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException.
Your enhanced for loop is just syntactic sugar around using an Iterator, so you can just make that explicit and then use the iterator's remove method:
Iterator<Message> it = messagesList.iterator();
while (it.hasNext()) {
if (it.next().getId().length == 0) {
it.remove();
}
}
Alternately, you can just use a simple for loop running backward and indexing into the ArrayList (since get(int) is a cheap and constant-time operation on an ArrayList, which isn't true of all Lists):
int index;
for (index = messagesList.length - 1; index >= 0; --index) {
if (messagesList.get(index).getId().length == 0) {
messagesList.remove(index);
}
}

for (Message msg : messagesList) {
if (msg.getId().length() == 0) {
messagesList.remove(msg);
}
}
In this code you using messagesList at a time you are also remove data from messagesList thats why you facing error Concurrent modification exception..
Here better way for solved your issue. Copy All data in one arraylist for remove & remove all that data from Main List.
Message removeMsg = new ArrayList<Message>();
for (Message msg : messagesList) {
if (msg.getId().length() == 0) {
removeMsg.add(msg);
}
}
messagesList.removeAll(removeMsg);

Iterators returned by ArrayList is fail-fast in nature.
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
You can call iterator.remove(); and change loop based on iterator explicitly rather than implicitly.
ArrayList<Message> messagesList = new ArrayList<Message>();
if (!clearList) {
messagesList.addAll(messages.getMessagesList());
for (ListIterator<Message> iterator = messagesList.listIterator();iterator.hasNext();) {
Message message = iterator.next();
if (message.getId().length() == 0) {
iterator.remove();
}
}
}
References:
The For-Each Loop
ArrayList Java docs

The for loop is potentially modifying the list over which it is iterating. This is the cause of the exception. The fact that the modification is based on a condition is the reason it does not occur all of the time as the list is not necessarily modified.
Using an Iterator is a possible solution, which provides a remove() method.

You should use Synchronize keyword for this class because static method doesn't belong to any instance

- Your problem is not associated with synchronization, but the problem of ConcurrentModification you are facing is used to protect collection from taking in object of wrong type.
Eg:
Preventing a Cat object enter into a Collection of Dog type.
- You can solve this problem by using Iterator
ArrayList<Message> messagesList = new ArrayList<Message>();
Iterator<Message> itr = messagesList.iterator();
while(itr.hasNext()){
Message m = itr.next();
itr.remove(); // Its remove() method of Iterator NOT ArrayList's
}