We Keep Coding

Why i get ConcurrentModificationException even using an Iterator?

I have 3 servers and a client sending messages. And i implement a BFT algorithm.
So i have this part of code
int tam = 0;
if (unordered.size() <= maxOrderSize) {
tam = unordered.size();
} else {
tam = maxOrderSize;
}
HashMap<String, byte[]> prop = new HashMap<String, byte[]>(tam);
Iterator<String> it = unordered.keySet().iterator();
for (int i = 0; i < tam; i++) {
if (it.hasNext()) {
String id = it.next();
prop.put(id, unordered.get(id));
it.remove();
unordered.remove(id);
}
}
and during the runtime objects are imported and removed from my Map unordered.
also i want to mention that unordered is defined:
Map<String, byte[]> unordered = Collections.synchronizedMap(new HashMap<String, byte[]>());
But suddenly it creates this exception:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.remove(HashMap.java:1456)
at edu.bft.comm.layer.BatchControl.createOrderMessage(BatchControl.java:123)
at edu.bft.comm.layer.BatchControlTPM.run(BatchControlTPM.java:24)
Any idea why this happens?
EDIT1: I tried to remove that line: unordered.remove(id);
and i got that error:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
at edu.bft.comm.layer.BatchControl.createOrderMessage(BatchControl.java:120)
at edu.bft.comm.layer.BatchControlTPM.run(BatchControlTPM.java:24)
EDIT2: Also i want to mention that while i iterate unordered ,some new objects may added, while new messages are coming from the client.

You're modifying the Map yourself.
Remove this line: unordered.remove(id);
From your edit, you're modifying the Map while iterating it so obviously it's the problem.
The doc said:
The set is backed by the map, so changes to the map are reflected in
the set, and vice-versa. If the map is modified while an iteration
over the set is in progress (except through the iterator's own remove
operation), the results of the iteration are undefined.
And HashMap will throw a ConcurrentModificationException as you see.

OK, after Edit2, I feel that another answer makes sense.
First of all: If you have a standard HashMap, it is not thread safe by default! So iterating over such a collection may always cause trouble if other threads concurrently modify it. Maybe adding new items won't cause an exception, maybe it will.
If you have the possibility to change the Map implementation of unordered, use something thread safe like a ConcurrentHashMap. This one won't have any trouble if someone modifies it while another thread iterates over its elements with an iterator - the iterator has its own collection of items which won't directly interfere with the original Map.
If you cannot change the implementation, you're in trouble...

That's exactly what ConcurrentModificationException prevented:
For example, if a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will throw this exception.
In general, you cannot modify the collection when the iterator is iterating.
See the SO answer and this one.

How can I fix Exception in thread "main" java.util.ConcurrentModificationException [duplicate]

This question already has an answer here:
Can anyone explain me over ConcurrentModificationException?
(1 answer)
Closed 9 years ago.
I have 2 HashMap<Integer,Point3D> objects names are positiveCoOrdinate and negativeCoOrdinates.
I am checking PositiveCoOrdinates with following condition.if it satisfies that corresponding point adding into negativeCoOrdinates and deleting from positiveCoOrdinates.
HashMap<Integer, Point3d> positiveCoOrdinates=duelList.get(1);
HashMap<Integer, Point3d> negativecoOrdinates=duelList.get(2);
//condition
Set<Integer> set=positiveCoOrdinates.keySet();
for (Integer pointIndex : set) {
Point3d coOrdinate=positiveCoOrdinates.get(pointIndex);
if (coOrdinate.x>xMaxValue || coOrdinate.y>yMaxValue || coOrdinate.z>zMaxValue) {
negativecoOrdinates.put(pointIndex, coOrdinate);
positiveCoOrdinates.remove(pointIndex);
}
}
While adding,deleting time I am getting the following error.
Exception in thread "main" java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
at java.util.HashMap$KeyIterator.next(Unknown Source)
at PlaneCoOrdinates.CoordinatesFiltering.Integration(CoordinatesFiltering.java:167)
at PlaneCoOrdinates.CoordinatesFiltering.main(CoordinatesFiltering.java:179)
For my testing,I mention System.out.println(coOrdinate.x); statement inside If condition.it's working fine.
If I add 2 lines(What I mention above) inside If condition,it throwing error.
How can I fix this.
Thanks.

The easiest way is to make a copy of the keySet:
Set<Integer> set= new HashSet<Integer>(positiveCoOrdinates.keySet());
The problem occurs because you are modifing the positiveCoOrdinates while you are using an Iterator that iterates through the keys.
You can also refactor your code and use an iterator over the entry set. This would be a better approach.
Set<Entry<Integer, Point3d>> entrySet = positiveCoOrdinates.entrySet();
for (Iterator<Entry<Integer, Point3d>> iterator = entrySet.iterator(); iterator.hasNext();) {
Entry<Integer, Point3d> entry = iterator.next();
Point3d coOrdinate = entry.getValue();
if (coOrdinate.x > xMaxValue || coOrdinate.y > yMaxValue
|| coOrdinate.z > zMaxValue) {
Integer pointIndex = entry.getKey();
negativecoOrdinates.put(pointIndex, coOrdinate);
iterator.remove();
}
}

You can't remove() from iterated collection when using the enhanced for-each loop. The for-each loop uses Iterator<Integer> implicitly. The JavaDoc clearly states that
The iterators returned by all of this class's "collection view
methods" are fail-fast: if the map is structurally modified at any
time after the iterator is created, in any way except through the
iterator's own remove() method, the iterator will throw a
ConcurrentModificationException. Thus, in the face of concurrent
modification, the iterator fails quickly and cleanly, rather than
risking arbitrary, non-deterministic behavior at an undetermined time
in the future.
The for-each loop creates an iterator internally and uses it to traverse the set. Then you change the structure of the set ... and the iterator has to fail. The thing is that you don't have access to the iterator's methods, so you have to use Iterator<Integer> explicitly. The generated traversing bytecode will be the same, the only difference being you being able to remove elements from the list as you traverse it.
Set<Integer> set = positiveCoOrdinates.keySet();
for (Iterator<Integer> iterator = set.iterator(); iterator.hasNext(); ) {
Integer pointIndex = iterator.next();
Point3d coOrdinate = positiveCoOrdinates.get(pointIndex);
if (coOrdinate.x>xMaxValue || coOrdinate.y>yMaxValue || coOrdinate.z>zMaxValue) {
negativecoOrdinates.put(pointIndex, coOrdinate);
iterator.remove(pointIndex); // this line changed!
}
}
If you are not familiar with iterators and their function, see the Oracle tutorial on Collections:
An Iterator is an object that enables you to traverse through a
collection and to remove elements from the collection selectively, if
desired. You get an Iterator for a collection by calling its iterator()
method.
Note that Iterator.remove() is the only safe way to modify a
collection during iteration; the behavior is unspecified if the
underlying collection is modified in any other way while the iteration
is in progress.
Use Iterator instead of the for-each construct when you need to:
Remove the current element. The for-each construct hides the iterator, so you cannot call remove(). Therefore, the for-each
construct is not usable for filtering.

If you want to modify collection at run time, you need to use Iterator instead of enhanced for loop. Because enhanced for loop provide only read only functionality. Following is the Iterator example:
Iterator<Entity> iterator = collection.Iterator();
while(iterator.hasNext()){
//DO Your Stuff
iterator.remove(); // this function call remove the element from collection at run time
}

As it has been pointed out by René the reason for this very common issue is concurrent modification to a collection while it is being read by another.
You can use ConcurrentHashMap or collections like CopyOnWriteArrayLit, but beware that those approaches might be little bit expensive and simple changers to code to eliminate reading of the same collection while in a iteration will resolve this type of issues.

How I can get ConcurrentModificationException while iterating Hashmap?

I am trying to add a key value pair to the hashmap inside the Iterator method.
But this is not giving me ConcurrentModificationException . Why?
Since Hashmap is failfast.
Map<String,String> m = new HashMap<>();
m.put("a", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
System.out.println(i.next());
m.put("dsad", "asfsdf");
}
If this is wrong, How i can produce ConcurrentModificationException ?
Thanks.
Update: Just checked.
Map<String,String> m = new HashMap<>();
m.put("a", "a");
m.put("abc", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
System.out.println(i.next());
m.put("dsad", "asfsdf");
}
This is giving me the exception.

It happens that the concurrent modification check done by the HashMap code fails to detect this situation. The code for HashMap's iterator's hasNext in Oracle's JDK7 is:
public final boolean hasNext() {
return next != null;
}
...where (confusingly!) that next is a private data member in the iterator class (not to be confused with the next method on the Iterator interface — to my mind, calling that data member next was a very poor choice).
Note that it doesn't do the check for concurrent modifications. Contrast with this code that is called (indirectly) from Iterator#next:
if (modCount != expectedModCount)
throw new ConcurrentModificationException();
...which does do the check.
So here's what happens in your code:
You create a HashMap.
You add one entry to it.
You begin an iteration.
hasNext is true, so you go into the body of your loop.
You get the element from next; at this point, the iterator remembers what the next element should be on its internal data member (the confusingly-named next), and in this case since there's no next element in the map, that next data member is set to null, meaning that the iteration is complete.
You add to the map.
Your code calls hasNext, which sees the next data member is null and returns false.
If you had two elements in the map before starting your loop instead of one, you'd get the exception (from next).
I've previously argued this is, or is very nearly, a bug, but it's a pretty fuzzy area, and others have argued quite reasonably that it isn't. The documentation doesn't say specifically which methods of Iterator<E> will throw the exception, just that it will get thrown. The documentation also says it's only thrown on a "best effort" basis, it's not guaranteed.
Whether one considers this a bug or not, it's unlikely to be changed at this point, as the pain of changing it (breaking some existing code that probably shouldn't have relied on this behavior) far outweighs the benefit (possibly being more "correct").

The iterator may throw ConcurrentModificationException but is not guaranteed to.
From the javadoc of HashMap:
Note that the fail-fast behavior of an iterator cannot be guaranteed
as it is, generally speaking, impossible to make any hard guarantees
in the presence of unsynchronized concurrent modification. Fail-fast
iterators throw ConcurrentModificationException on a best-effort
basis. Therefore, it would be wrong to write a program that depended
on this exception for its correctness: the fail-fast behavior of
iterators should be used only to detect bugs.

Try this:
Map<String,String> m = new HashMap<>();
m.put("a", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
m.remove("a");
System.out.println(i.next());
}

fail-fast iterator

I get this definition : As name suggest fail-fast Iterators fail as soon as they realized that structure of Collection has been changed since iteration has begun.
what it mean by since iteration has begun? is that mean after Iterator it=set.iterator() this line of code?
public static void customize(BufferedReader br) throws IOException{
Set<String> set=new HashSet<String>(); // Actual type parameter added
**Iterator it=set.iterator();**

First of all, they are fail-fast, not fail-safe.
The contract is that structural modifications (i.e. insertions/deletions) of certain types of collections invalidate existing iterators into the collection. Fail-fast iterators attempt to detect that they are not supposed to be valid and throw a ConcurrentModificationException. This is done as a service to you, the programmer, to help discover this type of bugs quicker.
In your example:
Iterator it = set.iterator();
it.next();
set.add("unique-entry"); // invalidates the iterator
it.next();
If you're lucky, the second it.next() will detect the invalid usage and throw an exception. Note that this is done on a best-effort basis and is not guaranteed.

is that mean after Iterator it=set.iterator() this line of code?
Yes. If you look at the code for HashSet.iterator() you'll see it's just this:
return map.keySet().iterator();
... which delegate's to HashMap.KeySet.iterator(). There are a few more links in the chain, but eventually you get to HashMap.HashIterator, which contains this in the constructor:
private abstract class HashIterator<E> implements Iterator<E> {
int expectedModCount; // For fast-fail
...
HashIterator() {
expectedModCount = modCount;
...
}
}
... where modCount is a field in the enclosing instance of HashMap which keeps track of the number of modifications.

The iterator being fail-fast means the following piece of code is expected to fail:
Set<String> set = new HashSet<String>();
Iterator<String> it = set.iterator();
set.add("");
it.next(); // the set has changed now, and the iterator will throw an exception
because the following series of events occur: The iterator is created, then its underlying collection changes, then the iterator is accessed.

yes, don't change the collection after using .iterator() if you are planning to iterate over it , you can use the .remove() if you want to remove the latest element though

Before Fail Fast Iterator is starting to work it’s getting count of collection and after any iteration it is checking if count is changed or not, and in case of changed count JVM will throw ConcurrentModificationException. Fail fast iterators are any iterator of collection which is inside java.util package(e.g. ArrayList, LinkedList etc) and Fail Safe iterators are iterators which are inside of java.concurrent package(e.g. CopyOnWriteArrayList, CopyOnWriteSet etc.). Fail Fast iterators will throw exception in case of concurrent modification but Fail Safe iterator is basically working with copy of collection which is not throwing exception in case of concurrent modification.

java : why is it safe to modify a map's entry's value while we are iterating over it?

Suppose a map contains integer keys, and a list of strings as its value. Then, I CAN'T do this :
for (Map.Entry<Integer, List<String>> entry : map.entrySet()){
for (String string : entry.getValue()){
if (string.startsWith("a")){
entry.getValue().remove(string);
}
}
}
It throws ConcurrentModificationException. But if I do the following :
for (Map.Entry<Integer, List<String>> entry : map.entrySet()){
entry.setValue(new ArrayList<String>());
}
This works perfectly. Aren't we modifying the underlying map even now ?

The problem has nothing to do with the Map, only the way you're using the value list. The following will fail with any ArrayList:
for (String string : list){
if (string.startsWith("a")){
list.remove(string);
}
}
The reason for this is discussed in the Javadoc of ArrayList:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
(To put it another way: if you remove the element from the list, the iterator might not be pointing to the right index in the underlying array anymore. So instead of allowing you to use a potentially corrupted iterator, it throws a ConcurrentModificationException as a courtesy to let you know that you need to redesign your program.)
A simple fix is to use
Iterator<String> itr = entry.getValue().iterator();
while (itr.hasNext()) {
if (itr.next().startsWith("a")) {
itr.remove();
}
}

Please look at the Javadoc for the entrySet() in HashMap.java and you will find out why!
From the documentation:
Returns a Set view of the mappings contained in this map. The set is backed by the map, so changes to the map are reflected in the set, and vice-versa. If the map is modified while an iteration over the set is in progress (except through the iterator's own remove operation, or through the setValue operation on a map entry returned by the iterator) the results of the iteration are undefined. The set supports element removal, which removes the corresponding mapping from the map, via the Iterator.remove, Set.remove, removeAll, retainAll and clear operations. It does not support the add or addAll operations.
Specified by: entrySet() in Map,
Overrides: entrySet() in AbstractMap
Returns: a set view of the mappings contained in this map