I am trying to add a key value pair to the hashmap inside the Iterator method.
But this is not giving me ConcurrentModificationException . Why?
Since Hashmap is failfast.
Map<String,String> m = new HashMap<>();
m.put("a", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
System.out.println(i.next());
m.put("dsad", "asfsdf");
}
If this is wrong, How i can produce ConcurrentModificationException ?
Thanks.
Update: Just checked.
Map<String,String> m = new HashMap<>();
m.put("a", "a");
m.put("abc", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
System.out.println(i.next());
m.put("dsad", "asfsdf");
}
This is giving me the exception.
It happens that the concurrent modification check done by the HashMap code fails to detect this situation. The code for HashMap's iterator's hasNext in Oracle's JDK7 is:
public final boolean hasNext() {
return next != null;
}
...where (confusingly!) that next is a private data member in the iterator class (not to be confused with the next method on the Iterator interface — to my mind, calling that data member next was a very poor choice).
Note that it doesn't do the check for concurrent modifications. Contrast with this code that is called (indirectly) from Iterator#next:
if (modCount != expectedModCount)
throw new ConcurrentModificationException();
...which does do the check.
So here's what happens in your code:
You create a HashMap.
You add one entry to it.
You begin an iteration.
hasNext is true, so you go into the body of your loop.
You get the element from next; at this point, the iterator remembers what the next element should be on its internal data member (the confusingly-named next), and in this case since there's no next element in the map, that next data member is set to null, meaning that the iteration is complete.
You add to the map.
Your code calls hasNext, which sees the next data member is null and returns false.
If you had two elements in the map before starting your loop instead of one, you'd get the exception (from next).
I've previously argued this is, or is very nearly, a bug, but it's a pretty fuzzy area, and others have argued quite reasonably that it isn't. The documentation doesn't say specifically which methods of Iterator<E> will throw the exception, just that it will get thrown. The documentation also says it's only thrown on a "best effort" basis, it's not guaranteed.
Whether one considers this a bug or not, it's unlikely to be changed at this point, as the pain of changing it (breaking some existing code that probably shouldn't have relied on this behavior) far outweighs the benefit (possibly being more "correct").
The iterator may throw ConcurrentModificationException but is not guaranteed to.
From the javadoc of HashMap:
Note that the fail-fast behavior of an iterator cannot be guaranteed
as it is, generally speaking, impossible to make any hard guarantees
in the presence of unsynchronized concurrent modification. Fail-fast
iterators throw ConcurrentModificationException on a best-effort
basis. Therefore, it would be wrong to write a program that depended
on this exception for its correctness: the fail-fast behavior of
iterators should be used only to detect bugs.
Try this:
Map<String,String> m = new HashMap<>();
m.put("a", "a");
Iterator<String> i = m.keySet().iterator();
while(i.hasNext()){
m.remove("a");
System.out.println(i.next());
}
Related
I am relying on list iterators to move through a list of characters. This is a single-threaded program and I use listIterator objects sequentially in 4 different methods. Each method has the same setup:
private void myMethod(ArrayList<Integer> input) {
ListIterator<Integer> i = input.listIterator();
while (i.hasNext()) {
Integer in = i.next();
if (in < 10)
i.remove();
else
i.set(in*in); // because its lucky
}
}
With this pattern, on the second iterator the following Exception is thrown:
java.util.ConcurrentModificationException
However, looking at the javadocs I don't see this Exception in the Exceptions thrown nor do I see a method to close the iterator after I am done. Am I using the listIterator incorrectly? I have to iterate over the same ArrayList multiple times, each time conditionally removing or mutating each element. Maybe there is a better way to iterate over the ArrayList and this use-case is not best solved by a ListIterator.
java docs for ListIterator
This is explained in the ArrayList javadoc, you are modifying the list with remove() and set() while using an Iterator:
The iterators returned by this class's iterator and listIterator methods are fail-fast: if the list is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException. Thus, in the face of concurrent modification, the iterator fails quickly and cleanly, rather than risking arbitrary, non-deterministic behavior at an undetermined time in the future.
It’s hard to give diagnostic for a problem when the shown code clearly isn’t the code that produced the exception, as it doesn’t even compile. The remove method of Iterator doesn’t take arguments and the set method is defined on ListIterator, but your code declares the variable i only as Iterator.
A fixed version
private void myMethod(ArrayList<Integer> input) {
ListIterator<Integer> i = input.listIterator();
while (i.hasNext()) {
Integer in = i.next();
if (in < 10)
i.remove();
else
i.set(in*in);
}
}
would run without problems. The answer to your general question is that each modification invalidates all existing iterators, except the one used to make the modification when you did use an iterator for the modification and not the collection interface directly.
But in your code, there is only one iterator, which is only created and used for this one operation. As long as there is no overlapping use of iterators to the same collection, there is no problem with the invalidation. Iterators existing from previous operations are abandoned anyway and the iterators used in subsequent operations do not exist yet.
Still, it’s easier to use
private void myMethod(ArrayList<Integer> input) {
input.removeIf(in -> in < 10);
input.replaceAll(in -> in*in);
}
instead. Unlike the original code, this does two iterations, but as explained in this answer, removeIf will be actually faster than iterator based removal in those cases, where performance really matters.
But still, the problem persists. The shown code can’t cause a ConcurrentModificationException, so your actual problem is somewhere else and may still be present, regardless of how this one method has been implemented.
I am not knowledgable enough about Java ListIterators to answer the question but it appears I have run into the XY problem here. The problem seems to be better solved with Java Streams to remove the element or map the element into a new ArrayList by exercising a function on each element in the original ArrayList.
private ArrayList<Integer> myMethod(ArrayList<Integer> input) {
ArrayList<Integer> results = input.stream().filter(
in -> (in < 10)).collect(Collectors.toCollection(ArrayList::new));
results = input.stream().map(
in -> in*in).collect(Collectors.toCollection(ArrayList::new));
return results;
}
I have 3 servers and a client sending messages. And i implement a BFT algorithm.
So i have this part of code
int tam = 0;
if (unordered.size() <= maxOrderSize) {
tam = unordered.size();
} else {
tam = maxOrderSize;
}
HashMap<String, byte[]> prop = new HashMap<String, byte[]>(tam);
Iterator<String> it = unordered.keySet().iterator();
for (int i = 0; i < tam; i++) {
if (it.hasNext()) {
String id = it.next();
prop.put(id, unordered.get(id));
it.remove();
unordered.remove(id);
}
}
and during the runtime objects are imported and removed from my Map unordered.
also i want to mention that unordered is defined:
Map<String, byte[]> unordered = Collections.synchronizedMap(new HashMap<String, byte[]>());
But suddenly it creates this exception:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.remove(HashMap.java:1456)
at edu.bft.comm.layer.BatchControl.createOrderMessage(BatchControl.java:123)
at edu.bft.comm.layer.BatchControlTPM.run(BatchControlTPM.java:24)
Any idea why this happens?
EDIT1: I tried to remove that line: unordered.remove(id);
and i got that error:
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
at edu.bft.comm.layer.BatchControl.createOrderMessage(BatchControl.java:120)
at edu.bft.comm.layer.BatchControlTPM.run(BatchControlTPM.java:24)
EDIT2: Also i want to mention that while i iterate unordered ,some new objects may added, while new messages are coming from the client.
You're modifying the Map yourself.
Remove this line: unordered.remove(id);
From your edit, you're modifying the Map while iterating it so obviously it's the problem.
The doc said:
The set is backed by the map, so changes to the map are reflected in
the set, and vice-versa. If the map is modified while an iteration
over the set is in progress (except through the iterator's own remove
operation), the results of the iteration are undefined.
And HashMap will throw a ConcurrentModificationException as you see.
OK, after Edit2, I feel that another answer makes sense.
First of all: If you have a standard HashMap, it is not thread safe by default! So iterating over such a collection may always cause trouble if other threads concurrently modify it. Maybe adding new items won't cause an exception, maybe it will.
If you have the possibility to change the Map implementation of unordered, use something thread safe like a ConcurrentHashMap. This one won't have any trouble if someone modifies it while another thread iterates over its elements with an iterator - the iterator has its own collection of items which won't directly interfere with the original Map.
If you cannot change the implementation, you're in trouble...
That's exactly what ConcurrentModificationException prevented:
For example, if a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will throw this exception.
In general, you cannot modify the collection when the iterator is iterating.
See the SO answer and this one.
HashMap is not supposed to be thread safe, then why do the iterators throw concurrentmodificationexception if someone has modified the hashMap.
Also ConcurrentHashMap does not throw this exception.
Is Iterator implementation different for different datastructures or there is someone method within these data structures which throw ConcurrentModificationException
When the structure of a HashMap is modified (i.e. entries are added or removed) while the HashMap is being iterated over, the iterator may fail in many ways.
The ConcurrentModificationException exception is meant to make any iterators fail-fast as a result of such modifications.
That's what the modCount field is for :
/**
* The number of times this HashMap has been structurally modified
* Structural modifications are those that change the number of mappings in
* the HashMap or otherwise modify its internal structure (e.g.,
* rehash). This field is used to make iterators on Collection-views of
* the HashMap fail-fast. (See ConcurrentModificationException).
*/
transient int modCount;
This behavior is not specific to Maps. Collections also throw this exception when they are being modified during iteration.
Don't be fooled into thinking that Concurrent relates solely to multi-threading.
The exception means that the collection was structurally modified whilst you are using an iterator created before the modification. That modification might be performed by a different thread, but it could be the same thread as the one doing the iteration.
The following single-threaded code yields a CME:
Map<String, String> map = new HashMap<>();
map.put("a", "a");
map.put("b", "b");
for (String k: map.entrySet()) {
map.clear();
}
Or, showing the iterator in the loop more clearly (the two loops are equivalent):
Iterator<String> it = map.entrySet().iterator();
while (it.hasNext()) {
String k = it.next();
map.clear();
}
it.hasNext() is called after map.clear(), resulting in a ConcurrentModificationException.
Is Iterator implementation different for different datastructures or
there is someone method within these data structures which throw
ConcurrentModificationException ?
Yes, there are different Iterator implementations inside Collection classes.
For example, HashMap class (uses HashIterator internally), ConcurrentHashMap (uses KeyIterator, ValueIterator, etc.. internally), ArrayList (uses Iterator from AbstractList), etc..
HashMap's Iterator is different from ConcurrentHashMap's Iterator implementation.
HashMap's Iterator maintain a version number (expectedModCount) and verify checkForComodification() as shown below:
final void checkForComodification() {
if (modCount != expectedModCount)
throw new ConcurrentModificationException();
}
So, in the middle of the iteration, if the underlying collection size is modified (by adding/removing the elements), then the Iterator throw ConcurrentModificationException as shown above.
Where as ConcurrentHashMap's Iterator implementations does not perform the above check, so it does not throw the ConcurrentModificationException. You can also find the same point from API the ConcurrentHashMap here.
They (ConcurrentHashMaps) do not throw ConcurrentModificationException. However, iterators
are designed to be used by only one thread at a time.
As the HashMap is a fail-fast Collection (All the collections under java.util package are fail fast), Iterator Throws exception if any modification is done to the structure or the keys majorly in case of Hashmap.
we have a variable in Hashmap that count the number of Modifications in the Map.and using that iterator keeps the track of the no of modifications done on the Collection.
transient volatile int modCount;
Please read the below link: the Difference is explained very well:
Java Modcount (ArrayList)
Good Example:
http://www.journaldev.com/122/java-concurrenthashmap-example-iterator
I get this definition : As name suggest fail-fast Iterators fail as soon as they realized that structure of Collection has been changed since iteration has begun.
what it mean by since iteration has begun? is that mean after Iterator it=set.iterator() this line of code?
public static void customize(BufferedReader br) throws IOException{
Set<String> set=new HashSet<String>(); // Actual type parameter added
**Iterator it=set.iterator();**
First of all, they are fail-fast, not fail-safe.
The contract is that structural modifications (i.e. insertions/deletions) of certain types of collections invalidate existing iterators into the collection. Fail-fast iterators attempt to detect that they are not supposed to be valid and throw a ConcurrentModificationException. This is done as a service to you, the programmer, to help discover this type of bugs quicker.
In your example:
Iterator it = set.iterator();
it.next();
set.add("unique-entry"); // invalidates the iterator
it.next();
If you're lucky, the second it.next() will detect the invalid usage and throw an exception. Note that this is done on a best-effort basis and is not guaranteed.
is that mean after Iterator it=set.iterator() this line of code?
Yes. If you look at the code for HashSet.iterator() you'll see it's just this:
return map.keySet().iterator();
... which delegate's to HashMap.KeySet.iterator(). There are a few more links in the chain, but eventually you get to HashMap.HashIterator, which contains this in the constructor:
private abstract class HashIterator<E> implements Iterator<E> {
int expectedModCount; // For fast-fail
...
HashIterator() {
expectedModCount = modCount;
...
}
}
... where modCount is a field in the enclosing instance of HashMap which keeps track of the number of modifications.
The iterator being fail-fast means the following piece of code is expected to fail:
Set<String> set = new HashSet<String>();
Iterator<String> it = set.iterator();
set.add("");
it.next(); // the set has changed now, and the iterator will throw an exception
because the following series of events occur: The iterator is created, then its underlying collection changes, then the iterator is accessed.
yes, don't change the collection after using .iterator() if you are planning to iterate over it , you can use the .remove() if you want to remove the latest element though
Before Fail Fast Iterator is starting to work it’s getting count of collection and after any iteration it is checking if count is changed or not, and in case of changed count JVM will throw ConcurrentModificationException. Fail fast iterators are any iterator of collection which is inside java.util package(e.g. ArrayList, LinkedList etc) and Fail Safe iterators are iterators which are inside of java.concurrent package(e.g. CopyOnWriteArrayList, CopyOnWriteSet etc.). Fail Fast iterators will throw exception in case of concurrent modification but Fail Safe iterator is basically working with copy of collection which is not throwing exception in case of concurrent modification.
Say I am iterating over a Map in Java... I am unclear about what I can to that Map while in the process of iterating over it. I guess I am mostly confused by this warning in the Javadoc for the Iterator interface remove method:
[...] The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.
I know for sure that I can invoke the remove method without any issues. But while iterating over the Map collection, can I:
Change the value associated with a key with the Map class put method (put with an existing key)?
Add a new entry with the Map class put method (put with a new key)?
Remove an entry with the Map class remove method?
My guess is that I can probably safely do #1 (put to an existing key) but not safely do #2 or #3.
Thanks in advance for any clarification on this.
You can use Iterator.remove(), and if using an entrySet iterator (of Map.Entry's) you can use Map.Entry.setValue(). Anything else and all bets are off - you should not change the map directly, and some maps will
not permit either or both of the aforementioned methods.
Specifically, your (1), (2) and (3) are not permitted.
You might get away with setting an existing key's value through the Map object, but the Set.iterator() documentation specifically precludes that and it will be implementation specific:
If the map is modified while an iteration over the set is in progress (except through the iterator's own remove operation, or through the setValue operation on a map entry returned by the iterator) the results of the iteration are undefined. (emphasis added)
If you take a look at the HashMap class, you'll see a field called 'modCount'. This is how the map knows when it's been modified during iteration. Any method that increments modCount when you're iterating will cause it to throw a ConcurrentModificationException.
That said, you CAN put a value into a map if the key already exists, effectively updating the entry with the new value:
Map<String, Object> test = new HashMap<String, Object>();
test.put("test", 1);
for(String key : test.keySet())
{
test.put(key, 2); // this works!
}
System.out.println(test); // will print "test->2"
When you ask if you can perform these operations 'safely,' you don't have to worry too much because HashMap is designed to throw that ConcurrentModificationException as soon as it runs into a problem like this. These operations will fail fast; they won't leave the map in a bad state.
There is no global answer. The map interface let the choice to the users. Unfortunately, I think that all the implementations in the jdk use the fail-fast implementation (here is the definition of fail-fast, as it stated in the HashMap Javadoc):
The iterators returned by all of this
class's "collection view methods" are
fail-fast: if the map is structurally
modified at any time after the
iterator is created, in any way except
through the iterator's own remove
method, the iterator will throw a
ConcurrentModificationException. Thus,
in the face of concurrent
modification, the iterator fails
quickly and cleanly, rather than
risking arbitrary, non-deterministic
behavior at an undetermined time in
the future.
In general, if you want to change the Map while iterating over it, you should use one of the iterator's methods. I have not actually tested to see if #1 will will work, but the others definitely will not.