I was trying to use the iterator methods on a BlockingQueue and discovered that hasNext() is non-blocking - i.e. it will not wait until more elements are added and will instead return false when there are no elements.
So here are the questions :
Is this bad design, or wrong
expectation?
Is there a way to use the blocking
methods of the BLockingQueue with
its parent Collection class methods
(e.g. if some method were expecting
a collection, can I pass a blocking
queue and hope that its processing
will wait until the Queue has more
elements)
Here is a sample code block
public class SomeContainer{
public static void main(String[] args){
BlockingQueue bq = new LinkedBlockingQueue();
SomeContainer h = new SomeContainer();
Producer p = new Producer(bq);
Consumer c = new Consumer(bq);
p.produce();
c.consume();
}
static class Producer{
BlockingQueue q;
public Producer(BlockingQueue q) {
this.q = q;
}
void produce(){
new Thread(){
public void run() {
for(int i=0; i<10; i++){
for(int j=0;j<10; j++){
q.add(i+" - "+j);
}
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
}.start();
}
}
static class Consumer{
BlockingQueue q;
public Consumer(BlockingQueue q) {
this.q = q;
}
void consume() {
new Thread() {
public void run() {
Iterator itr = q.iterator();
while (itr.hasNext())
System.out.println(itr.next());
}
}.start();
}
}
}
This Code only prints the iteration once at the most.
Just don't use iterators with Queues. Use peek() or poll() instead or take() if it's a BlockingQueue:
void consume() {
new Thread() {
#Override
public void run() {
Object value;
// actually, when using a BlockingQueue,
// take() would be better than poll()
while ((value=q.poll())!=null)
System.out.println(value);
}
}.start();
}
A Queue is an Iterable because it is a Collection and hence needs to provide an iterator() method, but that shouldn't ever be used, or you shouldn't be using a Queue in the first place.
1) Is this bad design, or wrong expectation?
Wrong expectations since it would otherwise violate the contract of Iterator which on Iterator.next() says: Throws: NoSuchElementException - iteration has no more elements.
If next() would block the exception would never be thrown.
2) Is there a way to use the blocking methods
Yes, for instance by extending the class and overriding the next and hasNext methods to use blocking routines instead. Note that hasNext would need to always return true in this case - which again violates the contract.
if an iterator blocked on hasNext then the iteration would never finish unless you explicitly broke out of it, this would be quite a strange design.
In any case the LinkedBlockingQueue javadoc has this to say
Returns an iterator over the elements in this queue in proper sequence.
The returned <tt>Iterator</tt> is a "weakly consistent" iterator that will
never throw {#link ConcurrentModificationException}, and guarantees to
traverse elements as they existed upon construction of the iterator, and
may (but is not guaranteed to) reflect any modifications subsequent to
construction.
I think that it may be reasonable under certain circumstances to have an Iterable whose iterator() will block, although having a seperate BlockingIteratorwould be foolish. The reason for this is because that lests you use an enhanced for loop, which can,in some cases, make your code cleaner. (If it would not accomplish that in your particular circumstance, do not do this at all.)
for(Request request:requests) process(request);
However, the iterator is still not free from a termination condition! The iterator should terminate once the queue has been closed to new items, and runs out of elements.
The issue still remains, though, that if the loop was already blocking on the iterator's next() method, the only way to exit if the queue is closed is to throw an exception, which the surrounding code would need to handle correctly, If you choose to do this, make sure you explain very clearly and precisely, how your implementation works in the javadoc comments.
The Iterator for LinkedBlockingQueue has this as its hasNext implementation:
private Node<E> current;
public boolean hasNext() {
return current != null;
}
so this will only work per call. You can wrap the method in a while(true) loop if you want to wait for elements and use the standard java Iterator idiom:
while (true) {
if(itr.hasNext()) {
System.out.println(itr.next());
}
}
Related
I have a below method which is called by multiple threads concurrently to get the live socket. It takes LinkedBlockingQueue as the parameter and then I iterate and see if there is any liveSocket available and if it is available then I remove and return that socket.
private Optional<Holder> getSocket(final LinkedBlockingQueue<Holder> endPoints) {
Optional<Holder> liveSocket = Optional.absent();
if (!endPoints.isEmpty()) {
for (Holder state : endPoints) {
// check if socket is live? if yes then remove and return that.
if (state.isLive()) {
liveSocket = Optional.of(state);
endPoints.remove(state);
return liveSocket;
}
}
}
return Optional.absent();
}
Wanted to check if my above code is thread safe or not? Here Holder is an immutable class.
The queue manipulation operations are thread safe, so the remove() will not throw ConcurrentModificationException. However, you have thread-safety problems around the state of the objects contained in the queue.
There's a race condition between when you check the "live" state of the Holder object and when you remove it from the queue. Another thread could be running in the same code at the same time, with the likely result that both threads would take the same object. Whichever thread got to the remove() call last would get a false return, but you don't examine the result so you'd never know. Both threads would then attempt to use the same object.
You need to synchronize around the search/remove operation.
For curiosity, here's the code I used to show that ConcurrentModificationException does not occur with LinkedBlockingQueue:
public static void main(String[] args) throws Exception
{
String[] data = { "a", "b", "c", "d", "e", "f","g" };
LinkedBlockingQueue<String> lb = new LinkedBlockingQueue<>(Arrays.asList(data));
new Thread(() ->
{
try
{
Thread.sleep(2000);
lb.add("x");
System.out.println("added");
Thread.sleep(1000);
lb.remove("e");
System.out.println("removed");
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}).start();
for (String s : lb)
{
System.out.println(s);
Thread.sleep(1000);
}
}
If you substitute LinkedList for LinkedBlockingQueue you get the ConcurrentModificationException as expected.
Output:
a
b
added
c
removed
d
f
g
x
It's not only not thread-safe, it's wrong even within a single thread. You will get a ConcurrentModificationException on the remove(). You need to use an explicit Iterator and to do the removal via the Iterator.
And for correctness via multiple threads you need synchronization or a semaphore around the loop.
NB The isEmpty() test is pointless. The iteration already has to check for that. Don't keep a dog and bark yourself.
I have to ensure while iterating the Vector; there is no update on that Vector to avoid ConcurrentModificationException. I can use concurrent collection. But i just want to give a try on Vector. Below is the code I have written.
public class TestConcurrentModification1 {
Vector a = new Vector();
public static void main(String[] args) {
final TestConcurrentModification1 obj = new TestConcurrentModification1();
new Thread(){
public void run(){
for(int i = 0; i < 5; i++){
try {
Thread.sleep(1);
} catch (InterruptedException e) {}
obj.a.add(""+i);
}
System.out.println(obj.a);
}
}.start();
new Thread(){
public void run(){
try {
Thread.sleep(10);
} catch (InterruptedException e) {
}
synchronized (obj.a) {
Iterator itr = obj.a.iterator();
while(itr.hasNext()) {
obj.a.add("TEST");//java.lang.OutOfMemoryError: Java heap space
//itr.remove(); //java.lang.IllegalStateException
}
}
}
}.start();
}
}
But the above code is throwing 1) OutOfMemoryError OR 2) IllegalStateException. Could you please explain what is causing these two exception. And how to achieve my goal of avoiding ConcurrentModificationException on a Vector?
I have to solve this for Java 1.4.2 or prior.
One part of your Problem:
Iterator itr = obj.a.iterator();
while(itr.hasNext()) {
obj.a.add("TEST");// <- if itr.hasNext() would have returned false in next iteration, now it won't
}
This is an infinite loop that increases memory usage in each iteration. So you'll run into an OutOfMemory sooner or later.
I suggest using a good old for-loop for inserting values. Use an iterator if you actually want to iterate something :)
More: You are synchronizing against a non-final member.
More: Iterator.remove throws ...
IllegalStateException - if the next method has not yet been called, or the remove method has already been called after the last call to the next method.
And last but not least: The race condition already mentioned by Sotirios (+1 for him).
Whenever you synchronize, make sure you synchronize every call on the critical resource.
You have a good old race condition on your hands.
Your first Thread, except for adding the first element to your Vector, serves absolutely no purpose. You can replace it with
obj.a.add("first");
The beef, as others have noted, is here
Iterator itr = obj.a.iterator();
while (itr.hasNext()) {
obj.a.add("TEST");// java.lang.OutOfMemoryError: Java
// heap space
// itr.remove(); //java.lang.IllegalStateException
}
itr.hasNext() is implemented as
public boolean hasNext() {
return cursor != elementCount;
}
Where cursor starts at 0 and elementCount is the size of your Vector. This call will never return false. Your while loop with loop, adding elements, until the program runs out of memory. The cursor never moves forward because you never call next(). If you do call next() while adding elements directly to the Vector, you will get a ConcurrentModificationException.
I have an immutable Iterable<X> with a large number of elements. (it happens to be a List<> but never mind that.)
What I would like to do is start a few parallel / asynchronous tasks to iterate over the Iterable<> with the same iterator, and I'm wondering what interface I should use.
Here's a sample implementation with the to-be-determined interface QuasiIteratorInterface:
public void process(Iterable<X> iterable)
{
QuasiIteratorInterface<X> qit = ParallelIteratorWrapper.iterate(iterable);
for (int i = 0; i < MAX_PARALLEL_COUNT; ++i)
{
SomeWorkerClass worker = new SomeWorkerClass(qit);
worker.start();
}
}
class ParallelIteratorWrapper<T> implements QuasiIteratorInterface<T>
{
final private Iterator<T> iterator;
final private Object lock = new Object();
private ParallelIteratorWrapper(Iterator<T> iterator) {
this.iterator = iterator;
}
static public <T> ParallelIteratorWrapper<T> iterate(Iterable<T> iterable)
{
return new ParallelIteratorWrapper(iterable.iterator());
}
private T getNextItem()
{
synchronized(lock)
{
if (this.iterator.hasNext())
return this.iterator.next();
else
return null;
}
}
/* QuasiIteratorInterface methods here */
}
Here's my problem:
it doesn't make sense to use Iterator directly, since hasNext() and next() have a synchronization problem, where hasNext() is useless if someone else calls next() before you do.
I'd love to use Queue, but the only method I need is poll()
I'd love to use ConcurrentLinkedQueue to hold my large number of elements... except I may have to iterate through the elements more than once, so I can't use that.
Any suggestions?
Create your own Producer interface with the poll() method or equivalent (Guava's Supplier for instance). The implementation options are many but if you have an immutable random access list then you can simply maintain a thread-safe monotonic counter (AtomicInteger for instance) and call list.get(int) eg:
class ListSupplier<T> implements Supplier<T> {
private final AtomicInteger next = new AtomicInteger();
private final List<T> elements; // ctor injected
…
public <T> get() {
// real impl more complicated due to bounds checks
// and what to do when exhausted
return elements.get(next.getAndIncrement());
}
}
That is thread-safe, but you'd probably want to either return an Option style thing or null when exhausted.
Have one dispatcher thread that iterates over Iterable and dispatches elements to multiple worker threads that perform the work on the elements. You can use ThreadPoolExecutor to automate this.
The another question is about synchronized. I have also a run() in class Note,because i want to output each element in notes every 5 minutes. But i get always exception:java.util.ConcurrentModificationException,if i try to make more meetings in main. so i applay synchronized to the list notes which may be added a new meeting when i iterate over notes.My run method like this:
Is it correct way hier on list notes to synchronized to prevent ConcurrentModificationException ?(In my program it works.I get never this exception now)
A Meeting class and Note class may likes this:
public class Meeting{
public Meeting(Note note_1,Note note_2){
note_1.addElement(this);
note_2.addElement(this);}
//another method hier
}
public class Note implements Runnable{
public final List<Meeting> notes = new ArrayList<Meeting>();
public void addElement(Meeting n){
entries.add(n);
}
#Override
public void run(){
while(true) {
for(Meeting n : notes){
System.out.println(n.toString);}
}
try{ Thread.sleep(10);}
}
}
}
I get always exception error about exception:java.util.ConcurrentModificationException if i try to make more Meeting in main ,So i changes a littel in class Note,likes this :
private static final List<Entry> entries = Collections.synchronizedList(new ArrayList<Entry>());
and in run()
#Override
public void run() {
while(true){
synchronized(notes){
for(Entry n : entries){
//do something
}
}
try {
Thread.sleep(10);
} catch (InterruptedException e ) {
}
}
}
}
From the javadoc
Note that this exception does not always indicate that an object has been concurrently modified by a different thread. If a single thread issues a sequence of method invocations that violates the contract of an object, the object may throw this exception. For example, if a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will thow this exception.
THis means do not change your collection in a loop and iterate over it at the same time even in the same thread.
Read to what #Navi had written.
In a nutshell - NEVER remove/add elements of a collection in for each loop.
I once had that kind of problem and I decided to use http://code.google.com/p/google-collections/
There are some map/filter functions there (if I recall that methods were in Collections2 library).
If you are not willing to make the things right, you can always use the old-school iterator technique.
I've used a CopyOnWriteArrayList before when I encountered this sort of problem.
This makes a complete copy of the underlying array for each write, so it's not very efficient, but I've found it very useful for specific circumstances (e.g. a class which manages specialised event notification to some other classes)
This array never changes during the lifetime of the iterator, so interference is impossible and the iterator is guaranteed not to throw ConcurrentModificationException
i have created synchronized arrayList like this
import java.text.SimpleDateFormat;
import java.util.*;
class HelloThread
{
int i=1;
List arrayList;
public void go()
{
arrayList=Collections.synchronizedList(new ArrayList());
Thread thread1=new Thread(new Runnable() {
public void run() {
while(i<=10)
{
arrayList.add(i);
i++;
}
}
});
thread1.start();
Thread thred2=new Thread(new Runnable() {
public void run() {
while(true)
{
Iterator it=arrayList.iterator();
while(it.hasNext())
{
System.out.println(it.next());
}
}
}
});
thred2.start();
}
}
public class test
{
public static void main(String[] args)
{
HelloThread hello=new HelloThread();
hello.go();
}
}
but getting exception like this
Exception in thread "Thread-1" java.util.ConcurrentModificationException
anything wrong in my approach ?
Iterator of synchronizedList is not (and can't be) synchronized, you need to synchronize on the list manually while iterating (see javadoc):
synchronized(arrayList) {
Iterator it=arrayList.iterator();
while(it.hasNext()) {
System.out.println(it.next());
}
}
Another approach is to use a CopyOnWriteArrayList instead of Collections.synchronizedList(). It implements a copy-on-write semantic and therefore doesn't require synchronization.
Consider using a CopyOnWriteArrayList which is thread-safe. Every time you add an item, a fresh copy of the underlying array is created. However, the iterator will not reflect additions to the list since the iterator was created, but is guaranteed not to throw ConcurrentModificationException.
arrayList=new CopyOnWriteArrayList();
Other answers have identified the problem:
The iterators for synchronized collections are not synchronized. In fact, they are simply the iterators returned by the collection objects inside the wrapper classes.
Many collection classes (including ArrayList) use a fail-fast mechanism to detect concurrent modifications during iteration. This behavior is clearly documented in the javadocs for the respective classes. This is what you are seeing.
Not all collection classes do this. For example, many of the java.util.Concurrent... collection classes allow concurrent modification during iteration, but relax the semantics of the iteration sequence so that the results of the modifications may or may not be apparent in the objects returned by the iterator.
The javadoc for the Collections.synchronizedList() explains how to synchronize the iterator. Basically you do this:
List list = Collections.synchronizedList(new ArrayList());
...
synchronized (list) {
Iterator i = list.iterator(); // Must be in synchronized block
while (i.hasNext())
foo(i.next());
}
(Aside: normally it is not safe to assume that doing something like this would work. In theory, the synchronized list could use a private lock object, and the synchronized statement would not lock out concurrent modifications. However the javadocs say that this is what to do in this case ... so it is safe.)
The problem with doing that is that locking the collection creates a potential concurrency bottleneck. The alternative to is to use a copy-on-write data structure that internally makes a copy of the relevant parts of the collection. This approach means that an iterator sees sees a snapshot of the collection. Modifications may be made to the collection concurrent with an iteration, but the iterator does not see them. The problem with copy-on-write is that modifications are potentially a lot more expensive.
Ultimately, you need to balance the characteristics and costs of the different collection types wrt concurrent modification versus your actual requirements. Can you get away with the iterator not seeing all concurrent modifications?
The java.util.ConcurrentModificationException occurs when you manipulate (add,remove) a collection while iterating over the same collection.
You probably want to consume the create entries in your second thread while after they have been created by your first thread. So you could use ArrayLists get( index ) and size() for control
As Spike said, you can't modify a collection while iterating it. However, I think the solution is to lock the list while iterating.
class HelloThread
{
int i=1;
List arrayList;
public void go()
{
arrayList=Collections.synchronizedList(new ArrayList());
Thread thread1=new Thread(new Runnable() {
public void run() {
while(i<=10)
{
synchronized(someLock) {
arrayList.add(i);
}
i++;
}
}
});
thread1.start();
Thread thred2=new Thread(new Runnable() {
public void run() {
while(true)
{
synchronized(someLock) {
Iterator it=arrayList.iterator();
while(it.hasNext())
{
System.out.println(it.next());
}
}
}
}
});
thred2.start();
}
}
public class test
{
public static void main(String[] args)
{
HelloThread hello=new HelloThread();
hello.go();
}
}
I'm not sure what you're trying to do, so I hope this doesn't break the functionality of your code.
You may not modify a Collection that you are iterating. You can work around this by accessing the array entries by index, not through an Iterator. I can provide more advice if you tell me the problem that you are trying to solve with this code.
Let's take a normal list (implemented by the ArrayList class) and make it synchronized. This is shown in the SynchronizedArrayList class.
We pass the Collections.synchronizedList method a new ArrayList of Strings. The method returns a synchronized List of Strings.
//Here is SynchronizedArrayList class
package com.mnas.technology.automation.utility;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import org.apache.log4j.Logger;
/**
*
* #author manoj.kumar
* #email kumarmanoj.mtech#gmail.com
*
*/
public class SynchronizedArrayList {
static Logger log = Logger.getLogger(SynchronizedArrayList.class.getName());
public static void main(String[] args) {
List<String> synchronizedList = Collections.synchronizedList(new ArrayList<String>());
synchronizedList.add("Aditya");
synchronizedList.add("Siddharth");
synchronizedList.add("Manoj");
// when iterating over a synchronized list, we need to synchronize access to the synchronized list
synchronized (synchronizedList) {
Iterator<String> iterator = synchronizedList.iterator();
while (iterator.hasNext()) {
log.info("Synchronized Array List Items: " + iterator.next());
}
}
}
}
Notice that when iterating over the list, this access is still done using a synchronized block that locks on the synchronizedList object.
In general, iterating over a synchronized collection should be done in a synchronized block
ArrayList is non-synchronized collection and should not be used in concurrent environment without explicit synchronization. To synchronize ArrayList, we can use two methods provided by JDK
Collections.synchronizedList() method – It returns synchronized list backed by the specified list. It is recommended that we should manually synchronize on the returned list when traversing it via Iterator, Spliterator or Stream. Else it may result in non-deterministic behavior. No explicit synchronization is needed to add, remove elements from synchronized arraylist
CopyOnWriteArrayList class – It is a thread-safe variant of ArrayList.
Since you are using iterator without explicit synchronization you are getting the error