We Keep Coding

What is a thread-safe List implementation that allows Collections.sort

I have to write a program that requires a list. This list needs to be thread-safe in its implementation (mostly to avoid ConcurrentModificationException) but ALSO needs to allow the
Collections.sort() method to be applied, for API reasons.
CopyOnWriteArrayList fulfills the former, but not the latter, and other implementations I can find allow the latter but not the former.
Does Java have a list implementation that will work for me?
EDIT: An important point to note is that unfortunately my code needs to be Java 6 compatible.

I am wondering if this actually possible on a conceptual level: for a sort operation to be consistent, I would expect that the whole list is blocked for any adds/removes while the sorting is going on.
But Collections.sort() has no idea that it would need to lock the whole list while doing its work. You give it a list, and if another thread is trying to modify the list at the same time ... good luck with that.
Or if you reverse the point of view: how should a "thread-safe" list understand that it is right now in the process of being sorted; so - some accesses (like swapping elements) are fine; but others (like adding/removing) elements are not?!
In other words: I think you can only do this: pick any of the "thread-safe" list implementations; and then you have to put your own wrapper in place that
"Locks" the list for changes
Does the sorting work
"Unlocks" the list
And of course, for "2."; you are free to turn to Collections.sort().
Or, if you are using Java8 - you use the CopyOnWriteArrayList and its already implemented sort() method (which is kind of proving my point: you can only do proper sorting if you own the list while running the sort operation!).
Giving your latest comment: of course, you could manually "backport" the Java8 version of CopyOnWriteArrayList into your environment and use that; but of course, that won't help; as I understand that Java6-Collections.sort() will not call the new sort() method from that class.
So, it seems that the sum of your requirements can't be resolved; and you will have to bite the bullet and doing most of that in your own code.

Well, CopyOnWriteArrayList locks the entire collection (for insertion) while sorting. No?
Looks like you are good with CopyOnWriteArrayList. Below is the snippet from this class -
public void sort(Comparator<? super E> c) {
final ReentrantLock lock = this.lock;
lock.lock();**
try {
Object[] elements = getArray();
Object[] newElements = Arrays.copyOf(elements, elements.length);
#SuppressWarnings("unchecked") E[] es = (E[])newElements;
Arrays.sort(es, c);
setArray(newElements);
} finally {
lock.unlock();
}
}
Hmm.... since you've updated the question that code needs to be Java6 compatible, I'd say that you should extend the normal list and make use of https://docs.oracle.com/javase/6/docs/api/java/util/concurrent/locks/ReadWriteLock.html. In this type of lock, readers are not blocked from reading even when some other thread has acquired writeLock already, and 2 threads can acquire 'read' lock simultaneously.
Btw, this technique will require your caller to know that Collection.sort(...) shouldn't be called since you will have to expose explicit sort() method on your list. Hmm.... not sure if this was helpful.

Can iterator.next() remove elements from the source? (Java)

There is a Java function
public <T> void batchWrite(Iterable<T> source, int number)
that writes a large number of items in a time-efficient way.
I want to use this batchWrite() on a ConcurrentLinkedQueue that may be written to while batchWrite() works. And I want batchWrite() to delete the items that it takes from the queue.
I can write an iterator (and wrap it into an Iterable) that will delete the returned items:
class IteratorThatRemovesReturnedValues<T> implements Iterator<T> {
Queue<T> queue;
IteratorThatRemovesReturnedValues(Queue<T> q) { queue = q; }
boolean hasNext() { return queue.peek() != null; }
T next() { return queue.poll(); }
}
The question is: will not that be an abuse of the concept?
The description of remove() says:
The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this (remove()) method.
This may be read in either way: a particular implementation of Iterator may either be or not be allowed to define what happens when the underlying wait-free queue is modified in some specific way.
Will removing returned elements from the underlying queue contradict the iterator's contract?
(The alternative is to poll() an amount of items into an auxiliary ArrayList and invoke batchWrite() on that list.)
EDIT The other side of this question: can an iterator be backed by a pipe?

Assuming that you're not considering the (many) caveats of the Iterator#remove() method, the behavior of an iterator, according to the contract, is simple. Particularly, it does not say anything about the underlying implementation.
A side note: Historically, Iterator was associated with the iteration over a Collection, but this may just be a remedy to the legacy Enumeration class. There are many Iterator implementations that are not related to a collection. (For example, Scanner, and many more). Nowadays, an Iterator is often really not much more than an "abstract source of things that allows you to query whether there are more things available".
But strictly speaking, your implementation already does not obey the contract. The documentation of the Iterator#next() method says:
Throws:
NoSuchElementException - if the iteration has no more elements
This is not the case for you. Your implementation would simply return null here. Of course, this could easily be alleviated by replacing the poll() call with a call to remove(), which would conveniently throw an NoSuchElementException, but I guess that would contradict your current goals.
I think that the cructial point is that your implementation of the next() method has a side-effect that will be visible to the outside world. One example was already given in the comments: When you create two of these iterator instances, then both may behave according to their contract, but the behavior of both together may differ from what one would expect.
The latter only refers to single-threaded usage. When two threads are independently using these iterators, then an interweaving of calls to hasNext() and next() may induce a race condition. (Even more when two threads are using a single iterator, but this is the case for all iterators, because they usually are stateful anyhow).
The bottom line is: Although some details of the intended usage are still not clear (particularly regarding the question of how exactly the iterator is used in the batchWrite method, and how the multiple threads are supposed to interact here), this way of implementing it is likely to break sooner or later, and the bugs will be hard to detect and reproduce. I'd recommend to consider alternative implementations here.

naming methods with different return types but the same functionality

I'm talking about Java, but the concept would apply to other languages --
We have two methods on the same class that will have the same basic functionality, but provide different return types. Both methods will get you all the things, but one gets an iterable which can provide access to all things, and one gets a collection which includes all things.
We know you cannot do this (because it won't compile):
Iterable getThings()
Collection getThings()
So, does anyone have thoughts about how to name the methods? It seems an easy fix to say:
Iterable getIterableOfThings
Collection getCollectionOfThings
However, this is obviously somewhat verbose and perhaps not the best solution. Does the question I'm asking imply that our organization/coding approach is somehow off the mark, or is this a valid question?
Has anyone else dealt with this scenario?

How about:
Iterable getIterator();
Collection getThings();

Iterable is a superinterface of Collection. Generally speaking you shouldn't need to do this.
Nonetheless, the question itself is valid. A better example is when you have, say, an underlying resource, and one function wrapping it in one access object and the second in another. It depends on coding style, of course, but I for one am used to seeing something like this (and thus find it more readable):
class A {
private Object[] underlying;
List<Object> getItemsAsList() {
return Arrays.asList(underlying);
}
Object[] getItemsAsArray() {
return underlying;
}
}

Iterable getThings() does not seem correct as you are not actually getting anything. Iterable provides access to iterate over a set of items but does not give you those items.
A collection will give you those items, so Collection getThings() seems right.
For example you can be given an iterator for an infinitely long set but no such equivalent collection could be given. This would work because the iterator would return an element at time but a collection would need to collect the infinite amount of objects and return them.

Java: why can't iterate over an iterator?

I read Why is Java's Iterator not an Iterable? and Why aren't Enumerations Iterable?, but I still don't understand why this:
void foo(Iterator<X> it) {
for (X x : it) {
bar(x);
baz(x);
}
}
was not made possible. In other words, unless I'm missing something, the above could have been nice and valid syntactic sugar for:
void foo(Iterator<X> it) {
for (X x; it.hasNext();) {
x = it.next();
bar(x);
baz(x);
}
}

Most likely the reason for this is because iterators are not reusable; you need to get a fresh Iterator from the Iterable collection each time you want to iterate over the elements. However, as a quick fix:
private static <T> Iterable<T> iterable(final Iterator<T> it){
return new Iterable<T>(){ public Iterator<T> iterator(){ return it; } };
}
//....
{
// ...
// Now we can use:
for ( X x : iterable(it) ){
// do something with x
}
// ...
}
//....
That said, the best thing to do is simply pass around the Iterable<T> interface instead of Iterator<T>

but I still don't understand why this [...] was not made possible.
I can see several reasons:
Iterators are not reusable, so a for/each would consume the iterator - not incorrect behavior, perhaps, but unintuitive to those who don't know how the for/each is desugared.
Iterators don't appear "naked" in code all that often so it would be complicating the JLS with little gain (the for/each construct is bad enough as it is, working on both Iterables and arrays).
There's an easy workaround. It may seem a little wasteful to allocate a new object just for this, but allocation is cheap as it is and escape analysis would rid you even of that small cost in most cases. (Why they didn't include this workaround in an Iterables utility class, analogous to Collections and Arrays, is beyond me, though.)
(Probably not true - see the comments.) I seem to recall that the JLS can only reference things in java.lang[citation needed], so they'd have to create an Iterator interface in java.lang which java.util.Iterator extends without adding anything to. Now we have two functionally equivalent iterator interfaces. 50% of the new code using naked iterators will choose the java.lang version, the rest use the one in java.util. Chaos ensues, compatibility problems abound, etc.
I think points 1-3 are very much in line with how the Java language design philosophy seems to go: Don't surprise newcomers, don't complicate the spec if it doesn't have a clear gain that overshadows the costs, and don't do with a language feature what can be done with a library.
The same arguments would explain why java.util.Enumeration isn't Iterable, too.

The for(Type t : iterable) syntax is only valid for classes that implement Iterable<Type>.
An iterator does not implement iterable.
You can iterate over things like Collection<T>, List<T>, or Set<T> because they implement Iterable.
The following code is equivalent:
for (Type t: list) {
// do something with t
}
and
Iterator<Type> iter = list.iterator();
while (iter.hasNext()) {
t = iter.next();
// do something with t
}
The reason this was not made possible, is because the for-each syntax was added to the language to abstract out the Iterator. Making the for-each loop work with iterators would not accomplish what the for-each loop was created for.

Actually, you can.
There is very short workaround available on java 8:
for (X item : (Iterable<X>) () -> iterator)
See How to iterate with foreach loop over java 8 stream for the detailed explanation of the trick.
And some explanations why this was not natively supported can be found in related question:
Why does Stream<T> not implement Iterable<T>?

Iterators are not meant be reused (i.e.: used in more than one iteration loop). In particular, Iterator.hasNext() guarantees that you can safely call Iterator.next() and indeed get the next value from the underlying collection.
When the same iterator is used in two concurrently running iterations (let's assume a multi-threading scenario), this promise can no longer be kept:
while(iter.hasNext() {
// Now a context switch happens, another thread is performing
// iter.hasNext(); x = iter.next();
String s = iter.next();
// A runtime exception is thrown because the iterator was
// exhausted by the other thread
}
Such scenarios completely break the protocol offered by Iterator. Actually, they can occur even in a single threaded program: an iteration loop calls another method which uses the same iterator to perform its own iteration. When this method returns, the caller is issuing an Iterator.next() call which, again, fails.

Because the for-each is designed to read as something like:
for each element of [some collection of elements]
An Iterator is not [some collection of elements]. An array and an Iterable is.

What are the benefits of the Iterator interface in Java?

I just learned about how the Java Collections Framework implements data structures in linked lists. From what I understand, Iterators are a way of traversing through the items in a data structure such as a list. Why is this interface used? Why are the methods hasNext(), next() and remove() not directly coded to the data structure implementation itself?
From the Java website: link text
public interface Iterator<E> An
iterator over a collection. Iterator
takes the place of Enumeration in the
Java collections framework. Iterators
differ from enumerations in two ways:
Iterators allow the caller to remove
elements from the underlying
collection during the iteration with
well-defined semantics. Method names
have been improved. This interface is
a member of the Java Collections
Framework.
I tried googling around and can't seem to find a definite answer. Can someone shed some light on why Sun chose to use them? Is it because of better design? Increased security? Good OO practice?
Any help will be greatly appreciated. Thanks.

Why is this interface used?
Because it supports the basic operations that would allow a client programmer to iterate over any kind of collection (note: not necessarily a Collection in the Object sense).
Why are the methods... not directly
coded to the data structure
implementation itself?
They are, they're just marked Private so you can't reach into them and muck with them. More specifically:
You can implement or subclass an Iterator such that it does something the standard ones don't do, without having to alter the actual object it iterates over.
Objects that can be traversed over don't need to have their interfaces cluttered up with traversal methods, in particular any highly specialized methods.
You can hand out Iterators to however many clients you wish, and each client may traverse in their own time, at their own speed.
Java Iterators from the java.util package in particular will throw an exception if the storage that backs them is modified while you still have an Iterator out. This exception lets you know that the Iterator may now be returning invalid objects.
For simple programs, none of this probably seems worthwhile. The kind of complexity that makes them useful will come up on you quickly, though.

You ask: "Why are the methods hasNext(), next() and remove() not directly coded to the data structure implementation itself?".
The Java Collections framework chooses to define the Iterator interface as externalized to the collection itself. Normally, since every Java collection implements the Iterable interface, a Java program will call iterator to create its own iterator so that it can be used in a loop. As others have pointed out, Java 5 allows us to direct usage of the iterator, with a for-each loop.
Externalizing the iterator to its collection allows the client to control how one iterates through a collection. One use case that I can think of where this is useful is when one has an an unbounded collection such as all the web pages on the Internet to index.
In the classic GoF book, the contrast between internal and external iterators is spelled out quite clearly.
A fundamental issue is deciding which party conrols the iteration, the iterator or the client that uses the iterator. When the client controls the iteration, the iterator is called an external iterator, and when the iterator controls it, the iterator is an internal iterator. Clients that use an external iterator must advance the traversal and request the next element explicitly from the iterator. In contrast, the client hands an internal iterator an operation to perform, and the iterator applies that operation to every element ....
External iterators are more flexible than internal iterators. It's easy to compare two collections for equality with an external iterator, for example, but it's practically impossible with internal iterators ... But on the other hand, internal iterators are easier to use, because they define the iteration logic for you.
For an example of how internal iterators work, see Ruby's Enumerable API, which has internal iteration methods such as each. In Ruby, the idea is to pass a block of code (i.e. a closure) to an internal iterator so that a collection can take care of its own iteration.

it is important to keep the collection apart from the pointer. the iterator points at a specific place in a collection, and thus is not an integral part of the collection. this way, for an instance, you can use several iterators over the same collection.
the down-side of this seperation is that the iterator is not aware to changes made to the collection it iterates on. so you cannot change the collection's structure and expect the iterator to continue it's work without "complaints".

Using the Iterator interface allows any class that implements its methods to act as iterators. The notion of an interface in Java is to have, in a way, a contractual obligation to provide certain functionalities in a class that implements the interface, to act in a way that is required by the interface. Since the contractual obligations must be met in order to be a valid class, other classes which see the class implements the interface and thus reassured to know that the class will have those certain functionalities.
In this example, rather than implement the methods (hasNext(), next(), remove()) in the LinkedList class itself, the LinkedList class will declare that it implements the Iterator interface, so others know that the LinkedList can be used as an iterator. In turn, the LinkedList class will implement the methods from the Iterator interface (such as hasNext()), so it can function like an iterator.
In other words, implementing an interface is a object-oriented programming notion to let others know that a certain class has what it takes to be what it claims to be.
This notion is enforced by having methods that must be implemented by a class that implements the interface. This makes sure that other classes that want to use the class that implements the Iterator interface that it will indeed have methods that Iterators should have, such as hasNext().
Also, it should be noted that since Java does not have multiple inheritance, the use of interface can be used to emulate that feature. By implementing multiple interfaces, one can have a class that is a subclass to inherit some features, yet also "inherit" the features of another by implementing an interface. One example would be, if I wanted to have a subclass of the LinkedList class called ReversibleLinkedList which could iterate in reverse order, I may create an interface called ReverseIterator and enforce that it provide a previous() method. Since the LinkedList already implements Iterator, the new reversible list would have implemented both the Iterator and ReverseIterator interfaces.
You can read more about interfaces from What is an Interface? from The Java Tutorial from Sun.

Multiple instances of an interator can be used concurrently. Approach them as local cursors for the underlying data.
BTW: favoring interfaces over concrete implementations looses coupling
Look for the iterator design pattern, and here: http://en.wikipedia.org/wiki/Iterator

Because you may be iterating over something that's not a data structure. Let's say I have a networked application that pulls results from a server. I can return an Iterator wrapper around those results and stream them through any standard code that accepts an Iterator object.
Think of it as a key part of a good MVC design. The data has to get from the Model (i.e. data structure) to the View somehow. Using an Iterator as a go-between ensures that the implementation of the Model is never exposed. You could be keeping a LinkedList in memory, pulling information out of a decryption algorithm, or wrapping JDBC calls. It simply doesn't matter to the view, because the view only cares about the Iterator interface.

An interesting paper discussing the pro's and con's of using iterators:
http://www.sei.cmu.edu/pacc/CBSE5/Sridhar-cbse5-final.pdf

I think it is just good OO practice. You can have code that deals with all kinds of iterators, and even gives you the opportunity to create your own data structures or just generic classes that implement the iterator interface. You don't have to worry about what kind of implementation is behind it.

Just M2C, if you weren't aware: you can avoid directly using the iterator interface in situations where the for-each loop will suffice.

Ultimately, because Iterator captures a control abstraction that is applicable to a large number of data structures. If you're up on your category theory fu, you can have your mind blown by this paper: The Essence of the Iterator Pattern.

Well it seems like the first bullet point allows for multi-threaded (or single threaded if you screw up) applications to not need to lock the collection for concurrency violations. In .NET for example you cannot enumerate and modify a collection (or list or any IEnumerable) at the same time without locking or inheriting from IEnumerable and overriding methods (we get exceptions).

Iterator simply adds a common way of going over a collection of items. One of the nice features is the i.remove() in which you can remove elements from the list that you are iterating over. If you just tried to remove items from a list normally it would have weird effects or throw and exception.
The interface is like a contract for all things that implement it. You are basically saying.. anything that implements an iterator is guaranteed to have these methods that behave the same way. You can also use it to pass around iterator types if that is all you care about dealing with in your code. (you might not care what type of list it is.. you just want to pass an Iterator) You could put all these methods independently in the collections but you are not guaranteeing that they behave the same or that they even have the same name and signatures.

Iterators are one of the many design patterns available in java. Design patterns can be thought of as convenient building blocks, styles, usage of your code/structure.
To read more about the Iterator design pattern check out the this website that talks about Iterator as well as many other design patterns. Here is a snippet from the site on Iterator: http://www.patterndepot.com/put/8/Behavioral.html
The Iterator is one of the simplest
and most frequently used of the design
patterns. The Iterator pattern allows
you to move through a list or
collection of data using a standard
interface without having to know the
details of the internal
representations of that data. In
addition you can also define special
iterators that perform some special
processing and return only specified
elements of the data collection.

Iterators can be used against any sort of collection. They allow you to define an algorithm against a collection of items regardless of the underlying implementation. This means you can process a List, Set, String, File, Array, etc.
Ten years from now you can change your List implementation to a better implementation and the algorithm will still run seamlessly against it.

Iterator is useful when you are dealing with Collections in Java.
Use For-Each loop(Java1.5) for iterating over a collection or array or list.

The java.util.Iterator interface is used in the Java Collections Framework to allow modification of the collection while still iterating through it. If you just want to cleanly iterate over an entire collection, use a for-each instead, but a upside of Iterators is the functionality that you get: a optional remove() operation, and even better for the List Iterator interface, which offers add() and set() operations too. Both of these interfaces allow you to iterate over a collection and changing it structurally at the same time. Trying to modify a collection while iterating through it with a for-each would throw a ConcurrentModificationException, usually because the collection is unexpectedly modified!
Take a look at the ArrayList class
It has 2 private classes inside it (inner classes)
called Itr and ListItr
They implement Iterator and the ListIterator interfaces respectively
public class ArrayList..... { //enclosing class
private class Itr implements Iterator<E> {
public E next() {
return ArrayList.this.get(index++); //rough, not exact
}
//we have to use ArrayList.this.get() so the compiler will
//know that we are referring to the methods in the
//enclosing ArrayList class
public void remove() {
ArrayList.this.remove(prevIndex);
}
//checks for...co mod of the list
final void checkForComodification() { //ListItr gets this method as well
if (ArrayList.this.modCount != expectedModCount) {
throw new ConcurrentModificationException();
}
}
}
private class ListItr extends Itr implements ListIterator<E> {
//methods inherted....
public void add(E e) {
ArrayList.this.add(cursor, e);
}
public void set(E e) {
ArrayList.this.set(cursor, e);
}
}
}
When you call the methods iterator() and listIterator(), they return
a new instance of the private class Itr or ListItr, and since these inner classes are "within" the enclosing ArrayList class, they can freely modify the ArrayList without triggering a ConcurrentModificationException, unless you change the list at the same time (conccurently) through set() add() or remove() methods of the ArrayList class.