We Keep Coding

What's the best way to implement a list of elements that will have to have elements added/removed from different threads?

I'm currently trying to implement a system list that would run in a few different threads:
1) First thread is listening to incoming requests and adds them to the list.
2) A new thread is created for each request to perform certain operations.
3) Another thread iterates through the list, checks the status of each request, and removes them from the list when they're complete.
Now, the way I have it in a very simplified pseudocode can be viewed below:
private List<Job> runningJobs = new ArrayList<>(); // our list of requests
private Thread monitorThread;
private Runnable monitor = new Runnable() { // this runnable is later called in a new thread to monitor the list and remove completed requests
#Override
public void run() {
boolean monitorRun = true;
while(monitorRun) {
try {
Thread.sleep(1000);
if (runningJobs.size()>0){
Iterator<Job> i = runningJobs.iterator();
while (i.hasNext()) {
try {
Job job = i.next();
if (job.jobStatus() == 1) { // if job is complete
i.remove();
}
}
catch (java.util.ConcurrentModificationException e){
e.printStackTrace();
}
}
}
if (Thread.currentThread().isInterrupted()){
monitorRun = false;
}
} catch (InterruptedException e) {
monitorRun = false;
}
}
}
};
private void addRequest(Job job){
this.runningJobs.add(newJob);
// etc
}
In short, the Runnable monitor is what runs continuously in the third thread; the first thread is calling addRequest() occasionally.
While my current implementation somewhat works, I'm concerned about the order of operations here and possible java.util.ConcurrentModificationException (and the system is anything but robust). I'm certain there is a much better way to organize this mess.
What's the proper or a better way to do this?

Your requirements would be met nicely with an ExecutorService. For each request, create Job, and submit it to the service. Internally, the service uses a BlockingQueue, which would address your question directly, but you don't have to worry about it with an ExecutorService.
Specifically, something like this:
/* At startup... */
ExecutorService workers = Executors.newCachedThreadPool();
/* For each request... */
Job job = ... ;
workers.submit(job); /* Assuming Job implements Runnable */
// workers.submit(job::jobEntryPoint); /* If Job has some other API */
/* At shutdown... */
workers.shutdown();

There are a few different ways.
You can synchronize the list. This is possibly the most brute-force and still wouldn't help prevent an insert while you are iterating over it.
There are a few synchronized* collections. These tend to be better but have ramifications. For instance CopyOnWriteArrayList will work but it creates a new array list each time (that you would assign back to the variable). This is good for occasionally updated collections.
There is a ConcurrentLinkedQueue--Since it's "Linked" you can't reference an item in the middle.
Look through the implementations of the "List" interface and pick the one that best suits your problem.
If your problem is a queue instead of a list, there are a few implementations of that as well and they will tend to be better suited for that type of problem.
In general my answer is that you should probably scan through the Javadocs every time java does a major release and examine (at least) the new collections. You might be surprised at the stuff that's in there.

Better Performance with single thread

I'm doing a tree search for a class assignment. I understand the tree search part, but as I have some extra time, I wanted to speed it up by adding more threads.
The final task is to take in a set of constraints, classes and time-slots, and output a schedule with all those classes, and which satisfies all the constraints. An empty or partial assignment goes in, a complete class assignment comes out.
Our search is designed like a tree, with the input being the root node. The function div(n) is as follows: for a node n, find an unused class C, and for each unused slot S, produce a child node with C in S. To make the search more efficient, we use a search control which ranks the quality of nodes, so that the best candidates are selected first, and we don't waste time on bad candidates
A node implements Comparable, with compareTo() implemented using the search control. I use a priority queue to store nodes awaiting processing, so the 'best' nodes are always next in line. A worker removes an node, applies div() and adds the children to the priority queue.
My first approach was using a shared priority queue, specifically PriorityBlockingQueue. The performance was abysmal, since the queue was almost always blocking.
I tried to fix it by adding a background worker and a ConcurrentLinkedQueue buffer. workers would add to the buffer, and the the worker would periodically move elements from the buffer to the priority queue. This didn't work either.
The best performance I have found is to give each worker it's own priority queue. I'm guessing that this is as good as it gets, as now threads aren't connected to the actions of others. with this config, on an 4C/8T machine, I get a speedup of ~2.5. I think the bottleneck here is the allocation of memory for all these nodes, but I could be wrong here.
From Searcher:
private PriorityQueue<Schedule> workQueue;
private static volatile boolean shutdownSignal = false;
private Schedule best;
public Searcher(List<Schedule> instances) {
workQueue = new PriorityQueue<>(instances);
}
public static void stop() {
shutdownSignal = true;
}
/**
* Run the search control starting with the first node in the workQueue
*/
#Override
public void run() {
while (!shutdownSignal) {
try {
Schedule next = workQueue.remove();
List<Schedule> children = next.div(checkBest);
workQueue.addAll(children);
} catch (Exception e) {
//TODO: handle exception
}
}
//For testing
System.out.println("Shutting down: " + workQueue.size());
}
//passing a function as a parameter
Consumer<Schedule> checkBest = new Consumer<Schedule>() {
public void accept(Schedule sched) {
if (best == null || sched.betterThan(best)) {
best = sched;
Model.checkBest.accept(sched);
}
}
};
From Schedule:
public List<Schedule> div(Consumer<Schedule> completion) {
List<Schedule> n = new ArrayList<>();
int selected = 0;
List<Slot> available = Model.getSlots();
List<Slot> allocated = getAssigned();
while (allocated.get(selected) != null) {
selected++;
} // find first available slot to fill.
// Iterate through all available slots
for (Slot t : available) {
//Prepare a fresh copy
List<Slot> newAssignment = new ArrayList<>(allocated.size());
Collections.copy(newAssignment, allocated);
//assign the course to the timeslot
newAssignment.set(selected, t);
Schedule next = new Schedule(this, newAssignment);
n.add(next);
}
/**
* Filter out nodes which violate the hard constraints and which are solved,
* and check if they are the best in a calling thread
*/
List<Schedule> unsolvedNodes = new ArrayList<>();
for (Schedule schedule: n) {
if (schedule.constr() && !schedule.solved()){
unsolvedNodes.add(schedule);
completion.accept(schedule);
}
}
return unsolvedNodes;
}

I would say that fork-join framework is an appropriate tool for your task. You need to extend your task from either ResursiveTask or ResursiveAction and submit it to ForkJoinPool. Here is a pseudo-code sample. Also your shutdown flag must be volatile.
public class Task extends RecursiveAction {
private final Node<Integer> node;
public Task(Node<Integer> node) {
this.node = node;
}
#Override
protected void compute() {
// check result and stop recursion if needed
List<Task> subTasks = new ArrayList<>();
List<Node<Integer>> nodes = div(this.node);
for (Node<Integer> node : nodes) {
Task task = new Task(node);
task.fork();
subTasks.add(task);
}
for(Task task : subTasks) {
task.join();
}
}
public static void main(String[] args) {
Node root = getRootNode();
new ForkJoinPool().invoke(new Task(root));
}

Non-Blocking Queue with batch oriented drainAll

We have a specialist, multi-producer (User) and single-consumer (Engine), queue. The User threads runs more frequently and always adds individual elements to the queue. The Engine thread operation runs less frequently and processes the stack elements in a batch. If the stack is empty, it'll park until the User thread has added an entry. This way a notify only needs to happen when the queue goes from empty to 1.
In this implementation, instead of the Engine thread iterating and removing one item at a time, it removes them all - a drainAll, instead of drainTo. No other operations can mutate the stack - just the User thread add, and the engine thread drainAll.
Currently we do this via a synchronised linked list, we are wondering if there is a non-blocking way to do this. The drainTo operation on JDK classes will iterate the stack, we just want to take everything in the stack in one operation, without iterating - as each iteration hits volatile/cas related logic, so we'd ideally just like to hit that once, per drainAll. The the engine thread can iterate and operate on each individual element, without touching sync/volatile/cas operations.
The current implementation looks something like:
public class SynchronizedPropagationQueue implements PropagatioQueue {
protected volatile PropagationEntry head;
protected volatile PropagationEntry tail;
protected synchronized void addEntry( PropagationEntry entry ) {
if ( head == null ) {
head = entry;
notifyWaitOnRest();
} else {
tail.setNext( entry );
}
tail = entry;
}
#Override
public synchronized PropagationEntry drainAll() {
PropagationEntry currentHead = head;
head = null;
tail = null;
return currentHead;
}
public synchronized void waitOnRest() {
try {
log.debug("Engine wait");
wait();
} catch (InterruptedException e) {
// do nothing
}
log.debug("Engine resumed");
}
#Override
public synchronized void notifyWaitOnRest() {
notifyAll();
}
}
asdf

Stacks have a very simple non-blocking implementation that supports a concurrent "pop all" operation easily, and can easily detect the empty->non-empty transition. You could have all your producers push items onto a stack and then have the engine empty the whole thing at once. It looks like this:
public class EngineQueue<T>
{
private final AtomicReference<Node<T>> m_lastItem = new AtomicReference<>();
public void add(T item)
{
Node<T> newNode = new Node<T>(item);
do {
newNode.m_next = m_lastItem.get();
} while(!m_lastItem.compareAndSet(newNode.m_next, newNode));
if (newNode.m_next == null)
{
// ... just went non-empty signal any waiting consumer
}
}
public List<T> removeAll()
{
Node<T> stack = m_lastItem.getAndSet(null);
// ... wait for non-empty if necessary
List<T> ret = new ArrayList<>();
for (;stack != null; stack=stack.m_next)
{
ret.add(stack.m_data);
}
Collections.reverse(ret);
return ret;
}
private static class Node<U>
{
Node<U> m_next;
final U m_data;
Node(U data)
{
super();
m_data = data;
}
}
}
For signaling around the empty -> non-empty transition, you can use normal synchronization. This is not going to be expensive if you only do it when you detect an empty state... since you only get to the empty state when you're out of work to do.

Currently we do this via a synchronised linked list, we are wondering if there is a non-blocking way to do this. The drainTo operation on JDK classes will iterate the stack, we just want to take everything in the stack in one operation, without iterating
Maybe I don't understand but it seems like using a BlockingQueue.drainTo(...) method would be better than your implementation. For example the LinkedBlockingQueue.drainTo(...) method just has one lock around that method -- there's no iterating overhead that I see.
If this is not an academic discussion then I'd doubt that your performance problems are with the queue itself and would concentrate your efforts in other areas. If it is academic then #Matt's answer might be better although certainly there's a lot more code to be written to support the full Collection method list.

Non blocking function that preserves order

I have the following method:
void store(SomeObject o) {
}
The idea of this method is to store o to a permanent storage but the function should not block. I.e. I can not/must not do the actual storage in the same thread that called store.
I can not also start a thread and store the object from the other thread because store might be called a "huge" amount of times and I don't want to start spawning threads.
So I options which I don't see how they can work well:
1) Use a thread pool (Executor family)
2) In store store the object in an array list and return. When the array list reaches e.g. 1000 (random number) then start another thread to "flush" the array list to storage. But I would still possibly have the problem of too many threads (thread pool?)
So in both cases the only requirement I have is that I store persistantly the objects in exactly the same order that was passed to store. And using multiple threads mixes things up.
How can this be solved?
How can I ensure:
1) Non blocking store
2) Accurate insertion order
3) I don't care about any storage guarantees. If e.g. something crashes I don't care about losing data e.g. cached in the array list before storing them.

I would use a SingleThreadExecutor and a BlockingQueue.
SingleThreadExecutor as the name sais has one single Thread. Use it to poll from the Queue and persist objects, blocking if empty.
You can add not blocking to the queue in your store method.
EDIT
Actually, you do not even need that extra Queue - JavaDoc of newSingleThreadExecutor sais:
Creates an Executor that uses a single worker thread operating off an unbounded queue. (Note however that if this single thread terminates due to a failure during execution prior to shutdown, a new one will take its place if needed to execute subsequent tasks.) Tasks are guaranteed to execute sequentially, and no more than one task will be active at any given time. Unlike the otherwise equivalent newFixedThreadPool(1) the returned executor is guaranteed not to be reconfigurable to use additional threads.
So I think it's exactly what you need.
private final ExecutorService persistor = Executors.newSingleThreadExecutor();
public void store( final SomeObject o ){
persistor.submit( new Runnable(){
#Override public void run(){
// your persist-code here.
}
} );
}
The advantage of using a Runnable that has a quasi-endless-loop and using an extra queue would be the possibility to code some "Burst"-functionality. For example you could make it wait to persist only when 10 elements are in queue or the oldest element has been added at least 1 minute ago ...

I suggest using a Chronicle-Queue which is a library I designed.
It allows you to write in the current thread without blocking. It was originally designed for low latency trading systems. For small messages it takes around 300 ns to write a message.
You don't need to use a back ground thread, or a on heap queue and it doesn't wait for the data to be written to disk by default. It also ensures consistent order for all readers. If the program dies at any point after you call finish() the message is not lost. (Unless the OS crashes/loses power) It also supports replication to avoid data loss.

Have one separate thread that gets items from the end of a queue (blocking on an empty queue), and writes them to disk. Your main thread's store() function just adds items to the beginning of the queue.
Here's a rough idea (though I assume there will be cleaner or faster ways for doing this in production code, depending on how fast you need things to be):
import java.util.*;
import java.io.*;
import java.util.concurrent.*;
class ObjectWriter implements Runnable {
private final Object END = new Object();
BlockingQueue<Object> queue = new LinkedBlockingQueue();
public void store(Object o) throws InterruptedException {
queue.put(o);
}
public ObjectWriter() {
new Thread(this).start();
}
public void close() throws InterruptedException {
queue.put(END);
}
public void run() {
while (true) {
try {
Object o = queue.take();
if (o == END) {
// close output file.
return;
}
System.out.println(o.toString()); // serialize as appropriate
} catch (InterruptedException e) {
}
}
}
}
public class Test {
public static void main(String[] args) throws Exception {
ObjectWriter w = new ObjectWriter();
w.store("hello");
w.store("world");
w.close();
}
}

The comments in your question make it sound like you are unfamilier with multi-threading, but it's really not that difficult.
You simply need another thread responsible for writing to the storage which picks items off a queue. - your store function just adds the objects to the in-memory queue and continues on it's way.
Some psuedo-ish code:
final List<SomeObject> queue = new List<SomeObject>();
void store(SomeObject o) {
// add it to the queue - note that modifying o after this will also alter the
// instance in the queue
synchronized(queue) {
queue.add(queue);
queue.notify(); // tell the storage thread there's something in the queue
}
}
void storageThread() {
SomeObject item;
while (notfinished) {
synchronized(queue) {
if (queue.length > 0) {
item = queue.get(0); // get from start to ensure same order
queue.removeAt(0);
} else {
// wait for something
queue.wait();
continue;
}
}
writeToStorage(item);
}
}

Java concurrency - improving a copy-on-read collection

I have a multithreaded application, where a shared list has write-often, read-occasionally behaviour.
Specifically, many threads will dump data into the list, and then - later - another worker will grab a snapshot to persist to a datastore.
This is similar to the discussion over on this question.
There, the following solution is provided:
class CopyOnReadList<T> {
private final List<T> items = new ArrayList<T>();
public void add(T item) {
synchronized (items) {
// Add item while holding the lock.
items.add(item);
}
}
public List<T> makeSnapshot() {
List<T> copy = new ArrayList<T>();
synchronized (items) {
// Make a copy while holding the lock.
for (T t : items) copy.add(t);
}
return copy;
}
}
However, in this scenario, (and, as I've learned from my question here), only one thread can write to the backing list at any given time.
Is there a way to allow high-concurrency writes to the backing list, which are locked only during the makeSnapshot() call?

synchronized (~20 ns) is pretty fast and even though other operations can allow concurrency, they can be slower.
private final Lock lock = new ReentrantLock();
private List<T> items = new ArrayList<T>();
public void add(T item) {
lock.lock();
// trivial lock time.
try {
// Add item while holding the lock.
items.add(item);
} finally {
lock.unlock();
}
}
public List<T> makeSnapshot() {
List<T> copy = new ArrayList<T>(), ret;
lock.lock();
// trivial lock time.
try {
ret = items;
items = copy;
} finally {
lock.unlock();
}
return ret;
}
public static void main(String... args) {
long start = System.nanoTime();
Main<Integer> ints = new Main<>();
for (int j = 0; j < 100 * 1000; j++) {
for (int i = 0; i < 1000; i++)
ints.add(i);
ints.makeSnapshot();
}
long time = System.nanoTime() - start;
System.out.printf("The average time to add was %,d ns%n", time / 100 / 1000 / 1000);
}
prints
The average time to add was 28 ns
This means if you are creating 30 million entries per second, you will have one thread accessing the list on average. If you are creating 60 million per second, you will have concurrency issues, however you are likely to be having many more resourcing issue at this point.
Using Lock.lock() and Lock.unlock() can be faster when there is a high contention ratio. However, I suspect your threads will be spending most of the time building the objects to be created rather than waiting to add the objects.

You could use a ConcurrentDoublyLinkedList. There is an excellent implementation here ConcurrentDoublyLinkedList.
So long as you iterate forward through the list when you make your snapshot all should be well. This implementation preserves the forward chain at all times. The backward chain is sometimes inaccurate.

First of all, you should investigate if this really is too slow. Adds to ArrayLists are O(1) in the happy case, so if the list has an appropriate initial size, CopyOnReadList.add is basically just a bounds check and an assignment to an array slot, which is pretty fast. (And please, do remember that CopyOnReadList was written to be understandable, not performant.)
If you need a non-locking operation, you can have something like this:
class ConcurrentStack<T> {
private final AtomicReference<Node<T>> stack = new AtomicReference<>();
public void add(T value){
Node<T> tail, head;
do {
tail = stack.get();
head = new Node<>(value, tail);
} while (!stack.compareAndSet(tail, head));
}
public Node<T> drain(){
// Get all elements from the stack and reset it
return stack.getAndSet(null);
}
}
class Node<T> {
// getters, setters, constructors omitted
private final T value;
private final Node<T> tail;
}
Note that while adds to this structure should deal pretty well with high contention, it comes with several drawbacks. The output from drain is quite slow to iterate over, it uses quite a lot of memory (like all linked lists), and you also get things in the opposite insertion order. (Also, it's not really tested or verified, and may actually suck in your application. But that's always the risk with using code from some random dude on the intertubes.)

Yes, there is a way. It is similar to the way ConcurrentHashMap made, if you know.
You should make your own data structure not from one list for all writing threads, but use several independent lists. Each of such lists should be guarded by it's own lock. .add() method should choose list for append current item based on Thread.currentThread.id (for example, just id % listsCount). This will gives you good concurrency properties for .add() -- at best, listsCount threads will be able to write without contention.
On makeSnapshot() you should just iterate over all lists, and for each list you grab it's lock and copy content.
This is just an idea -- there are many places to improve it.

You can use a ReadWriteLock to allow multiple threads to perform add operations on the backing list in parallel, but only one thread to make the snapshot. While the snapshot is being prepared all other add and snapshot request are put on hold.
A ReadWriteLock maintains a pair of associated locks, one for
read-only operations and one for writing. The read lock may be held
simultaneously by multiple reader threads, so long as there are no
writers. The write lock is exclusive.
class CopyOnReadList<T> {
// free to use any concurrent data structure, ConcurrentLinkedQueue used as an example
private final ConcurrentLinkedQueue<T> items = new ConcurrentLinkedQueue<T>();
private final ReadWriteLock rwLock = new ReentrantReadWriteLock();
private final Lock shared = rwLock.readLock();
private final Lock exclusive = rwLock.writeLock();
public void add(T item) {
shared.lock(); // multiple threads can attain the read lock
// try-finally is overkill if items.add() never throws exceptions
try {
// Add item while holding the lock.
items.add(item);
} finally {
shared.unlock();
}
}
public List<T> makeSnapshot() {
List<T> copy = new ArrayList<T>(); // probably better idea to use a LinkedList or the ArrayList constructor with initial size
exclusive.lock(); // only one thread can attain write lock, all read locks are also blocked
// try-finally is overkill if for loop never throws exceptions
try {
// Make a copy while holding the lock.
for (T t : items) {
copy.add(t);
}
} finally {
exclusive.unlock();
}
return copy;
}
}
Edit:
The read-write lock is so named because it is based on the readers-writers problem not on how it is used. Using the read-write lock we can have multiple threads achieve read locks but only one thread achieve the write lock exclusively. In this case the problem is reversed - we want multiple threads to write (add) and only thread to read (make the snapshot). So, we want multiple threads to use the read lock even though they are actually mutating. Only thread is exclusively making the snapshot using the write lock even though snapshot only reads. Exclusive means that during making the snapshot no other add or snapshot requests can be serviced by other threads at the same time.
As #PeterLawrey pointed out, the Concurrent queue will serialize the writes aqlthough the locks will be used for as minimal a duration as possible. We are free to use any other concurrent data structure, e.g. ConcurrentDoublyLinkedList. The queue is used only as an example. The main idea is the use of read-write locks.