We Keep Coding

is there a way to make a multithread that only calculates on request in java?

I tried out multithreading for a project I'm making. in the project I need to do a certain calculation multiple times every time I call for a certain function. I tried making some testing code to understand how to do it, but I can't get it to work properly (the code seems to work perfectly when I debug it, but if I run it normally it doesn't work past the first cycle).
in the code there is an endless loop that mimics my project's calling for a function multiple times. I tried to do it so the thread runs while changeflag is true, and change the flag to false after every run of the calculation so it would stop from calculating it again and again, and after "calling" the function I change it to true back, so it would be able to calculate again.
following is my code:
import java.util.ArrayList;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.SynchronousQueue;
public class Main {
public static void main(String[] args) throws InterruptedException {
BlockingQueue<Result> queue = new SynchronousQueue<>();
int loops = 0;
MyThread[] arr = new MyThread[10];
ArrayList<Result> ress = new ArrayList<>();
for (int i = 0; i < arr.length; i++) {
arr[i] = new MyThread(i, queue);
arr[i].start();
}
while (true) {
System.out.println(loops++);
while (ress.size() < arr.length){
ress.add(queue.take());
}
while (!ress.isEmpty()){
arr[ress.get(0).getSign()].setChangeflag(true);
ress.remove(0);
}
}
}
}
import java.util.Random;
import java.util.concurrent.BlockingQueue;
public class MyThread extends Thread{
private boolean changeflag = true;
private boolean runflag = true;
private int sign;
private BlockingQueue<Result> queue;
Random rnd = new Random();
public MyThread(int sign, BlockingQueue<Result> queue){
this.sign = sign;
this.queue = queue;
}
public void run(){
while (runflag){
if(changeflag){
changeflag = false;
try {
queue.put(sense());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
public Result sense(){
return new Result( rnd.nextInt(10), sign);
}
public synchronized void setChangeflag(boolean changeflag) {
this.changeflag = changeflag;
}
}
public class Result {
private double res;
private int sign;
public Result(double res, int sign) {
this.res = res;
this.sign = sign;
}
public int getSign() {
return sign;
}
}

I recommend using Executors.newCachedThreadPool(). This will return an ExecutorService which you can use to queue your calculations using submit(Callable), which returns a Future on which you can block as desired. If you queue many tasks you can just keep a list of Futures as needed or a list of tasks then submit them to the ExecutorService.
Also note it's usually not recommended to extend from Thread.
Hope this helps!

The only reason I, at least, can see why you need Threads here is to do other work while waiting for the sense method to complete in the background. For example render some graphics or interact with the user.
If your main Thread is required to wait until all the sense job is complete for each request, then you don't need Threads. Just call the method sense directly in the main Thread.
On the other hand, if you need a background Thread doing the sense job while the main Thread is doing other work, then you will need two Threads: one is the main, and the other is the background-job. Then you probably need to have a producer-consumer pattern, where the producer (the main Thread) creates the requests and the consumer (the background Thread) executes the sense method. But then it seems like the roles are turned around again like you want to wait in the main Thread all the requests to complete after you submit them. If that is the case then you can start all the MyThreads and then call join on them when you are ready to wait for their results. For example:
import java.util.ArrayList;
import java.util.Collection;
import java.util.Objects;
public class Main {
public static class Result {
private final int index;
private final Object value;
public Result(final int index,
final Object value) {
this.index = index;
this.value = value;
}
public int getIndex() {
return index;
}
public Object getValue() {
return value;
}
}
public static class MyRunnable implements Runnable {
private final int index;
private final Collection<Result> sharedResults;
public MyRunnable(final int index,
final Collection<Result> sharedResults) {
this.index = index;
this.sharedResults = Objects.requireNonNull(sharedResults);
}
#Override
public void run() {
final Result res = sense(); //Calculating outside the synchronized block.
synchronized (sharedResults) { //Synchronizing, because the actual instance of this collection might not be synchronized.
sharedResults.add(res);
}
}
private Result sense() {
return new Result(index, "Value" + index);
}
}
public static void main(final String[] args) {
final Thread[] t = new Thread[10];
final Collection<Result> sharedResults = new ArrayList<>();
for (int i = 0; i < t.length; ++i) {
t[i] = new Thread(new MyRunnable(i, sharedResults));
t[i].start();
}
for (final Thread thread: t)
try { thread.join(); } catch (final InterruptedException ix) { ix.printStackTrace(); }
sharedResults.forEach(res -> System.out.println("Result " + res.getIndex() + " with value \"" + res.getValue() + "\"."));
}
}
Another way is to use an ExecutorService like suggested by #m0skit0 and utilize the returned Future objects to wait for the results.

Algorithm - execute task only for unique entries in queue, common entries should wait

We are creating a rest application. And we have an edge condition where parallel actions are not supported on same object.
For example :
Not supported in parallel
Request 1 for action XYZ for object A
Request 2 for action XYZ for object A
Request 3 for action ABC for object A
Supported in parallel
Request 1 for action XYZ for object A
Request 2 for action XYZ for object B
Request 3 for action ABC for object C
Now, the object count is not fixed. we can have n number of such objects.
I want that if a request for object A is under progress then other request for object A should wait for existing task on object A to get over.
But I am not able to figure out the algorithm for this purpose.
I could plan for below design but not able to figure out on how to use the locking since all objects can be different.
A queue which stores the entry for object A when request comes.
Entry gets deleted if response is sent
If an entry is already present, then wait for existing request to get over.
If entry not present, then execute immediately.
Now task on object A should not impact the task on object B. So they must accept unique locks.
And also, request cannot go standalone and be queued. Somehow I have to make the current thread sleep so that I can send response to user.
Can anyone guide here?

UPDATED based on comments from my original response
The ideal model for something like that would be using an actor system such as Akka.
But your comment states that this will happen in the context on a REST application where threads will be blocked already by request processing.
In this case, the idea would be using a per-object-guard such as:
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
public class ObjectGuard<K> {
private final ConcurrentMap<K, CountDownLatch> activeTasks = new ConcurrentHashMap<>();
public Guard guardFor(final K key) throws InterruptedException {
if (key == null) {
throw new NullPointerException("key cannot be null");
}
final CountDownLatch latch = new CountDownLatch(1);
while (true) {
final CountDownLatch currentOwner = activeTasks.putIfAbsent(key, latch);
if (currentOwner == null) {
break;
} else {
currentOwner.await();
}
}
return () -> {
activeTasks.remove(key);
latch.countDown();
};
}
public interface Guard extends AutoCloseable {
#Override
void close();
}
}
You would use it as follows:
class RequestProcessor {
private final ObjectGuard<String> perObjectGuard = new ObjectGuard<>();
public String process(String objectId, String op) throws InterruptedException {
// Only one thread per object id can be present at any given time
try (ObjectGuard.Guard ignore = perObjectGuard.guardFor(objectId)) {
String result = ... // compute response
}
}
}
If two concurrent calls to process are received for the same object id, only one will be processed, the others wait their turn to process a request on that object.

An object which executes requests serially is known as Actor. The most widely known java actor library is named Akka. The most simple (one page) actor implementation is my SimpleActor.java.

Signalling like juancn does in his answer is not my strong suit, so I made an even cruder solution using one Semaphore for signalling combined with a request-counter.
There is one lock involved (subjectsLock) which synchronizes everything at one point in time. The lock is required to ensure there are no memory leaks: since there can be any number of subjects (a.k.a. object identifiers in your question), cleanup is essential. And cleanup requires knowing when something can be removed and that is difficult to determine without a lock that brings everything to one known state at a certain point in time.
The test in the main-method in the code shown below is a bit hard to read, but it serves as a starting point for a demonstration of how the code works internally. The main logic is in the methods executeRequest, addSubject and removeSubject. If those three methods do not make sense, another solution should be used.
Stress-testing will have to determine if this solution is fast enough: it depends on the number of requests (per second) and the amount of time it takes to complete an action. If there are many requests and the action is short/fast, the (synchronization) overhead from the lock could be to high.
// package so;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.IntStream;
public class RequestQueue {
public static void main(String[] args) {
// Randomized test for "executeRequest" method below.
final int threadCount = 4;
ExecutorService threadPool = Executors.newFixedThreadPool(threadCount);
try {
final int requestCount = 100;
final RequestQueue rq = new RequestQueue();
final Random random = new Random();
IntStream.range(0, requestCount).forEach(i -> threadPool.execute(new Runnable() {
#Override
public void run() {
try {
String subject = "" + (char) (((int)'A') + random.nextInt(threadCount));
rq.executeRequest(subject, new SleepAction(i, subject, 50 + random.nextInt(5)));
} catch (Exception e) {
e.printStackTrace();
}
}
}));
sleep(100); // give threads a chance to start executing.
while (true) {
sleep(200);
List<String> subjects = rq.getSubjects();
System.out.println("Subjects: " + subjects);
if (subjects.isEmpty()) {
break;
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
threadPool.shutdown();
}
}
private Map<String, QueueLock> subjects = new LinkedHashMap<>();
// a fair ReentrantLock is a little bit slower but ensures everybody gets their turn in orderly fashion.
private final ReentrantLock subjectsLock = new ReentrantLock(true);
private class QueueLock {
// a fair Semaphore ensures all requests are executed in the order they arrived.
final Semaphore turn = new Semaphore(1, true);
final AtomicInteger requests = new AtomicInteger(1);
public String toString() { return "request: " + requests.get(); }
}
/**
* Allow all requests for different subjects to execute in parallel,
* execute actions for the same subject one after another.
* Calling thread runs the action (possibly after waiting a bit when an action for a subject is already in progress).
*/
public String executeRequest(String subject, Runnable action) throws InterruptedException {
QueueLock qlock = addSubject(subject);
try {
int requestsForSubject = qlock.requests.get();
if (requestsForSubject > 1) {
System.out.println(action.toString() + " waiting for turn " + requestsForSubject);
}
qlock.turn.acquire();
if (requestsForSubject > 1) {
System.out.println(action.toString() + " taking turn " + qlock.requests.get());
}
action.run();
} catch (Exception e) {
e.printStackTrace();
} finally {
removeSubject(subject);
}
return timeSinceStart() + " " + subject;
}
private QueueLock addSubject(String s) {
QueueLock qlock = null;
subjectsLock.lock();
try {
qlock = subjects.get(s);
if (qlock == null) {
qlock = new QueueLock();
subjects.put(s, qlock);
} else {
qlock.requests.incrementAndGet();
}
} finally {
subjectsLock.unlock();
}
return qlock;
}
private boolean removeSubject(String s) {
boolean removed = false;
subjectsLock.lock();
try {
QueueLock qlock = subjects.get(s);
if (qlock.requests.decrementAndGet() == 0) {
subjects.remove(s);
removed = true;
} else {
qlock.turn.release();
}
} finally {
subjectsLock.unlock();
}
return removed;
}
public List<String> getSubjects() {
List<String> subjectsBeingProcessed = new ArrayList<>();
subjectsLock.lock();
try {
// maintains insertion order, see https://stackoverflow.com/a/18929873/3080094
subjectsBeingProcessed.addAll(subjects.keySet());
} finally {
subjectsLock.unlock();
}
return subjectsBeingProcessed;
}
public static class SleepAction implements Runnable {
final int requestNumber;
final long sleepTime;
final String subject;
public SleepAction(int requestNumber, String subject, long sleepTime) {
this.requestNumber = requestNumber;
this.sleepTime = sleepTime;
this.subject = subject;
}
#Override
public void run() {
System.out.println(toString() + " sleeping for " + sleepTime);
sleep(sleepTime);
System.out.println(toString() + " done");
}
public String toString() {return timeSinceStart() + " " + subject + " [" + Thread.currentThread().getName() + "] " + String.format("%03d",requestNumber); }
}
public static final long START_TIME = System.currentTimeMillis();
public static String timeSinceStart() {
return String.format("%05d", (System.currentTimeMillis() - START_TIME));
}
public static void sleep(long milliseconds) {
try {
Thread.sleep(milliseconds);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}

Both sequential and parallel processing

I have one producer and many consumers.
the producer is fast and generating a lot of results
tokens with the same value need to be processed sequentially
tokens with different values must be processed in parallel
creating new Runnables would be very expensive and also the production code could work with 100k of Tokens(in order to create a Runnable I have to pass to the constructor some complex to build objects)
Can I achieve the same results with a simpler algorithm? Nesting a syncronization block with a reentrant lock seems a bit unnatural.
Are there any race conditions you might notice?
Update: a second solution I found was working with 3 collections. One to cache the producer results, second a blocking queue and 3rd using a list to track in the tasks in progress. Again a bit to complicated.
My version of code
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.ReentrantLock;
public class Main1 {
static class Token {
private int order;
private String value;
Token() {
}
Token(int o, String v) {
order = o;
value = v;
}
int getOrder() {
return order;
}
String getValue() {
return value;
}
}
private final static BlockingQueue<Token> queue = new ArrayBlockingQueue<Token>(10);
private final static ConcurrentMap<String, Object> locks = new ConcurrentHashMap<String, Object>();
private final static ReentrantLock reentrantLock = new ReentrantLock();
private final static Token STOP_TOKEN = new Token();
private final static List<String> lockList = Collections.synchronizedList(new ArrayList<String>());
public static void main(String[] args) {
ExecutorService producerExecutor = Executors.newSingleThreadExecutor();
producerExecutor.submit(new Runnable() {
public void run() {
Random random = new Random();
try {
for (int i = 1; i <= 100; i++) {
Token token = new Token(i, String.valueOf(random.nextInt(1)));
queue.put(token);
}
queue.put(STOP_TOKEN);
}catch(InterruptedException e){
e.printStackTrace();
}
}
});
ExecutorService consumerExecutor = Executors.newFixedThreadPool(10);
for(int i=1; i<=10;i++) {
// creating to many runnable would be inefficient because of this complex not thread safe object
final Object dependecy = new Object(); //new ComplexDependecy()
consumerExecutor.submit(new Runnable() {
public void run() {
while(true) {
try {
//not in order
Token token = queue.take();
if (token == STOP_TOKEN) {
queue.add(STOP_TOKEN);
return;
}
System.out.println("Task start" + Thread.currentThread().getId() + " order " + token.getOrder());
Random random = new Random();
Thread.sleep(random.nextInt(200)); //doLongRunningTask(dependecy)
lockList.remove(token.getValue());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}});
}
}}

You can pre-create set of Runnables which will pick incoming tasks (tokens) and place them in queues according to their order value.
As pointed out in comments, it's not guaranteed that tokens with different values will always execute in parallel (all in all, you are bounded, at least, by nr of physical cores in your box). However, it is guaranteed that tokens with same order will be executed in the order of arrival.
Sample code:
/**
* Executor which ensures incoming tasks are executed in queues according to provided key (see {#link Task#getOrder()}).
*/
public class TasksOrderingExecutor {
public interface Task extends Runnable {
/**
* #return ordering value which will be used to sequence tasks with the same value.<br>
* Tasks with different ordering values <i>may</i> be executed in parallel, but not guaranteed to.
*/
String getOrder();
}
private static class Worker implements Runnable {
private final LinkedBlockingQueue<Task> tasks = new LinkedBlockingQueue<>();
private volatile boolean stopped;
void schedule(Task task) {
tasks.add(task);
}
void stop() {
stopped = true;
}
#Override
public void run() {
while (!stopped) {
try {
Task task = tasks.take();
task.run();
} catch (InterruptedException ie) {
// perhaps, handle somehow
}
}
}
}
private final Worker[] workers;
private final ExecutorService executorService;
/**
* #param queuesNr nr of concurrent task queues
*/
public TasksOrderingExecutor(int queuesNr) {
Preconditions.checkArgument(queuesNr >= 1, "queuesNr >= 1");
executorService = new ThreadPoolExecutor(queuesNr, queuesNr, 0, TimeUnit.SECONDS, new SynchronousQueue<>());
workers = new Worker[queuesNr];
for (int i = 0; i < queuesNr; i++) {
Worker worker = new Worker();
executorService.submit(worker);
workers[i] = worker;
}
}
public void submit(Task task) {
Worker worker = getWorker(task);
worker.schedule(task);
}
public void stop() {
for (Worker w : workers) w.stop();
executorService.shutdown();
}
private Worker getWorker(Task task) {
return workers[task.getOrder().hashCode() % workers.length];
}
}

By the nature of your code, the only way to guarantee that the tokens with the
same value are processed in serial manner is to wait for STOP_TOKEN to arrive.
You'll need single producer-single consumer setup, with consumer collecting and sorting
the tokens by their value (into the Multimap, let say).
Only then you know which tokens can be process serially and which may be processed in parallel.
Anyway, I advise you to look at LMAX Disruptor, which offers very effective way for sharing data between threads.
It doesn't suffer from synchronization overhead as Executors as it is lock free (which may give you nice performance benefits, depending on the way how you process the data).
The solution using two Disruptors
// single thread for processing as there will be only on consumer
Disruptor<InEvent> inboundDisruptor = new Disruptor<>(InEvent::new, 32, Executors.newSingleThreadExecutor());
// outbound disruptor that uses 3 threads for event processing
Disruptor<OutEvent> outboundDisruptor = new Disruptor<>(OutEvent::new, 32, Executors.newFixedThreadPool(3));
inboundDisruptor.handleEventsWith(new InEventHandler(outboundDisruptor));
// setup 3 event handlers, doing round robin consuming, effectively processing OutEvents in 3 threads
outboundDisruptor.handleEventsWith(new OutEventHandler(0, 3, new Object()));
outboundDisruptor.handleEventsWith(new OutEventHandler(1, 3, new Object()));
outboundDisruptor.handleEventsWith(new OutEventHandler(2, 3, new Object()));
inboundDisruptor.start();
outboundDisruptor.start();
// publisher code
for (int i = 0; i < 10; i++) {
inboundDisruptor.publishEvent(InEventTranslator.INSTANCE, new Token());
}
The event handler on the inbound disruptor just collects incoming tokens. When STOP token is received, it publishes the series of tokens to outbound disruptor for further processing:
public class InEventHandler implements EventHandler<InEvent> {
private ListMultimap<String, Token> tokensByValue = ArrayListMultimap.create();
private Disruptor<OutEvent> outboundDisruptor;
public InEventHandler(Disruptor<OutEvent> outboundDisruptor) {
this.outboundDisruptor = outboundDisruptor;
}
#Override
public void onEvent(InEvent event, long sequence, boolean endOfBatch) throws Exception {
if (event.token == STOP_TOKEN) {
// publish indexed tokens to outbound disruptor for parallel processing
tokensByValue.asMap().entrySet().stream().forEach(entry -> outboundDisruptor.publishEvent(OutEventTranslator.INSTANCE, entry.getValue()));
} else {
tokensByValue.put(event.token.value, event.token);
}
}
}
Outbound event handler processes tokens of the same value sequentially:
public class OutEventHandler implements EventHandler<OutEvent> {
private final long order;
private final long allHandlersCount;
private Object yourComplexDependency;
public OutEventHandler(long order, long allHandlersCount, Object yourComplexDependency) {
this.order = order;
this.allHandlersCount = allHandlersCount;
this.yourComplexDependency = yourComplexDependency;
}
#Override
public void onEvent(OutEvent event, long sequence, boolean endOfBatch) throws Exception {
if (sequence % allHandlersCount != order ) {
// round robin, do not consume every event to allow parallel processing
return;
}
for (Token token : event.tokensToProcessSerially) {
// do procesing of the token using your complex class
}
}
}
The rest of the required infrastructure (purpose described in the Disruptor docs):
public class InEventTranslator implements EventTranslatorOneArg<InEvent, Token> {
public static final InEventTranslator INSTANCE = new InEventTranslator();
#Override
public void translateTo(InEvent event, long sequence, Token arg0) {
event.token = arg0;
}
}
public class OutEventTranslator implements EventTranslatorOneArg<OutEvent, Collection<Token>> {
public static final OutEventTranslator INSTANCE = new OutEventTranslator();
#Override
public void translateTo(OutEvent event, long sequence, Collection<Token> tokens) {
event.tokensToProcessSerially = tokens;
}
}
public class InEvent {
// Note that no synchronization is used here,
// even though the field is used among multiple threads.
// Memory barrier used by Disruptor guarantee changes are visible.
public Token token;
}
public class OutEvent {
// ... again, no locks.
public Collection<Token> tokensToProcessSerially;
}
public class Token {
String value;
}

If you have lots of different tokens, then the simplest solution is to create some number of single-thread executors (about 2x your number of cores), and then distribute each task to an executor determined by the hash of its token.
That way all tasks with the same token will go to the same executor and execute sequentially, because each executor only has one thread.
If you have some unstated requirements about scheduling fairness, then it is easy enough to avoid any significant imbalances by having the producer thread queue up its requests (or block) before distributing them, until there are, say, less than 10 requests per executor outstanding.

The following solution will only use a single Map that is used by the producer and consumers to process orders in sequential order for each order number while processing different order numbers in parallel. Here is the code:
public class Main {
private static final int NUMBER_OF_CONSUMER_THREADS = 10;
private static volatile int sync = 0;
public static void main(String[] args) {
final ConcurrentHashMap<String,Controller> queues = new ConcurrentHashMap<String, Controller>();
final CountDownLatch latch = new CountDownLatch(NUMBER_OF_CONSUMER_THREADS);
final AtomicBoolean done = new AtomicBoolean(false);
// Create a Producer
new Thread() {
{
this.setDaemon(true);
this.setName("Producer");
this.start();
}
public void run() {
Random rand = new Random();
for(int i =0 ; i < 1000 ; i++) {
int order = rand.nextInt(20);
String key = String.valueOf(order);
String value = String.valueOf(rand.nextInt());
Controller controller = queues.get(key);
if (controller == null) {
controller = new Controller();
queues.put(key, controller);
}
controller.add(new Token(order, value));
Main.sync++;
}
done.set(true);
}
};
while (queues.size() < 10) {
try {
// Allow the producer to generate several entries that need to
// be processed.
Thread.sleep(5000);
} catch (InterruptedException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
// System.out.println(queues);
// Create the Consumers
ExecutorService consumers = Executors.newFixedThreadPool(NUMBER_OF_CONSUMER_THREADS);
for(int i = 0 ; i < NUMBER_OF_CONSUMER_THREADS ; i++) {
consumers.submit(new Runnable() {
private Random rand = new Random();
public void run() {
String name = Thread.currentThread().getName();
try {
boolean one_last_time = false;
while (true) {
for (Map.Entry<String, Controller> entry : queues.entrySet()) {
Controller controller = entry.getValue();
if (controller.lock(this)) {
ConcurrentLinkedQueue<Token> list = controller.getList();
Token token;
while ((token = list.poll()) != null) {
try {
System.out.println(name + " processing order: " + token.getOrder()
+ " value: " + token.getValue());
Thread.sleep(rand.nextInt(200));
} catch (InterruptedException e) {
}
}
int last = Main.sync;
queues.remove(entry.getKey());
while(done.get() == false && last == Main.sync) {
// yield until the producer has added at least another entry
Thread.yield();
}
// Purge any new entries added
while ((token = list.poll()) != null) {
try {
System.out.println(name + " processing order: " + token.getOrder()
+ " value: " + token.getValue());
Thread.sleep(200);
} catch (InterruptedException e) {
}
}
controller.unlock(this);
}
}
if (one_last_time) {
return;
}
if (done.get()) {
one_last_time = true;
}
}
} finally {
latch.countDown();
}
}
});
}
try {
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
consumers.shutdown();
System.out.println("Exiting.. remaining number of entries: " + queues.size());
}
}
Note that the Main class contains a queues instance that is a Map. The map key is the order id that you want to process sequentially by the consumers. The value is a Controller class that will contain all of the orders associated with that order id.
The producer will generate the orders and add the order, (Token), to its associated Controller. The consumers will iterator over the queues map values and call the Controller lock method to determine if it can process orders for that particular order id. If the lock returns false it will check the next Controller instance. If the lock returns true, it will process all orders and then check the next Controller.
updated Added the sync integer that is used to guarantee that when an instance of the Controller is removed from the queues map. All of its entries will be consumed. There was an logic error in the consumer code where the unlock method was called to soon.
The Token class is similar to the one that you've posted here.
class Token {
private int order;
private String value;
Token(int order, String value) {
this.order = order;
this.value = value;
}
int getOrder() {
return order;
}
String getValue() {
return value;
}
#Override
public String toString() {
return "Token [order=" + order + ", value=" + value + "]\n";
}
}
The Controller class that follows is used to insure that only a single thread within the thread pool will be processing the orders. The lock/unlock methods are used to determine which of the threads will be allowed to process the orders.
class Controller {
private ConcurrentLinkedQueue<Token> tokens = new ConcurrentLinkedQueue<Token>();
private ReentrantLock lock = new ReentrantLock();
private Runnable current = null;
void add(Token token) {
tokens.add(token);
}
public ConcurrentLinkedQueue<Token> getList() {
return tokens;
}
public void unlock(Runnable runnable) {
lock.lock();
try {
if (current == runnable) {
current = null;
}
} finally {
lock.unlock();
}
}
public boolean lock(Runnable runnable) {
lock.lock();
try {
if (current == null) {
current = runnable;
}
} finally {
lock.unlock();
}
return current == runnable;
}
#Override
public String toString() {
return "Controller [tokens=" + tokens + "]";
}
}
Additional information about the implementation. It uses a CountDownLatch to insure that all produced orders will be processed prior to the process exiting. The done variable is just like your STOP_TOKEN variable.
The implementation does contain an issue that you would need to resolve. There is the issue that it does not purge the controller for an order id when all of the orders have been processed. This will cause instances where a thread in the thread pool gets assigned to a controller that contains no orders. Which will waste cpu cycles that could be used to perform other tasks.

Is all you need is to ensure that tokens with the same value are not being processed concurrently? Your code is too messy to understand what you mean (it does not compile, and has lots of unused variables, locks and maps, that are created but never used). It looks like you are greatly overthinking this. All you need is one queue, and one map.
Something like this I imagine:
class Consumer implements Runnable {
ConcurrentHashMap<String, Token> inProcess;
BlockingQueue<Token> queue;
public void run() {
Token token = null;
while ((token = queue.take()) != null) {
if(inProcess.putIfAbsent(token.getValue(), token) != null) {
queue.put(token);
continue;
}
processToken(token);
inProcess.remove(token.getValue());
}
}
}

tokens with the same value need to be processed sequentially
The way to insure that any two things happen in sequence is to do them in the same thread.
I'd have a collection of however many worker threads, and I'd have a Map. Any time I get a token that I've not seen before, I'll pick a thread at random, and enter the token and the thread into the map. From then on, I'll use that same thread to execute tasks associated with that token.
creating new Runnables would be very expensive
Runnable is an interface. Creating new objects that implement Runnable is not going to be significantly more expensive than creating any other kind of object.

Maybe I'm misunderstanding something. But it seems that it would be easier to filter the Tokens with same value from the ones with different values into two different queues initially.
And then use Stream with either map or foreach for the sequential. And simply use the parallel stream version for the rest.
If your Tokens in production environment are lazily generated and you only get one at a time you simply make some sort of filter which distributes them to the two different queues.
If you can implement it with Streams I suqqest doing that as they are simple, easy to use and FAST!
https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html
I made a brief example of what I mean. In this case the numbers Tokens are sort of artificially constructed but thats beside the point. Also the streams are both initiated on the main thread which would probably also not be ideal.
public static void main(String args[]) {
ArrayList<Token> sameValues = new ArrayList<Token>();
ArrayList<Token> distinctValues = new ArrayList<Token>();
Random random = new Random();
for (int i = 0; i < 100; i++) {
int next = random.nextInt(100);
Token n = new Token(i, String.valueOf(next));
if (next == i) {
sameValues.add(n);
} else {
distinctValues.add(n);
}
}
distinctValues.stream().parallel().forEach(token -> System.out.println("Distinct: " + token.value));
sameValues.stream().forEach(token -> System.out.println("Same: " + token.value));
}

I am not entirely sure I have understood the question but I'll take a stab at an algorithm.
The actors are:
A queue of tasks
A pool of free executors
A set of in-process tokens currently being processed
A controller
Then,
Initially all executors are available and the set is empty
controller picks an available executor and goes through the queue looking for a task with a token that is not in the in-process set and when it finds it
adds the token to the in-process set
assigns the executor to process the task and
goes back to the beginning of the queue
the executor removes the token from the set when it is done processing and adds itself back to the pool

One way of doing this is having one executor for sequence processing and one for parallel processing. We also need a single threaded manager service that will decide to which service token needs to be submitted for processing.
// Queue to be shared by both the threads. Contains the tokens produced by producer.
BlockingQueue tokenList = new ArrayBlockingQueue(10);
private void startProcess() {
ExecutorService producer = Executors.newSingleThreadExecutor();
final ExecutorService consumerForSequence = Executors
.newSingleThreadExecutor();
final ExecutorService consumerForParallel = Executors.newFixedThreadPool(10);
ExecutorService manager = Executors.newSingleThreadExecutor();
producer.submit(new Producer(tokenList));
manager.submit(new Runnable() {
public void run() {
try {
while (true) {
Token t = tokenList.take();
System.out.println("consumed- " + t.orderid
+ " element");
if (t.orderid % 7 == 0) { // any condition to check for sequence processing
consumerForSequence.submit(new ConsumerForSequenceProcess(t));
} else {
ConsumerForParallel.submit(new ConsumerForParallelProcess(t));
}
}
}
catch (InterruptedException e) { // TODO Auto-generated catch
// block
e.printStackTrace();
}
}
});
}

I think there is a more fundamental design issue hidden behind this task, but ok. I cant figure out from you problem description if you want in-order execution or if you just want operations on tasks described by single tokens to be atomic/transactional. What i propose below feels more like a "quick fix" to this issue than a real solution.
For the real "ordered execution" case I propose a solution which is based on queue proxies which order the output:
Define a implementation of Queue which provides a factory method generating proxy queues which are represented to the producer side by a this single queue object; the factory method should also register these proxy queue objects. adding an element to the input queue should add it directly to one of the output queues if it matches one of the elements in one of the output queues. Otherwise add it to any (the shortest) output queue. (implement the check for this efficiently). Alternatively (slightly better): don't do this when the element is added, but when any of the output queues runs empty.
Give each of your runnable consumers an field storing an individual Queue interface (instead of accessing a single object). Initialize this field by a the factory method defined above.
For the transaction case i think it's easier to span more threads than you have cores (use statistics to calculate this), and implement the blocking mechanism on an lower (object) level.

java thread safety between multiple variables

I have a Metrics class that's supposed to keep track of how many transactions we process each second and how long they take. The relevant part of its structure looks like this:
public class Metrics {
AtomicLong sent = new AtomicLong();
AtomicLong totalElapsedMsgTime = new AtomicLong();
AtomicLong sentLastSecond = new AtomicLong();
AtomicLong avgTimeLastSecond = new AtomicLong();
public void outTick(long elapsedMsgTime){
sent.getAndIncrement();
totalElapsedMsgTime.getAndAdd(elapsedMsgTime);
}
class CalcMetrics extends TimerTask {
#Override
public void run() {
sentLastSecond.set(sent.getAndSet(0));
long tmpElapsed = totalElapsedMsgTime.getAndSet(0);
long tmpSent = sentLastSecond.longValue();
if(tmpSent != 0) {
avgTimeLastSecond.set(tmpElapsed / tmpSent);
} else {
avgTimeLastSecond.set(0);
}
}
}
}
My issue is that the outTick function will get called hundreds of times a second from lots of different threads. Being AtomicLong already ensures that each variable is individually thread safe, and they don't interact with each other in that function, so I don't want a lock that will make one call to outTick block another thread's call to outTick. It's perfectly fine if a couple of different threads increment the sent variable and then they both add to the totalElapsedMsgTime variable.
However, once it gets into CalcMetrics run method (which only happens once each second), they do interact. I want to ensure that I can pick up and reset both of those variables without being in the middle of an outTick call or having another outTick call occur between picking up one variable and the next.
Is there any way of doing this? (Does my explanation even make sense?) Is there a way of saying that A cannot interleave with B but multiple B's can interleave with each other?
EDIT:
I went with the ReadWriteLock that James suggested. Here's what my result looks like for anyone interested:
public class Metrics {
AtomicLong numSent = new AtomicLong();
AtomicLong totalElapsedMsgTime = new AtomicLong();
long sentLastSecond = 0;
long avgTimeLastSecond = 0;
private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
private final Lock readLock = readWriteLock.readLock();
private final Lock writeLock = readWriteLock.writeLock();
public void outTick(long elapsedMsgTime) {
readLock.lock();
try {
numSent.getAndIncrement();
totalElapsedMsgTime.getAndAdd(elapsedMsgTime);
}
finally
{
readLock.unlock();
}
}
class CalcMetrics extends TimerTask {
#Override
public void run() {
long elapsed;
writeLock.lock();
try {
sentLastSecond = numSent.getAndSet(0);
elapsed = totalElapsedMsgTime.getAndSet(0);
}
finally {
writeLock.unlock();
}
if(sentLastSecond != 0) {
avgTimeLastSecond = (elapsed / sentLastSecond);
} else {
avgTimeLastSecond = 0;
}
}
}
}

The usual solution is to wrap all variables as one atomic data type.
class Data
{
long v1, v2;
Data add(Data another){ ... }
}
AtomicReference<Data> aData = ...;
public void outTick(long elapsedMsgTime)
{
Data delta = new Data(1, elapsedMsgTime);
aData.accumulateAndGet( delta, Data:add );
}
In your case, it may not be much faster than just locking.
There is another interesting lock in java8 - StampedLock . The javadoc example pretty much matches your use case. Basically, you can do optimistic reads on multiple variables; afterwards, check to make sure that no writes were done during the reads. In your case, "hundreds" of writes per second, the optimistic reads mostly would succeed.

Sounds like you need a reader/writer lock. (java.util.concurrent.locks.ReentrantReadWriteLock).
Your outTick() function would lock the ReaderLock. Any number of threads are allowed to lock the ReaderLock at the same time.
Your calcMetrics() would lock the WriterLock. No new readers are allowed in once a thread is waiting for the writer lock, and the writer is not allowed in until all the readers are out.
You would still need the atomics to protect the individual counters that are incremented by outTick().

Use locks ( https://docs.oracle.com/javase/tutorial/essential/concurrency/locksync.html ). Once you implement locks you'll have finer control. An additional side effect will be that you won't need to use AtomicLong anymore (although you still can); you can use volatile long instead, which would be more efficient. I did not make that change in the example.
Basically just create a new Object:
private Object lock = new Object();
Then, use the synchronized keyword with that object around all the code that should never happen at the same time as another synchronized block with the same lock. Example:
synchronized(lock)
{
sent.getAndIncrement();
totalElapsedMsgTime.getAndAdd(elapsedMsgTime);
}
So your whole program will look like this (note: untested code)
public class Metrics {
private Object lock = new Object();
AtomicLong sent = new AtomicLong();
AtomicLong totalElapsedMsgTime = new AtomicLong();
AtomicLong sentLastSecond = new AtomicLong();
AtomicLong avgTimeLastSecond = new AtomicLong();
public void outTick(long elapsedMsgTime){
synchronized (lock)
{
sent.getAndIncrement();
totalElapsedMsgTime.getAndAdd(elapsedMsgTime);
}
}
class CalcMetrics extends TimerTask {
#Override
public void run() {
synchronized (lock)
{
sentLastSecond.set(sent.getAndSet(0));
long tmpElapsed = totalElapsedMsgTime.getAndSet(0);
long tmpSent = sentLastSecond.longValue();
if(tmpSent != 0) {
avgTimeLastSecond.set(tmpElapsed / tmpSent);
} else {
avgTimeLastSecond.set(0);
}
}
}
}
}
Edit: I threw together a quick (and ugly) efficiency test program and found that when I synchronize with locks, I get overall better performance. Note that the results of the first 2 runs are discarded because the timing results when the Java JIT still hasn't compiled all code paths to machine code are not representative of the long term runtime.
Results:
With Locks: 8365ms
AtomicLong: 21254ms
Code:
import java.util.concurrent.atomic.AtomicLong;
public class Main
{
private AtomicLong testA_1 = new AtomicLong();
private AtomicLong testB_1 = new AtomicLong();
private volatile long testA_2 = 0;
private volatile long testB_2 = 0;
private Object lock = new Object();
private volatile boolean a = false;
private volatile boolean b = false;
private volatile boolean c = false;
private static boolean useLocks = false;
public static void main(String args[])
{
System.out.println("Locks:");
useLocks = true;
test();
System.out.println("No Locks:");
useLocks = false;
test();
System.out.println("Locks:");
useLocks = true;
test();
System.out.println("No Locks:");
useLocks = false;
test();
}
private static void test()
{
final Main main = new Main();
new Thread()
{
public void run()
{
for (int i = 0; i < 80000000; ++i)
main.outTick(10);
main.a = true;
}
}.start();
new Thread()
{
public void run()
{
for (int i = 0; i < 80000000; ++i)
main.outTick(10);
main.b = true;
}
}.start();
new Thread()
{
public void run()
{
for (int i = 0; i < 80000000; ++i)
main.outTick(10);
main.c = true;
}
}.start();
long startTime = System.currentTimeMillis();
// Okay this isn't the best way to do this, but it's good enough
while (!main.a || !main.b || !main.c)
{
try
{
Thread.sleep(1);
} catch (InterruptedException e)
{
}
}
System.out.println("Elapsed time: " + (System.currentTimeMillis() - startTime) + "ms");
System.out.println("Test A: " + main.testA_1 + " " + main.testA_2);
System.out.println("Test B: " + main.testB_1 + " " + main.testB_2);
System.out.println();
}
public void outTick(long elapsedMsgTime)
{
if (!useLocks)
{
testA_1.getAndIncrement();
testB_1.getAndAdd(elapsedMsgTime);
}
else
{
synchronized (lock)
{
++testA_2;
testB_2 += elapsedMsgTime;
}
}
}
}

semantic of local final variable in the Java Memory Model?

The following code sometimes prints "valueWrapper.isZero()" on my Windows-PC and a Mac,
both running their JVM in server mode.
Ok this happens because the value field isn't final in the ValueWrapper class,
so its possible that some thread sees the stale value 0.
public class ConcurrencyApp {
private final Random rand = new Random(System.currentTimeMillis());
private ValueWrapper valueWrapper;
private static class ValueWrapper {
private int value;
public ValueWrapper(int value) {
this.value = value;
}
public boolean isZero() {
return value == 0;
}
}
private void go() {
while (true) {
valueWrapper = new ValueWrapper(randomInt(10, 1024));
Thread thread = new Thread(new Runnable() {
#Override
public void run() {
if (valueWrapper.isZero()) {
System.out.println("valueWrapper.isZero()");
}
}
});
thread.start();
}
}
private int randomInt(int min, int max) {
int randomNum = rand.nextInt((max - min) + 1) + min;
return randomNum;
}
public static void printVMInfos() {
String vmName = System.getProperty("java.vm.name");
System.out.println("vm name: " + vmName);
int cores = Runtime.getRuntime().availableProcessors();
System.out.println("available cores: " + cores);
}
public static void main(String[] args) {
ConcurrencyApp app = new ConcurrencyApp();
printVMInfos();
app.go();
}
}
But what about the following modification, here i used a local final variable:
private void go() {
while (true) {
final ValueWrapper valueWrapper = new ValueWrapper(randomInt(10, 1024));
Thread thread = new Thread(new Runnable() {
#Override
public void run() {
if (valueWrapper.isZero()) {
System.out.println("valueWrapper.isZero()");
}
}
});
thread.start();
}
}
It looks like that now no thread sees a stale value of 0.
But is this guaranteed by the JMM?
A brief look in the spec doesn't convinced me.

It looks like that now no thread sees a stale value of 0. But is this guaranteed by the JMM? A brief look in the spec doesn't convinced me.
It is guaranteed but not because of the final. There is a happens-before guarantee when you fork a thread. Any memory operations done in the forking thread before you start a new thread are guaranteed to be seen by the new thread as fully constructed and published. To quote from JLS 17.4.4 - Synchronization Order:
An action that starts a thread synchronizes-with the first action in the thread it starts.
This is different from a final field when we are talking about object construction and publishing. If a field is final then it is guaranteed to be properly initialized when the constructor finishes and the object is published to multiple threads. In your case, the final is necessary because of the anonymous class. If you weren't using an anonymous class and then you could remove the final on your ValueWrapper, your object would still be guaranteed to be fully constructed because of the above.
FYI, see here for final field info: Java concurrency: is final field (initialized in constructor) thread-safe?

I am addressing one point Gray didn't, but I would accept his, as his answer is spot on
The following code sometimes prints "valueWrapper.isZero()" on my Windows-PC and a Mac, both running their JVM in server mode.... It looks like that now no thread sees a stale value of 0. But is this
guaranteed by the JMM? A brief look in the spec doesn't convinced me.
The reason you are seeing valueWrapper.isZero() returning true sometimes because valueWrapper is changing after the start is invoked and before run gets to the boolean test. If you only have one instance created it will always not be zero as Gray mentioned.
The reason final ValueWrapper valueWrapper = new ValueWrapper(randomInt(10, 1024)); works all the time is because the field is thread (and method) local and the semantics for a local object and anonymous inner classes is to copy the original reference into the class instance.