We Keep Coding

How to use multithreading in a loop in Java

Here's what I'm trying to do. I am recording data from different sensors in a while loop until the user stops the recording. I want to record as much data as possible per second. The sensors require different time to return a value, between 200ms and 3 seconds. Therefore, sequentially calling the sensors successively is not an option.
Sequentially calling the sensors looks like this:
List<DataRow> dataRows= new ArrayList<DataRow>();
while (recording) {
DataRow dataRow = new DataRow();
dataRow.setDataA(sensorA.readData());
dataRow.setDataB(sensorB.readData());
dataRow.setDataC(sensorC.readData());
dataRows.add(dataRow);
}
Depending on the sensor, reading the data looks (much simplified) like that
public class SensorA {
public SensorAData readData(){
sensorA.startSensing();
try {
TimeUnit.MILLISECONDS.sleep(750);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return sensorA.readAndConvertByteStream();
}
}
To utilize Multithreading can SensorA implement Callable and receive Future objects in the loop? Or should the while loop be placed within a run() method implementing the interface Runnable?
Basically, can Java (or a thread) write to the correct dataRow object even if the loop is already at least one iteration further? If not, how can one solve this problem?

If I understand your needs correctly, this may be solution you want:
In each iteration n sensors are read by n concurrent threads,
If all threads has sensors data collected, new result row is added to list
Working code:
public class TestX {
private final ExecutorService pool = Executors.newFixedThreadPool(3);
private final int N = 10;
// all sensors are read sequentially and put in one row
public void testSequential() {
int total = 0;
long t = System.currentTimeMillis();
for (int i = 0; i < N; i++) {
System.out.println("starting iteration " + i);
int v1 = getSensorA(); // run in main thread
int v2 = getSensorB(); // run in main thread
int v3 = getSensorC(); // run in main thread
// collection.add( record(v1, v2, v3)
total += v1 + v2 + v3;
}
System.out.println("total = " + total + " time = " + (System.currentTimeMillis() - t) + " ms");
}
// all sensors are read concurrently and then put in one row
public void testParallel() throws ExecutionException, InterruptedException {
int total = 0;
long t = System.currentTimeMillis();
final SensorCallable s1 = new SensorCallable(1);
final SensorCallable s2 = new SensorCallable(3);
final SensorCallable s3 = new SensorCallable(3);
for (int i = 0; i < N; i++) {
System.out.println("starting iteration " + i);
Future<Integer> future1 = pool.submit(s1); // run in thread 1
Future<Integer> future2 = pool.submit(s2); // run in thread 2
Future<Integer> future3 = pool.submit(s3); // run in thread 3
int v1 = future1.get();
int v2 = future2.get();
int v3 = future3.get();
// collection.add( record(v1, v2, v3)
total += v1 + v2 + v3;
}
System.out.println("total = " + total + " time = " + (System.currentTimeMillis() - t) + " ms");
}
private class SensorCallable implements Callable<Integer> {
private final int sensorId;
private SensorCallable(int sensorId) {
this.sensorId = sensorId;
}
#Override
public Integer call() throws Exception {
switch (sensorId) {
case 1: return getSensorA();
case 2: return getSensorB();
case 3: return getSensorC();
default:
throw new IllegalArgumentException("Unknown sensor id: " + sensorId);
}
}
}
private int getSensorA() {
sleep(700);
return 1;
}
private int getSensorB() {
sleep(500);
return 2;
}
private int getSensorC() {
sleep(900);
return 2;
}
private void sleep(long ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException e) {
// ignore
}
}
public static void main(String[] args) throws ExecutionException, InterruptedException {
new TestX().testSequential();
new TestX().testParallel();
}
}
and output:
starting iteration 0
starting iteration 1
starting iteration 2
starting iteration 3
starting iteration 4
starting iteration 5
starting iteration 6
starting iteration 7
starting iteration 8
starting iteration 9
total = 50 time = 21014 ms
starting iteration 0
starting iteration 1
starting iteration 2
starting iteration 3
starting iteration 4
starting iteration 5
starting iteration 6
starting iteration 7
starting iteration 8
starting iteration 9
total = 50 time = 9009 ms
-- EDIT --
in java 8 you can use method reference to get rid of Callable classes and just write:
Future<Integer> future1 = pool.submit( this::getSensorA() );
Future<Integer> future2 = pool.submit( this::getSensorB() );
Future<Integer> future3 = pool.submit( this::getSensorC() );

Keeping a counter with ExecutorService?

I'd like to keep a counter of executed threads, to use in the same threads that I am executing.
The problem here is that although the counter increases, it increases unevenly and from the console output I got this (I have a for loop that executes 5 threads with ExecutorService):
This is a test. N:3
This is a test. N:4
This is a test. N:4
This is a test. N:4
This is a test. N:4
As you can see instead of getting 1,2,3,4,5 I got 3,4,4,4,4.
I assume this is because the for loop is running fast enough to execute the threads, and the threads are fast enough to execute the code requesting for the counter faster than the counter can update itself (does that even make sense?).
Here is the code (it is smaller and there is no meaningful use for the counter):
for (int i = 0; i < 5; i++)
{
Thread thread;
thread = new Thread()
{
public void run()
{
System.out.println("This is test. N: "+aldo );
//In here there is much more stuff, saying it because it might slow down the execution (if that is the culprit?)
return;
}
};
threadList.add(thread);
}
//later
for (int i = 0; i < threadList.size(); i++)
{
executor.execute(threadList.get(i));
aldo = aldo + 1;
}
executor.shutdown();
try
{
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException e)
{
}
Yes, aldo the counter ( with a few other lists, I think) are missing from the code (they are very simple).

The best way I know of doing this is by creating a custom thread class with a constructor that passes in a number. The variable holding the number can then be used later for any needed logging. Here is the code I came up with.
public static void main(String[] args) {
class NumberedThread implements Runnable {
private final int number;
public NumberedThread(int number) {
this.number = number;
}
#Override
public void run() {
System.out.println("This is test. N: " + number);
}
}
List<Thread> threadList = new ArrayList<>();
for (int i = 1; i < 6; i++) threadList.add(new Thread(new NumberedThread(i)));
ExecutorService executor = Executors.newFixedThreadPool(10);;
for (Thread thread : threadList) executor.execute(thread);
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException ignored) { }
}
You could also use a string object instead if you wanted to name the threads.

aldo is not modified by the tasks in the thread, but instead is modified in the main thread, here:
for (int i = 0; i < threadList.size(); i++) {
executor.execute(threadList.get(i));
//here...
aldo = aldo + 1;
}
Also, since you want a counter that can increase its value in several threads, then you may use an AtomicInteger rather than int.
Your code should look like this:
AtomicInteger aldo = new AtomicInteger(1);
for (int i = 0; i < 5; i++) {
executor.execute( () -> {
System.out.println("This is test. N: " + aldo.getAndIncrement());
});
}

Why the concurrent threads limit doesn't works as expected?

Though there are similar issues, I couldn't found any similar examples like the one I got. I really appreciate any help understanding where I got wrong with my implementation.
What I'm trying to do:
I have a Main class Driver, which can instantiates unknown number of threads. Each thread call a singleton class which should simulate a 'fake' file transfer action.
The issue I have is that I need to limit the concurrent transfers to 2 transfers, regardless the number of concurrent requests.
The way I tried to solve my problem is by adding each new Thread in a ConcurrentLinkedQueue and managing it by using Executors.newFixedThreadPool(POOL_SIZE) to limit the concurrent threads to be 2. for every interation - I poll new thread from the pool using pool.submit.
The Problem I have is my output is like this:
[Thread1], [Thread1, Thread2], [Thread1, Thread2, Thread3]...
While it should be:
[Thread1, Thread2], [Thread3, Thread4]
Why the limitation doesn't work here?
My implementation:
Copier - this is my singleton class.
public class Copier {
private final int POOL_SIZE = 2;
private static volatile Copier instance = null;
private Queue<Reportable> threadQuere = new ConcurrentLinkedQueue();
private static FileCopier fileCopier = new FileCopier();
private Copier() {
}
public static Copier getInstance() {
if (instance == null) {
synchronized (Copier.class) {
if (instance == null) {
instance = new Copier();
}
}
}
return instance;
}
public void fileTransfer(Reportable reportable) {
threadQuere.add(reportable);
ExecutorService pool = Executors.newFixedThreadPool(POOL_SIZE);
for (int i=0; i < threadQuere.size(); i++) {
System.out.println("This is the " + (i+1) + " thread");
pool.submit(new CopyThread());
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
CopyThread - represend a thread class
public class CopyThread implements Reportable, Runnable {
private static FileCopier fileCopier = new FileCopier();
#Override
public void report(String bitrate) {
System.out.println(bitrate);
}
#Override
public void run() {
synchronized(fileCopier) {
long startTime = System.nanoTime();
long bytes = fileCopier.copyFile();
long endTime = System.nanoTime();
double duration = (double)(endTime - startTime) / 1000000000; // get in seconds
double bytesInMegas = (double) bytes / 1000000;
report(bytesInMegas + "MB were transferred in " + duration + " seconds");
}
}
}
Driver - my main class where do I create all the threads
public class Driver {
public static void main(String[] args) {
Copier copier = Copier.getInstance();
CopyThread copyThread1 = new CopyThread();
CopyThread copyThread2 = new CopyThread();
CopyThread copyThread3 = new CopyThread();
CopyThread copyThread4 = new CopyThread();
copier.fileTransfer(copyThread1);
copier.fileTransfer(copyThread2);
copier.fileTransfer(copyThread3);
copier.fileTransfer(copyThread4);
int q = 0;
}
}

A simpler solution would be a Semaphore with 2 permits.
This makes sure that "outside" threads can't bypass the limit either, since your solution expects that the simultaneous tasks are limited by the size of the threadpool.
Your solution uses several concurrency tools when a single one would suffice. Your DCL singleton is a bit outdated too.

Everything is probably fine here (although a bit weird). You are printing the thread numbers before submiting, what you need to do is put print in a run method, and you will see that everything works fine. The print are all gonna go off normally, because the area where you are using print has nothing to do with Executors. There is more problems with your code, but I think you did all that just for testing/learning so that's why it's like that.
In that case, like I said, put prints in the run method (you can use some static variable in CopyThread class for counting threads). Your output will be something like 2 prints about thread numbers (1 and 2), 2 prints about how long transfer took and then prints about thread 3 and 4 (I say probably, because we are working with threads, can't be sure of anything) - all this at the step 4 ofcourse, when your fileTransfer submits 4 runnables. Your singleton is outdated, because it uses double checked locking, which is wrong on multithreaded machine, check this: here. That's not ruining your program so worry about it later. About everything else (weird queue usage, fileTransfer method making new threads pools etc.) like I said, it's probably for learning, but if it's not - your queue may as well be deleted, you are using it only for counting and counting like this could be done with some counter variable, and your fileTransfer method should just submit new runnable to pool (which would be instance variable) to transfer a file, not create pool and submit few runnables, it's kinda anty-intuitive.
Edit: check this, I put all in Cat.java for simplicity, changed some things that I had to change (I don't have FileCopier class etc., but answer to your problem is here):
import java.util.*;
import java.util.concurrent.*;
class Copier {
private final int POOL_SIZE = 2;
private static volatile Copier instance = null;
private Copier() {
}
public static Copier getInstance() {
if (instance == null) {
synchronized (Copier.class) {
if (instance == null) {
instance = new Copier();
}
}
}
return instance;
}
public void fileTransfer() {
ExecutorService pool = Executors.newFixedThreadPool(POOL_SIZE);
for (int i=0; i < 4; i++) {
pool.submit(new CopyThread());
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class CopyThread implements Runnable {
private static int counter = 0;
public void report(String bitrate) {
System.out.println(bitrate);
}
Object obj = new Object();
#Override
public void run() {
synchronized(obj) {
System.out.println("This is the " + (++counter) + " thread");
long startTime = System.nanoTime();
long bytes = 0;
for(int i=0; i<100000; i++)
bytes+=1;
long endTime = System.nanoTime();
double duration = (double)(endTime - startTime) / 1000000000; // get in seconds
double bytesInMegas = (double) bytes / 1000000;
report(bytesInMegas + "MB were transferred in " + duration + " seconds");
}
}
}
public class Cat {
public static void main(String[] args) {
Copier copier = Copier.getInstance();
copier.fileTransfer();
}
}

unexpected answers in multithreading in java

This is my code that sum variable 'res' by one 4*10^7 time using 4 threads:
class MathSin extends Thread {
public double a;
public MathSin(int degree) {
a = degree;
}
#Override
public void run() {
for (int i = 0; i < Math.pow(10., 7); i++)
MathThreads.res++;
}
}
class MathThreads {
public static double res = 0;
public static void main(String args[]) {
MathSin st = new MathSin(8);
MathSin ct = new MathSin(8);
MathSin tt = new MathSin(8);
MathSin qt = new MathSin(8);
st.start();
ct.start();
tt.start();
qt.start();
try { // wait for completion of all thread and then sum
st.join();
ct.join(); // wait for completion of MathCos object
tt.join();
qt.join();
System.out.println(res);
} catch (InterruptedException IntExp) {
}
}
}
and these are some of answers :
1.8499044E7
2.3446789E7
.
.
.
I expected get 3.0E7 but get another different answers.
how can fix this problem?

What is the problem?
You are observing race conditions while updating the static variable res.
MathThreads.res++
is equivalent to:
double tmp = MathThreads.res;
MathThreads.res = tmp + 1;
Now what happened if two threads reads at the same time a value for tmp, and both update res with tmp + 1? Well, one increment has simply been forgotten: res ends being tmp + 1 instead of being tmp + 1 + 1!
So with 4 threads updating res concurrently, you simply end up with an undefined behavior : it is impossible to predict the final value of res because of those race conditions. Two executions of the same code will give you different answers.
How to solve this issue?
To make your code thread-safe, you need to use a thread-safe structure for res: a structure that can be concurrently updated and accessed.
In your case, an AtomicLong seems the perfect choice:
public static AtomicLong res = new AtomicLong(0);
And in the run method:
for (int i = 0; i < Math.pow(10., 7); i++) {
MathThreads.res.incrementAndGet();
}

Why does ThreadPoolExecutor reduce threads below corePoolSize after the keepAliveTime?

I've been messing around with different strategies for thread pooling using ThreadPoolExecutor with JDK6. I have a Priority queue working but wasn't sure if I liked how the pool didn't size after keepAliveTime (what you get with an unbounded queue). So, I'm looking at a ThreadPoolExecutor using a LinkedBlockingQueue and the CallerRuns policy.
The issue I'm having with it now is that the pool ramps up, as the docs explain that it should, but after the tasks complete and the keepAliveTime comes into play getPoolSize shows the pool getting reduced to zero. The example code below should let you see the basis for my question:
public class ThreadPoolingDemo {
private final static Logger LOGGER =
Logger.getLogger(ThreadPoolingDemo.class.getName());
public static void main(String[] args) throws Exception {
LOGGER.info("MAIN THREAD:starting");
runCallerTestPlain();
}
private static void runCallerTestPlain() throws InterruptedException {
//10 core threads,
//50 max pool size,
//100 tasks in queue,
//at max pool and full queue - caller runs task
ThreadPoolExecutor tpe = new ThreadPoolExecutor(10, 50,
5L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(100),
new ThreadPoolExecutor.CallerRunsPolicy());
//dump 5000 tasks on the queue
for (int i = 0; i < 5000; i++) {
tpe.submit(new Runnable() {
#Override
public void run() {
//just to eat some time and give a little feedback
for (int j = 0; j < 20; j++) {
LOGGER.info("First-batch Task, looping:" + j + "["
+ Thread.currentThread().getId() + "]");
}
}
}, null);
}
LOGGER.info("MAIN THREAD:!!Done queueing!!");
//check tpe statistics forever
while (true) {
LOGGER.info("Active count: " + tpe.getActiveCount() + " Pool size: "
+ tpe.getPoolSize() + " Largest Pool: " + tpe.getLargestPoolSize());
Thread.sleep(1000);
}
}
}
I found an old bug that seems to be this issue but it was closed: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6458662. Could this still be present in 1.6 or am I missing something?
It looks like I Rubber Ducked this one (http://www.codinghorror.com/blog/2012/03/rubber-duck-problem-solving.html). The bug I linked above is related to this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6576792, where the issue seems to be resolved in 1.7 (I loaded up 1.7 and verified - fixed...). I guess my main problem was that a bug this fundamental remained for almost a decade. I spent too much time writing this up to not post it now, hope it helps someone.

... after the tasks complete and the keepAliveTime comes into play getPoolSize shows the pool getting reduced to zero.
So this looks to be a race condition in the ThreadPoolExecutor. I guess it is working according to design albeit not expected. In the getTask() method which the worker threads loop through to get tasks from the blocking queue, you see this code:
if (state == SHUTDOWN) // Help drain queue
r = workQueue.poll();
else if (poolSize > corePoolSize || allowCoreThreadTimeOut)
r = workQueue.poll(keepAliveTime, TimeUnit.NANOSECONDS);
else
r = workQueue.take();
if (r != null)
return r;
if (workerCanExit()) {
if (runState >= SHUTDOWN) // Wake up others
interruptIdleWorkers();
return null;
}
If the poolSize grows above the corePoolSize then if the poll times out after keepAliveTime, the code falls down to workerCanExit() since r is null. All of the threads can return true from that method since it is just testing the state of poolSize:
mainLock.lock();
boolean canExit;
try {
canExit = runState >= STOP ||
workQueue.isEmpty() ||
(allowCoreThreadTimeOut &&
poolSize > Math.max(1, corePoolSize)); << test poolSize here
} finally {
mainLock.unlock(); << race to workerDone() begins
}
Once that returns true then the worker thread exits and then the poolSize is decremented. If all of the worker threads do that test at the same time then they will all exit because of the race between the testing of poolSize and the stopping of the worker when --poolSize occurs.
What surprises me is how consistent that race condition is. If you add some randomization to the sleep() inside of the run() below then you can get some core threads to not quit but I would have thought the race condition would have been harder to hit.
You can see this behavior in the following test:
#Test
public void test() throws Exception {
int before = Thread.activeCount();
int core = 10;
int max = 50;
int queueSize = 100;
ThreadPoolExecutor tpe =
new ThreadPoolExecutor(core, max, 1L, TimeUnit.SECONDS,
new LinkedBlockingQueue<Runnable>(queueSize),
new ThreadPoolExecutor.CallerRunsPolicy());
tpe.allowCoreThreadTimeOut(false);
assertEquals(0, tpe.getActiveCount());
// if we start 1 more than can go into core or queue, poolSize goes to 0
int startN = core + queueSize + 1;
// if we only start jobs the core can take care of, then it won't go to 0
// int startN = core + queueSize;
for (int i = 0; i < startN; i++) {
tpe.submit(new Runnable() {
#Override
public void run() {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
}
while (true) {
System.out.println("active = " + tpe.getActiveCount() + ", poolSize = " + tpe.getPoolSize()
+ ", largest = " + tpe.getLargestPoolSize() + ", threads = " + (Thread.activeCount() - before));
Thread.sleep(1000);
}
}
If you change the sleep line inside of the run() method to something like this:
private final Random random = new Random();
...
Thread.sleep(100 + random.nextInt(100));
This will make the race condition harder to hit so some core threads will still be around.