We Keep Coding

How can I schedule some work in n threads separately

Lets say I have n threads concurrently taking values from a shared queue:
public class WorkerThread implements Runnable{
private BlockingQueue queue;
private ArrayList<Integer> counts = new ArrayList<>();
private int count=0;
public void run(){
while(true) {
queue.pop();
count++;
}
}
}
Then for each thread, I want to count every 5 seconds how many items it has dequeued, and then store it in its own list (counts)
I've seen here Print "hello world" every X seconds how you can run some code every x seconds:
Timer t = new Timer();
t.scheduleAtFixedRate(new TimerTask(){
#Override
public void run(){
counts.add(count);
count = 0
}
}, 0, 5000);
The problem with this is that I can't access count variable and the list of counts unless they are static. But I don't want them to be static because I don't want the different threads to share those variables.
Any ideas of how to handle this?

I don't think it's possible to use scheduled execution for you case(neither Timer nor ScheduledExecutorService), because each new scheduled invocation will create a new tasks with while loop. So number of tasks will increase constantly.
If you don't need to access this list of counts in runtime i would suggest something like this one:
static class Task implements Runnable {
private final ThreadLocal<List<Integer>> counts = ThreadLocal.withInitial(ArrayList::new);
private volatile List<Integer> result = new ArrayList<>();
private BlockingQueue<Object> queue;
public Task(BlockingQueue<Object> queue) {
this.queue = queue;
}
#Override
public void run() {
int count = 0;
long start = System.nanoTime();
try {
while (!Thread.currentThread().isInterrupted()) {
queue.take();
count++;
long end = System.nanoTime();
if ((end - start) >= TimeUnit.SECONDS.toNanos(1)) {
counts.get().add(count);
count = 0;
start = end;
}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
// the last value
counts.get().add(count);
// copy the result cause it's not possible
// to access thread local variable outside of this thread
result = counts.get();
}
public List<Integer> getCounts() {
return result;
}
}
public static void main(String[] args) throws Exception {
ExecutorService executorService = Executors.newFixedThreadPool(3);
BlockingQueue<Object> blockingQueue = new LinkedBlockingQueue<>();
Task t1 = new Task(blockingQueue);
Task t2 = new Task(blockingQueue);
Task t3 = new Task(blockingQueue);
executorService.submit(t1);
executorService.submit(t2);
executorService.submit(t3);
for (int i = 0; i < 50; i++) {
blockingQueue.add(new Object());
Thread.sleep(100);
}
// unlike shutdown() interrupts running threads
executorService.shutdownNow();
executorService.awaitTermination(1, TimeUnit.SECONDS);
System.out.println("t1 " + t1.getCounts());
System.out.println("t2 " + t2.getCounts());
System.out.println("t3 " + t3.getCounts());
int total = Stream.concat(Stream.concat(t1.getCounts().stream(), t2.getCounts().stream()), t3.getCounts().stream())
.reduce(0, (a, b) -> a + b);
// 50 as expected
System.out.println(total);
}

Why not a static AtomicLong?
Or the WorkerThread(s) can publish that they poped to the TimerTask or somewhere else? And the TimerTask reads that info?

Why the concurrent threads limit doesn't works as expected?

Though there are similar issues, I couldn't found any similar examples like the one I got. I really appreciate any help understanding where I got wrong with my implementation.
What I'm trying to do:
I have a Main class Driver, which can instantiates unknown number of threads. Each thread call a singleton class which should simulate a 'fake' file transfer action.
The issue I have is that I need to limit the concurrent transfers to 2 transfers, regardless the number of concurrent requests.
The way I tried to solve my problem is by adding each new Thread in a ConcurrentLinkedQueue and managing it by using Executors.newFixedThreadPool(POOL_SIZE) to limit the concurrent threads to be 2. for every interation - I poll new thread from the pool using pool.submit.
The Problem I have is my output is like this:
[Thread1], [Thread1, Thread2], [Thread1, Thread2, Thread3]...
While it should be:
[Thread1, Thread2], [Thread3, Thread4]
Why the limitation doesn't work here?
My implementation:
Copier - this is my singleton class.
public class Copier {
private final int POOL_SIZE = 2;
private static volatile Copier instance = null;
private Queue<Reportable> threadQuere = new ConcurrentLinkedQueue();
private static FileCopier fileCopier = new FileCopier();
private Copier() {
}
public static Copier getInstance() {
if (instance == null) {
synchronized (Copier.class) {
if (instance == null) {
instance = new Copier();
}
}
}
return instance;
}
public void fileTransfer(Reportable reportable) {
threadQuere.add(reportable);
ExecutorService pool = Executors.newFixedThreadPool(POOL_SIZE);
for (int i=0; i < threadQuere.size(); i++) {
System.out.println("This is the " + (i+1) + " thread");
pool.submit(new CopyThread());
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
CopyThread - represend a thread class
public class CopyThread implements Reportable, Runnable {
private static FileCopier fileCopier = new FileCopier();
#Override
public void report(String bitrate) {
System.out.println(bitrate);
}
#Override
public void run() {
synchronized(fileCopier) {
long startTime = System.nanoTime();
long bytes = fileCopier.copyFile();
long endTime = System.nanoTime();
double duration = (double)(endTime - startTime) / 1000000000; // get in seconds
double bytesInMegas = (double) bytes / 1000000;
report(bytesInMegas + "MB were transferred in " + duration + " seconds");
}
}
}
Driver - my main class where do I create all the threads
public class Driver {
public static void main(String[] args) {
Copier copier = Copier.getInstance();
CopyThread copyThread1 = new CopyThread();
CopyThread copyThread2 = new CopyThread();
CopyThread copyThread3 = new CopyThread();
CopyThread copyThread4 = new CopyThread();
copier.fileTransfer(copyThread1);
copier.fileTransfer(copyThread2);
copier.fileTransfer(copyThread3);
copier.fileTransfer(copyThread4);
int q = 0;
}
}

A simpler solution would be a Semaphore with 2 permits.
This makes sure that "outside" threads can't bypass the limit either, since your solution expects that the simultaneous tasks are limited by the size of the threadpool.
Your solution uses several concurrency tools when a single one would suffice. Your DCL singleton is a bit outdated too.

Everything is probably fine here (although a bit weird). You are printing the thread numbers before submiting, what you need to do is put print in a run method, and you will see that everything works fine. The print are all gonna go off normally, because the area where you are using print has nothing to do with Executors. There is more problems with your code, but I think you did all that just for testing/learning so that's why it's like that.
In that case, like I said, put prints in the run method (you can use some static variable in CopyThread class for counting threads). Your output will be something like 2 prints about thread numbers (1 and 2), 2 prints about how long transfer took and then prints about thread 3 and 4 (I say probably, because we are working with threads, can't be sure of anything) - all this at the step 4 ofcourse, when your fileTransfer submits 4 runnables. Your singleton is outdated, because it uses double checked locking, which is wrong on multithreaded machine, check this: here. That's not ruining your program so worry about it later. About everything else (weird queue usage, fileTransfer method making new threads pools etc.) like I said, it's probably for learning, but if it's not - your queue may as well be deleted, you are using it only for counting and counting like this could be done with some counter variable, and your fileTransfer method should just submit new runnable to pool (which would be instance variable) to transfer a file, not create pool and submit few runnables, it's kinda anty-intuitive.
Edit: check this, I put all in Cat.java for simplicity, changed some things that I had to change (I don't have FileCopier class etc., but answer to your problem is here):
import java.util.*;
import java.util.concurrent.*;
class Copier {
private final int POOL_SIZE = 2;
private static volatile Copier instance = null;
private Copier() {
}
public static Copier getInstance() {
if (instance == null) {
synchronized (Copier.class) {
if (instance == null) {
instance = new Copier();
}
}
}
return instance;
}
public void fileTransfer() {
ExecutorService pool = Executors.newFixedThreadPool(POOL_SIZE);
for (int i=0; i < 4; i++) {
pool.submit(new CopyThread());
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class CopyThread implements Runnable {
private static int counter = 0;
public void report(String bitrate) {
System.out.println(bitrate);
}
Object obj = new Object();
#Override
public void run() {
synchronized(obj) {
System.out.println("This is the " + (++counter) + " thread");
long startTime = System.nanoTime();
long bytes = 0;
for(int i=0; i<100000; i++)
bytes+=1;
long endTime = System.nanoTime();
double duration = (double)(endTime - startTime) / 1000000000; // get in seconds
double bytesInMegas = (double) bytes / 1000000;
report(bytesInMegas + "MB were transferred in " + duration + " seconds");
}
}
}
public class Cat {
public static void main(String[] args) {
Copier copier = Copier.getInstance();
copier.fileTransfer();
}
}

How to use multiple cores with java?

I am trying to understand how java threads works with CPU cores. I have a 4 core CPU and when I try to run below codes execution times are interesting. Why it doesn't speed up with multiple threads? Am I doing something wrong?
availableProcessors() returns 4 by the way.
Below code takes nearly 27 seconds;
Runnable runnable = new Runnable() {
#Override
public void run() {
int x = 0;
while(x < 10000000){
System.out.println(x);
x++;
}
}
};
Thread t1 = new Thread(runnable);
t1.start();
When I use multiple threads it takes 33 seconds;
Runnable runnable = new Runnable() {
#Override
public void run() {
int x = 0;
while(x < 5000000){
System.out.println(x);
x++;
}
}
};
Runnable runnable2 = new Runnable() {
#Override
public void run() {
int x = 5000000;
while(x < 10000000){
System.out.println(x);
x++;
}
}
};
Thread t1 = new Thread(runnable);
t1.start();
Thread t2 = new Thread(runnable2);
t2.start();

Your program has a sequential bottleneck which is printing to the terminal.
System.out.println has a synchronized block in it and hence writes are one at a time, and hence your code is not parallel.
Sequential parts (including coordination) of the program is what governs its performance according to Amdahl's law

Unable to understand the behaviour of thread synchronization in following java code

When I execute the following piece of code
public class ThreadTalk {
public static void main(String[] args) {
SimpleThread obj = new SimpleThread();
Thread t = new Thread(obj, "NewThread");
t.start();
synchronized (obj) {
System.out.println("In Synchronized BLOCK");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Out of Synchronized BLOCK");
}
}
}
class SimpleThread implements Runnable {
public void run() {
System.out.println("The thread running now " + Thread.currentThread());
for (int i = 0; i < 10; i++) {
System.out.println("The val of i= " + i);
}
}
}
The output that I am getting is
In Synchronized BLOCK
The thread running now Thread[NewThread,5,main]
The val of i= 0
The val of i= 1
The val of i= 2
The val of i= 3
The val of i= 4
The val of i= 5
The val of i= 6
The val of i= 7
The val of i= 8
The val of i= 9
Out of Synchronized BLOCK
where as I am expecting an output like
In Synchronized BLOCK
Out of Synchronized BLOCK
The thread running now Thread[NewThread,5,main]
The val of i= 0
The val of i= 1
The val of i= 2
The val of i= 3
The val of i= 4
The val of i= 5
The val of i= 6
The val of i= 7
The val of i= 8
The val of i= 9
If I am putting a Lock on the SimpleThread object using the Synchronized block of main thread, how is my NewThread running when main thread is going to sleep.I mean shouldn't the NewThread wait till the Main thread has removed the lock on the SimpleThread object, as both threads are running on the same object.

run() and/or start() do not take any locks. They just run code. You need to actually have SimpleTread take the same lock as the main thread for those two threads to synchronize in some fashion.
Rather than try to synchronize on the the Runnable object, I think best practice would be to explicitly declare a separate object to use as a lock.
class ThreadTalk{
public static void main(String[] args){
Object lock = new Object();
SimpleThread obj=new SimpleThread( lock );
Thread t=new Thread(obj,"NewThread");
t.start();
synchronized(lock){
System.out.println("In Synchronized BLOCK");
try{
Thread.sleep(5000);
}catch(InterruptedException e){
e.printStackTrace();
}
System.out.println("Out of Synchronized BLOCK");
}
}
}
class SimpleThread implements Runnable{
private final Object lock;
public SimpleThread( Object lock ) { this.lock = lock;}
public void run(){
synchronized( lock ) {
System.out.println("The thread running now "+Thread.currentThread());
for(int i=0;i<10;i++){
System.out.println("The val of i= "+i);
}
}
}
}

synchronized block does not do what you think. It means that only one thread can be inside of it (or rather inside any synchronized block on the same object) at the same time. In your case, there is only one (main) thread inside the block. The other one is executing different code. That is expected.

You need to synchronize in both threads on the same object (the so-called "monitor") to make them mutually exclusive.
The simplest way to do that is to make the run() method itself synchronized:
class SimpleThread implements Runnable {
// See the synchronized modifier on the next line
public synchronized void run() {
System.out.println("The thread running now " + Thread.currentThread());
for (int i = 0; i < 10; i++) {
System.out.println("The val of i= " + i);
}
}
}
You would also need to make sure that you synchronize on the SimpleThread object before you start it in a Thread, so you need to move the t.start(); statement inside the synchronized (obj) { block. If you don't do that, the two threads are still improperly synchronized, and there's no telling which thread will run first.

how to run the main thread after all child threads have completed there exceution

I have a requirement in which 28 threads have to complete some functionality. I have created these threads as in anonymous inner classes like :
Thread t=new Thread(new Runnable(){public void run()
{//code
}}
);
t.start();
Now I want that the further execution should start after all these threads have finished there work.
Note : I am confused about join() method as it makes my threads run sequentially.
So can anyone suggest me how can I make main thread run once these threads are done with work.

Note : I am confused about join() method as it makes my threads run sequentially.
It will do that if you have code like this:
for (Runnable runnable : runnables) {
Thread t = new Thread(runnable);
t.start();
t.join();
}
However, you can start all the threads you want to run in parallel, then call join on them all. For example:
List<Thread> threads = new ArrayList<>();
for (Runnable runnable : runnables) {
Thread t = new Thread(runnable);
t.start();
threads.add(t);
}
// Now everything's running - join all the threads
for (Thread thread : threads) {
thread.join();
}
// Now you can do whatever you need to after all the
// threads have finished.
There are many other approaches, of course - starting threads directly may well not be as suitable in your code as using a higher level abstraction; it depends on what you're trying to achieve. The above should work fine though - assuming all the Runnables are able to run in parallel without blocking each other through synchronization.

Make use of CountDownLatch.
public static void main(String... args) {
final CountDownLatch latch = new CountDownLatch(28);
for(int i=0;i<28;i++) {
Thread t=new Thread(new Runnable(){
public void run()
{
try {
//code
} finally {
latch.countDown();
}
}
});
t.start();
}
latch.await();
// Continue Code
}

Use a CountDownLatch and wait for all your threads to complete. :) .
PS : I gotto agree, using join() is also correct and more efficient.
example code :
public static void main(String[] args) throws InterruptedException {
Thread t1 = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0; i < 10; i++) {
System.out.println("t1 : " + i);
}
}
});
t1.start();
Thread t2 = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0; i < 10; i++) {
System.out.println("t2 : " + i);
}
}
});
t2.start();
t1.join();
t2.join();
System.out.println("main");
}
O/P :
t1 : 0
t1 : 1
t2 : 0
t1 : 2
t1 : 3
t2 : 1
t1 : 4
t1 : 5
t2 : 2
t1 : 6
t1 : 7
t2 : 3
t1 : 8
t1 : 9
t2 : 4
t2 : 5
t2 : 6
t2 : 7
t2 : 8
t2 : 9
main

According to the behaviour you're giving for join, I'm guessing you're starting and joining the threads within a single loop.
If you check the javadoc on this page, you'll note that a call to join will halt the execution of the calling thread until the other thread has finished executing.
You might want to keep an array or a list of threads when creating them, and starting them all in one loop, and only then joining them all.
Thread[] workers = new Thread[28];
for (int i = 0; i < workers.length; i++) {
workers[i] = new Thread { ... };
}
// Start running all threads
for (Thread worker: workers) {
worker.start();
}
// Make sure all threads are completed
for (Thread worker: workers) {
worker.join(); // if the worker already stopped, it'll return immediately.
}
// Now all threads have finished running and you can start post-processing
It's not the most elegant solution, but it'll do the trick.
As mentioned by others, you should probably use a CountDownLatch (haven't used one yet, so I can't provide feedback)
Edit: I've been beaten to it by Jon Skeet, sorry for the redundant answer...

CountDownLatch is better option.
I have created dummy program.
In this program I am sum 1000 number. I created 10 thread. In main thread I am doing dome of all child thread sum. you will get understand to simply viewing the code.
package Test1;
import java.util.concurrent.CountDownLatch;
class Sum extends Thread {
private int from;
private int to;
private int sum = 0;
CountDownLatch latch;
public int getSum() {
return sum;
}
public Sum(int from, int to, CountDownLatch latch) {
this.from = from;
this.to = to;
this.latch = latch;
}
public void run() {
for (int i = from; i < to; i++) {
sum += i;
}
latch.countDown();
}
}
public class Test5 {
public static void main(String[] args) throws InterruptedException {
int n = 1000;
int tn = 10;
int from = 1;
int to;
int sum = 0;
Sum[] sumArray = new Sum[tn];
final CountDownLatch latch = new CountDownLatch(tn);
for (int i = 0; i < tn; i++) {
to = from + n / tn;
Sum s = new Sum(from, to, latch);
sumArray[i] = s;
s.start();
from = to;
}
// Thread.sleep(1000);
latch.await();
for (int i = 0; i < tn; i++) {
sum += sumArray[i].getSum();
}
System.out.println(sum);
}
}