Lets say I have n threads concurrently taking values from a shared queue:
public class WorkerThread implements Runnable{
private BlockingQueue queue;
private ArrayList<Integer> counts = new ArrayList<>();
private int count=0;
public void run(){
while(true) {
queue.pop();
count++;
}
}
}
Then for each thread, I want to count every 5 seconds how many items it has dequeued, and then store it in its own list (counts)
I've seen here Print "hello world" every X seconds how you can run some code every x seconds:
Timer t = new Timer();
t.scheduleAtFixedRate(new TimerTask(){
#Override
public void run(){
counts.add(count);
count = 0
}
}, 0, 5000);
The problem with this is that I can't access count variable and the list of counts unless they are static. But I don't want them to be static because I don't want the different threads to share those variables.
Any ideas of how to handle this?
I don't think it's possible to use scheduled execution for you case(neither Timer nor ScheduledExecutorService), because each new scheduled invocation will create a new tasks with while loop. So number of tasks will increase constantly.
If you don't need to access this list of counts in runtime i would suggest something like this one:
static class Task implements Runnable {
private final ThreadLocal<List<Integer>> counts = ThreadLocal.withInitial(ArrayList::new);
private volatile List<Integer> result = new ArrayList<>();
private BlockingQueue<Object> queue;
public Task(BlockingQueue<Object> queue) {
this.queue = queue;
}
#Override
public void run() {
int count = 0;
long start = System.nanoTime();
try {
while (!Thread.currentThread().isInterrupted()) {
queue.take();
count++;
long end = System.nanoTime();
if ((end - start) >= TimeUnit.SECONDS.toNanos(1)) {
counts.get().add(count);
count = 0;
start = end;
}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
// the last value
counts.get().add(count);
// copy the result cause it's not possible
// to access thread local variable outside of this thread
result = counts.get();
}
public List<Integer> getCounts() {
return result;
}
}
public static void main(String[] args) throws Exception {
ExecutorService executorService = Executors.newFixedThreadPool(3);
BlockingQueue<Object> blockingQueue = new LinkedBlockingQueue<>();
Task t1 = new Task(blockingQueue);
Task t2 = new Task(blockingQueue);
Task t3 = new Task(blockingQueue);
executorService.submit(t1);
executorService.submit(t2);
executorService.submit(t3);
for (int i = 0; i < 50; i++) {
blockingQueue.add(new Object());
Thread.sleep(100);
}
// unlike shutdown() interrupts running threads
executorService.shutdownNow();
executorService.awaitTermination(1, TimeUnit.SECONDS);
System.out.println("t1 " + t1.getCounts());
System.out.println("t2 " + t2.getCounts());
System.out.println("t3 " + t3.getCounts());
int total = Stream.concat(Stream.concat(t1.getCounts().stream(), t2.getCounts().stream()), t3.getCounts().stream())
.reduce(0, (a, b) -> a + b);
// 50 as expected
System.out.println(total);
}
Why not a static AtomicLong?
Or the WorkerThread(s) can publish that they poped to the TimerTask or somewhere else? And the TimerTask reads that info?
Related
While testing concurrency, I found something unexpected.
Concurrency was controlled using concurrentHashMap and AtomicLong.
public class HumanRepository {
private final static Map<Long, Human> STORE = new ConcurrentHashMap<>();
private AtomicLong sequence = new AtomicLong();
public void save(Human human) {
STORE.put(sequence.incrementAndGet(), human);
}
public int size() {
return STORE.size();
}
public Long getSeq() {
return sequence.get();
}
}
I tested saving in multiple threads.
#Test
void name() throws NoSuchMethodException, InterruptedException {
final int threads = 3_500;
final ExecutorService es = Executors.newFixedThreadPool(threads);
final CountDownLatch count = new CountDownLatch(threads);
final HumanRepository repository = new HumanRepository();
for (int i = 0; i < threads; i++) {
try {
es.execute(() -> repository.save(new Human("aa")));
} finally {
count.countDown();
}
}
count.await();
System.out.println("seq = " + repository.getSeq());
System.out.println("size = " + repository.size());
}
I tested it with 3500 threads simultaneously. The result I expected is 3500 for both seq and size.
But sometimes I get seq=3499, size=3500.
That's weird. It is strange that seq does not come out as 3500, and even though the size is 3500, it does not make sense that seq is 3499.
I don't know why the data number and seq in the map are not the same and 3500 is not coming out.
** If you do Thread.sleep(400L); after count.await();, surprisingly, the value of seq is 3500
You are not actually waiting for all tasks to complete. Which means that if you get the 3500/3500 output, it's by chance.
Specifically, you decrease the countdown latch on the main thread after scheduling the job, instead of inside of the job, once it's done. That means your countdownlatch is basically just another glorified loop variable that doesn't do any inter-thread communication. Try something like this instead:
for (int i = 0; i < threads; i++) {
es.execute(() -> {
repository.save(new Human("aa"));
count.countDown();
});
}
You are calling count.countDown() outside the thread executing the HumanRepository.save(). So its possible that the main thread is not synchronized for the completion of the threads.
So you may see the results of repository.getSeq() while one thread is running. Can you try with the following code?
final int threads = 3_500;
final ExecutorService es = Executors.newFixedThreadPool(threads);
final CountDownLatch count = new CountDownLatch(threads);
final HumanRepository repository = new HumanRepository();
for (int i = 0; i < threads; i++) {
try {
es.execute(() -> {
repository.save(new Human("aa"));
count.countDown();
});
} finally {
}
}
count.await();
System.out.println("seq = " + repository.getSeq());
System.out.println("size = " + repository.size());
public class ConTest {
#Test
void name2() {
final MyCounter myCounter = new MyCounter();
final Thread t1 = new Thread(() ->
myCounter.increment()
);
final Thread t2 = new Thread(() ->
myCounter.increment()
);
t1.start();
t2.start();
System.out.println(myCounter.count);
}
#Test
void name3() {
final MyCounter myCounter = new MyCounter();
final ExecutorService service = Executors.newFixedThreadPool(2);
for (int i = 0; i < 2; i++) {
service.execute(() -> {
myCounter.increment();
});
}
System.out.println(myCounter.count);
}
static class MyCounter {
private AtomicLong count = new AtomicLong();
public void increment() {
count.incrementAndGet();
}
}
}
AtomicLong is safe when multi thread.
That is, in the example above, it was executed with 2 threads, so the result should be 2 no matter how many times it is executed.
However, after trying both tests several times, the result is sometimes 1. Why is this happening?
You aren't waiting for any of the background threads or tasks to end before you print the value of the counter. To wait on the tasks to exit, you'll need to add this for threads:
t1.join();
t2.join();
Add this for the service, which prevents new tasks being added and waits a sensible period for them to end:
service.shutdown();
boolean done = awaitTermination(pickSuitablyLongPeriod, TimeUnit.MILLISECONDS);
Once you have ensured the background tasks are completed, the correct result should be printed when you run:
System.out.println(myCounter.count);
This is because the threads are still processing when you call the System.out.println. In this case you would need to block the main thread before you print out the counter.
in the example of the Executor you can just await the termination:
final ExecutorService service = Executors.newFixedThreadPool(2);
final MyCounter myCounter = new MyCounter();
for (int i = 0; i < 100; i++) {
service.submit(myCounter::increment);
}
service.shutdown();
while (!service.awaitTermination(100, TimeUnit.MILLISECONDS)) {
System.out.println("waiting");
}
System.out.println(myCounter.count);
you should avoid to block in productive code, have a look at the Publish/Subscribe design pattern
dont forget to use shutdown() with Executors
see the comments here :
// Here you start the 2 threads
for (int i = 0; i < 2; i++) {
service.execute(() -> {
myCounter.increment();
});
}
// we are not sure here that your 2 threads terminate their tasks or not !!
// the print will be executed by the Main Thread and maybe before the 2 threads terminate their
// job ,
// maybe just one terminate , maybe no one from your 2 threads increment the count .
System.out.println(myCounter.count);
You can use Future class , instead of execute you can use submit() , the retrun type will be of Type Futre<?> (accept void ) , after that with the Future object returned the method get() will block the execution until the result returned from the service :
Example method name3() : will return always 2
void name3() {
final MyCounter myCounter = new MyCounter();
final ExecutorService service = Executors.newFixedThreadPool(2);
Future<?> f = null;
for (int i = 0; i < 2; i++) {
f =service.submit(() -> {
myCounter.increment();
});
try {
f.get();
} catch (InterruptedException | ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println(myCounter.count);
service.shutdown();
}
In addition to the above answers, you could add some prints to better understand what is happening.
In summary. You need to wait for the threads to finish executing before expecting the results, so it is not an issue of AtomicLong.
I modified the code, added some prints, and here are results from an execution.
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import org.junit.jupiter.api.Test;
public class ConTest {
#Test
void name2() {
final MyCounter myCounter = new MyCounter();
final Thread t1 = new Thread(() -> {
myCounter.increment();
System.out.println("Counter increment t1 completed and the value is " + myCounter.getCount());
});
final Thread t2 = new Thread(() -> {
myCounter.increment();
System.out.println("Counter increment t2 completed and the value is " + myCounter.getCount());
});
t1.start();
t2.start();
System.out.println(myCounter.count.get());
}
#Test
void name3() {
final MyCounter myCounter = new MyCounter();
final ExecutorService service = Executors.newFixedThreadPool(2);
for (int i = 0; i < 2; i++) {
service.execute(() -> {
myCounter.increment();
System.out.println("incrementing for count and the value is " + myCounter.getCount());
});
}
System.out.println(myCounter.count.get());
}
class MyCounter {
private AtomicLong count = new AtomicLong();
public void increment() {
count.incrementAndGet();
}
public long getCount(){
return count.get();
}
}
}
Results (name2)
1
Counter increment t1 completed and the value is 1
Counter increment t2 completed and the value is 2
Results (name3)
incrementing for count and the value is 1
1
incrementing for count and the value is 2
You could also use a debugger to have a better understanding.
For our assignment for class, we have to count the amount of words in a txt file by splitting it into n segments, which we are supposed to be able to set before launching the programm. Each segment should then get its own thread, which counts the words and then stops. At the end, the main thread should collect all the individual word counts and add them together.
This is (part of) what I wrote so far
for (int i = 0; i < segments; i++){
Thread thread = new Thread();
thread.start();
int words = counting(stringarray[i]);
totalwords += words;
long nanos = ManagementFactory.getThreadMXBean().getThreadCpuTime(Thread.currentThread().getId());
System.out.println("This Thread read " + words + " words. The total word count now is " + totalwords +
". The time it took to finish for this thread is " + nanos +".");
System.out.println("Number of active threads from the given thread: " + Thread.activeCount());
}
Now, while this gets the primary job done (counting the words in different threads and adding them to the total), I dont know how to just "leave the thread be" and then add the individual wordcounts together after every thread has done its job.
Additionally, while this is definitely starting multiple threads, it only ever prints out that I have 2, or maybe 3 threads running at a time, even if I split the txt into 100 segments. Is there a way to have them all run at the same time?
The wording of the question suggest that each thread has its own counter, so I would declare a thread class:
public class WordCounter extends Thread {
private String text;
private int count;
public WordCounter(String text) {
this.text = text;
}
public int getCount() {
return count;
}
#Override
public void run() {
count = counting(text);
}
}
and use it as follows:
WordCounter[] threads = new WordCounter[segments];
for (int i = 0; i < segments; ++i) {
threads[i] = new WordCounter(stringarray[i]);
threads[i].start();
}
int total = 0;
for (int i = 0; i < segments; ++i) {
threads[i].join();
total += threads[i].getCount();
}
You may use next code snippet as a basis.
Note, that in case you increment common variable in different threads, this operation has to be thread-safe. That's why AtomicInteger variable is used as a counter
final List<String> segments = new ArrayList<>();
//TODO:Fill segments ... this is up to you
//In case threads will increment same variable it has to be thread-safe
final AtomicInteger worldCount = new AtomicInteger();
//Create Thread for each segment (this is definitely not optimal)
List<Thread> workers = new ArrayList<>(segments.size());
for (int i = 0; i < segments.size(); i++) {
final String segment = segments.get(i);
Thread worker = new Thread(new Runnable() {
#Override
public void run() {
//increment worldCount
worldCount.addAndGet(counting(segment));
}
});
workers.add(worker);
worker.start();
}
//Wait until all Threads are finished
for (Thread worker : workers) {
worker.join();
}
int result = worldCount.get();
Same solutions, but with Executors:
final List<String> segments = new ArrayList<>();
segments.add("seg1");
segments.add("seg2");
segments.add("seg 3");
final AtomicInteger worldCount = new AtomicInteger();
List<Future> workers = new ArrayList<>(segments.size());
ExecutorService executor = Executors.newFixedThreadPool(segments.size());
for (String segment : segments) {
Future<Integer> worker = executor.submit(() -> worldCount.addAndGet(counting(segment)));
workers.add(worker);
}
executor.shutdown();
if (!executor.awaitTermination(5, TimeUnit.SECONDS)) {
System.out.println("Still waiting...");
System.exit(0);
}
int result = worldCount.get();
System.out.println("result = " + result);
I have a requirement in which 28 threads have to complete some functionality. I have created these threads as in anonymous inner classes like :
Thread t=new Thread(new Runnable(){public void run()
{//code
}}
);
t.start();
Now I want that the further execution should start after all these threads have finished there work.
Note : I am confused about join() method as it makes my threads run sequentially.
So can anyone suggest me how can I make main thread run once these threads are done with work.
Note : I am confused about join() method as it makes my threads run sequentially.
It will do that if you have code like this:
for (Runnable runnable : runnables) {
Thread t = new Thread(runnable);
t.start();
t.join();
}
However, you can start all the threads you want to run in parallel, then call join on them all. For example:
List<Thread> threads = new ArrayList<>();
for (Runnable runnable : runnables) {
Thread t = new Thread(runnable);
t.start();
threads.add(t);
}
// Now everything's running - join all the threads
for (Thread thread : threads) {
thread.join();
}
// Now you can do whatever you need to after all the
// threads have finished.
There are many other approaches, of course - starting threads directly may well not be as suitable in your code as using a higher level abstraction; it depends on what you're trying to achieve. The above should work fine though - assuming all the Runnables are able to run in parallel without blocking each other through synchronization.
Make use of CountDownLatch.
public static void main(String... args) {
final CountDownLatch latch = new CountDownLatch(28);
for(int i=0;i<28;i++) {
Thread t=new Thread(new Runnable(){
public void run()
{
try {
//code
} finally {
latch.countDown();
}
}
});
t.start();
}
latch.await();
// Continue Code
}
Use a CountDownLatch and wait for all your threads to complete. :) .
PS : I gotto agree, using join() is also correct and more efficient.
example code :
public static void main(String[] args) throws InterruptedException {
Thread t1 = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0; i < 10; i++) {
System.out.println("t1 : " + i);
}
}
});
t1.start();
Thread t2 = new Thread(new Runnable() {
#Override
public void run() {
for (int i = 0; i < 10; i++) {
System.out.println("t2 : " + i);
}
}
});
t2.start();
t1.join();
t2.join();
System.out.println("main");
}
O/P :
t1 : 0
t1 : 1
t2 : 0
t1 : 2
t1 : 3
t2 : 1
t1 : 4
t1 : 5
t2 : 2
t1 : 6
t1 : 7
t2 : 3
t1 : 8
t1 : 9
t2 : 4
t2 : 5
t2 : 6
t2 : 7
t2 : 8
t2 : 9
main
According to the behaviour you're giving for join, I'm guessing you're starting and joining the threads within a single loop.
If you check the javadoc on this page, you'll note that a call to join will halt the execution of the calling thread until the other thread has finished executing.
You might want to keep an array or a list of threads when creating them, and starting them all in one loop, and only then joining them all.
Thread[] workers = new Thread[28];
for (int i = 0; i < workers.length; i++) {
workers[i] = new Thread { ... };
}
// Start running all threads
for (Thread worker: workers) {
worker.start();
}
// Make sure all threads are completed
for (Thread worker: workers) {
worker.join(); // if the worker already stopped, it'll return immediately.
}
// Now all threads have finished running and you can start post-processing
It's not the most elegant solution, but it'll do the trick.
As mentioned by others, you should probably use a CountDownLatch (haven't used one yet, so I can't provide feedback)
Edit: I've been beaten to it by Jon Skeet, sorry for the redundant answer...
CountDownLatch is better option.
I have created dummy program.
In this program I am sum 1000 number. I created 10 thread. In main thread I am doing dome of all child thread sum. you will get understand to simply viewing the code.
package Test1;
import java.util.concurrent.CountDownLatch;
class Sum extends Thread {
private int from;
private int to;
private int sum = 0;
CountDownLatch latch;
public int getSum() {
return sum;
}
public Sum(int from, int to, CountDownLatch latch) {
this.from = from;
this.to = to;
this.latch = latch;
}
public void run() {
for (int i = from; i < to; i++) {
sum += i;
}
latch.countDown();
}
}
public class Test5 {
public static void main(String[] args) throws InterruptedException {
int n = 1000;
int tn = 10;
int from = 1;
int to;
int sum = 0;
Sum[] sumArray = new Sum[tn];
final CountDownLatch latch = new CountDownLatch(tn);
for (int i = 0; i < tn; i++) {
to = from + n / tn;
Sum s = new Sum(from, to, latch);
sumArray[i] = s;
s.start();
from = to;
}
// Thread.sleep(1000);
latch.await();
for (int i = 0; i < tn; i++) {
sum += sumArray[i].getSum();
}
System.out.println(sum);
}
}
I have a list of tasks and a limited number of threads. The goal is to time how long the tasks take to finish using this number of threads.
I know something is wrong with the way I am using threads and Runnable object. I am new to them and can't seem to figure out how to fix it.
It errors with a java.lang.OutOfMemoryError: Java heap space error on the line worker.start() after a few seconds.
Here is my code:
public class Tasks {
static Timer timer; //times how long it takes to complete all tasks
public static void main(String[] args) {
Job t1 = new Sleep(5);
Job t2 = new Sum(1000);
Job t3 = new Sleep(3);
Job t4 = new Sleep(10);
Job t5 = new Sum(10);
Graph g = new Graph(5);
g.getNumEdges();
g.addEdge(t1, t2);
g.addEdge(t2, t3);
g.addEdge(t2, t4);
g.addEdge(t3, t5);
g.addEdge(t4, t5);
//System.out.println(t5.getPredecessor());
System.out.println(parseGraph(g, 2));
}
public static String parseGraph(Graph graph, int K)
{
long startTime = System.nanoTime();//start timer
int numThreads = K;
ArrayList<Job> x = graph.getNodes();
//check for cycles
CycleFinder dc = new CycleFinder(graph);
if(dc.hasCycle()==true)
{
System.out.println(dc.cycle());
return ("The graph has cycles and could not be parsed through.");
}
List<Thread> threads = new ArrayList<Thread>();
ArrayList<Job> ready = new ArrayList<Job>();
while (x.isEmpty()!= true)
{
for(int i=0; i<x.size(); i++)
{
Job y= x.get(i);
System.out.println(y);
if(y.getComplete()== true)
{
ready.remove(y);
graph.removeNode(y);
x.remove(y);
}
if(y.getPredecessor().isEmpty() || y.getPredecessor() ==null)
ready.add(y);
}
for (int i = 0; i < numThreads && i < ready.size(); i++) {
System.out.println("test");
Runnable task = new MyRunnable(ready.get(i));
Thread worker = new Thread(task);
worker.setName(String.valueOf(i));
worker.start();
threads.add(worker);
}
//int running = 0;
//do {
//running = 0;
//for (Thread thread : threads) {
//if (thread.isAlive()) {
// running++;
// }
// }System.out.println("We have " + running + " running threads. ");
// } while (running > 0);
}
long endTime = System.nanoTime();
long duration = endTime - startTime;
return ("The Tasks took " + (duration/1000) + " seconds");
}
}
You don't need to.. After thread has completed the run() method of Runnable, it should be completed.
The OOM error you are facing is something to do with the logic inside the run() method of the Runnable.