I am trying to write a part of a multithreaded program where each thread from a fixed thread pool tries to fetch an object from a Queue and if the Queue is empty the thread waits.
The problem I am experiencing is that the memory used by the program keeps increasing.
public class Ex3 {
public static LinkedBlockingQueue<Integer> myLBQ = new LinkedBlockingQueue<Integer>(10);
public static void main(String argc[]) throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(3);
myLBQ.add(new Integer(1));
for (;;) {
executor.execute(new MyHandler(myLBQ));
}
}
}
class MyHandler implements Runnable {
LinkedBlockingQueue<Integer> myLBQ;
MyHandler(LinkedBlockingQueue<Integer> myLBQ) {
this.myLBQ = myLBQ;
}
public void run() {
try {
myLBQ.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I don't understand why the executor.execute keeps firing when the threads should be waiting for an item to be added to the Queue. How do I modify my code to reflect this?
This adds tasks to the executor as fast as it can.
for (;;) {
executor.execute(new MyHandler(myLBQ));
}
This will consume about 200 MB per second. It doesn't have anything to do with whether there are tasks to perform or not.
If you don't want to do this I suggest you move the loop to the runnable and add only one. This will cause it to wait for tasks forever.
A better approach is to use the ExecutorService's builtin queue to queue tasks.
ExecutorService executor = Executors.newFixedThreadPool(3);
final int taskId = 1;
executor.submit(new Runnable() {
#Override
public void run() {
doSomething(taskId);
}
});
executor.shutdown();
This does the same thing, but is much simpler IMHO.
it's because you're creating a gazillion instances of MyHandler and inserting them in the internal queue of the executor.
That infinite for loop is quite mean.
Related
I'm having trouble cleaning up my threads in a threaded application. I have various runnables that are kicked off using a thread pool. Most of these threads are normal runnables that only execute once on a Scheduled Fixed Rate. Two of them I have are scheduled to run once and have a while(true) loop in them. When I get to cleaning up the threads, it seems I'm having trouble calling ScheduledFuture.close(false) on the threads with the while loop in them. They don't seem to close.
An example of the format of these threads with while loops are:
public void run()
{
while (true)
{
QueryItem qi = null;
String query = null;
try
{
// This is a BlockingQueue!
qi = (QueryItem) FB2DatabaseRecorder.dbProcQueue.take();
query = qi.getQuery();
} catch (InterruptedException e)
{
errorLog.error("Unable to fetch message from message processing queue.");
}
// DO SOME STUFF
}
}
When I try to do the .close() on this thread, it's typically sitting at the blocking queue waiting for an item to come in. Before closing the threads I ensure that the queues are flushed as to not leave any data behind.
Is there a better way to close this type of thread? It seems like it is just not dying with handle.close(false);
A better way to shutdown your worker thread is to use Thread.interrupt().
Your worker thread is waiting on the take call, and the take call throws if the thread is interrupted. You can send an interrupt and manage the shutdown in the catch clause. In code,
package stackOv;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
class MyQueueWorker implements Runnable {
private BlockingQueue<Object> q;
MyQueueWorker(BlockingQueue<Object> q) {
this.q = q;
}
#Override
public void run() {
while (true) {
try {
Object item = q.take();
// work here
System.out.println("obj=" + item);
} catch (InterruptedException e) {
System.out.println("worker thread is interrupted");
break;
}
}
System.out.println("interrupted, exiting worker thread");
}
}
public class InterruptTake {
public static void main(String[] args) throws Exception {
BlockingQueue<Object> q = new LinkedBlockingQueue<>();
Thread worker = new Thread( new MyQueueWorker(q ), "worker" );
worker.start();
q.put("hello");
q.put("world");
q.put("waiting..");
Thread.sleep(1000);
worker.interrupt();
}
}
I have the following piece of code which I expected to print "DONE" at the end. But when I ran, "DONE" was never printed and the JVM in fact never terminated.
What did I do wrong?
// File: Simple.java
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class Simple {
public static void main(String[] args) throws Exception{
doTest(3);
}
private static void doTest(final int times) {
ScheduledExecutorService tp = Executors.newScheduledThreadPool(times);
Thread[] runnables = new Thread[times];
for (int i = 0; i < runnables.length; ++i) {
runnables[i] = new Thread(new MyRunnable());
}
// schedule for them all to run
for (Thread t : runnables) {
tp.schedule(t, 1, TimeUnit.SECONDS);
}
try {
tp.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
System.out.println("DONE!");
}catch (InterruptedException e) {
e.printStackTrace();
}
}
static class MyRunnable implements Runnable {
#Override
public void run() {
System.out.println("hello world");
}
}
}
There are two things that you're doing wrong here.
First off, if you're using an ExecutorService, you shouldn't then also be creating your own threads. Just submit Runnables to the executor directly - the executor service has its own collection of threads, and runs anything you submit on its own threads, so the threads you created won't even get started.
Second, if you're done with an ExecutorService, and are going to wait until it's terminated, you need to call shutdown() on the executor service after you submit your last job.
Executors.newScheduledThreadPool(times) uses Executors.defaultThreadFactory() for its ThreadFactory.
Here is the documentation
When a Java Virtual Machine starts up, there is usually a single
non-daemon thread (which typically calls the method named main of some
designated class). The Java Virtual Machine continues to execute
threads until either of the following occurs:
The exit method of class Runtime has been called and the security manager has permitted the exit operation to take place.
All threads that are not daemon threads have died, either by returning from the call to the run method or by throwing an exception
that propagates beyond the run method.
So, here is what you had; but I changed 3 things.
Added the shutdown hook
Made the schedule delayed by an extra second for each to demo that the shutdown is being called even before some of the threads are being called to run by the scheduler.
Lastly, you were telling it to wait forever using the Long.MAX. But I think you were doing it because the shutdown feels like it would shutdown now. But it won't.
public static void main(String[] args) throws Exception {
doTest(3);
}
private static void doTest(final int times) {
ScheduledExecutorService tp = Executors.newScheduledThreadPool(times);
Thread[] runnables = new Thread[times];
for (int i = 0; i < runnables.length; ++i) {
runnables[i] = new Thread(new MyRunnable());
}
// schedule for them all to run
int i = 1;
for (Thread t : runnables) {
tp.schedule(t, i++, TimeUnit.SECONDS);
}
System.out.println("Calling shutdown");
tp.shutdown();
}
static class MyRunnable implements Runnable {
#Override
public void run() {
System.out.println("hello world");
}
}
Hope that answers your question. Now, you're kinda duplicating stuff.
If you are going to use the executerservice, you should just tell it to schedule stuff for you.
public static void main(String[] args) throws Exception {
doTest(3);
}
private static void doTest(final int times) {
ScheduledExecutorService tp = Executors.newScheduledThreadPool(times);
for (int i = 0; i < times; ++i) {
tp.schedule(new MyRunnable(), 1, TimeUnit.SECONDS);
}
System.out.println("Calling shutdown");
tp.shutdown();
}
static class MyRunnable implements Runnable {
#Override
public void run() {
System.out.println("hello world");
}
}
You forgot to shutdown your ExecutorService:
tp.shutdown(); // <-- add this
tp.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
Also I should mention that creating Threads to use them as Runnables is not only meaningless, but can be misleading as well. You should really use Runnable instead of Thread for runnables variable.
Runnable[] runnables = new Runnable[times];
I have a web service which is continuously accessed by more than 20 servers which are sending data. I have used activeMQ where data is queued for some time and than using async task this data is dequed.
My async task thread class is shown below.
public class myConsumer {
public void asyncConsumer() throws InterruptedException, ExecutionException{
final MyReceiver receiver = new MyReceiver();
final ExecutorService executorService = Executors.newSingleThreadExecutor();
try{
Future future = executorService.submit(new Runnable() {
public void run() {
receiver.receiveMessage();
}
});
executorService.shutdown();
executorService.awaitTermination(5, TimeUnit.SECONDS);
}catch (InterruptedException e) {
logger.error("tasks interrupted");
}finally {
if (!executorService.isTerminated()) {
logger.error("cancel non-finished tasks");
}
executorService.shutdownNow();
}
}
}
I have 10000 of threads running. My applcation server is stopped due to unusual behavior. How to handle this many no of thread using above code.
You should use a thread pool executor rather than single thread executor, and make sure it's only one instance of the thread pool receiving messages. This way you can keep track of and limit the amount of concurrent threads.
Here's an example of how it can be done. By having the executorservice static you assure that it's only one instance, and it's limited to max 10 simultaneous threads. When you call asyncConsumer to process a received message a MyReceiver (which I assume is going to process the message) is created and invoked inside the thread pool.
public class MyConsumer {
static final ExecutorService executorService = Executors.newFixedThreadPool(10);
public void asyncConsumer() {
Future future = executorService.submit(new Runnable() {
public void run() {
new MyReceiver().receiveMessage();
}
});
}
}
The server application I am running gets multiple requests for tasks which I want to handle using a task system.
Each task is represented as a Runnable that will demand n number of threads from a thread pool where n is smaller or equal to the thread pool size. The thread pool of course is necessary in order to not overload the CPU with too many threads.
However, some of those tasks can be multi threaded and some can not. That is why it might be necessary for one task to wait for all its specific threads to finish in order to merge the results from those threads for the final result.
If one uses multiple Thread instances one might join those like this:
try {
// Wait for all threads to finish their tasks
for (Thread thread : threads) {
thread.join();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
// Finish job here ..
but I'd need something like this using java.util.concurrent.Executor or anything similar that works with a thread pool.
You can use ExecutorService along with a CyclicBarrier for each task as follows:
public class ThreadedTask implements Runnable {
CyclicBarrier barrier;
public ThreadedTask(CyclicBarrier barrier) {
this.barrier = barrier;
}
#Override
public void run() {
// do something
barrier.await();
}
}
ExecutorService executor = Executors.newFixedThreadPool(pool_size);
...
CyclicBarrier barrier = new CyclicBarrier(n+1);
for(int i=0; i<n; i++) {
// launch all tasks
executor.submit(new ThreadedTask(barrier));
}
// waits for the tasks to finish or timeout
barrier.await(seconds, TimeUnit.SECONDS);
If I understand you correctly, you will need something like this (but your architecture seems too complicated):
class MyTask implements Runnable {
#Override
public void run() {
// some work
}
}
After that:
ExecutorService executorService = Executors.newFixedThreadPool(2000);
ArrayList<Future> futures = new ArrayList<>();
futures.add(executorService.submit(new MyTask()));
futures.add(executorService.submit(new MyTask()));
futures.add(executorService.submit(new MyTask()));
for (Future future: futures) {
try {
future.get(100, TimeUnit.SECONDS);
} catch (Throwable cause) {
// process cause
}
}
Each future.get() will wait for task ending (max 100 seconds in this example).
This question already has answers here:
Run Java Threads sequentially
(13 answers)
Closed 9 years ago.
how to Execute threads sequentially ? (e.g i have 3 threads T1,T2,T3 and i want to start these threads same time but ensure that they should run sequentially one after other like first T1 then T2 and at last T3.)
Thread t1= new Thread(new Runnable()
{
void run()
{System.out.println("inside Thread1");}
});
Thread t2= new Thread(new Runnable()
{
void run()
{System.out.println("inside Thread2");}
});
Thread t3= new Thread(new Runnable()
{
void run()
{System.out.println("inside Thread3");}
});
t1.start();
t2.strat();
t3.start();
output:
inside Thread1
inside Thread2
inside Thread3
each time u run o/p should be as shown above.
In my humble opinion, you perhaps do not need threads, just call T1(), T2(), T3() methods sequentially in your code?
Threads are used to run multiple tasks in parallel.
You can synchronize these threads through flag/s. You can also use inbuilt synchronizers provided by Java like BlockingQueue.
Use BlockingQueues to synchronize the threads
final BlockingQueue q1 = new SynchronousQueue();
final BlockingQueue q2 = new SynchronousQueue();
Thread t1 = new Thread() {
public void run() {
...
try {
q1.put(new Object());
} catch (InterruptedException e) {
}
};
};
Thread t2 = new Thread() {
public void run() {
try {
q1.take();
...
q2.put(new Object());
} catch (InterruptedException e) {
}
}
};
Thread t3 = new Thread() {
public void run() {
try {
q2.take();
...
} catch (InterruptedException e) {
}
}
};
t1.start();
t2.start();
t3.start();
Threads are used to run multiple tasks at a same time.
In your case you need different methods called sequentially rather than Threads.
You should use:
class Methods_Than_Threads{
void T1()
{
//something
}
void T2()
{
//something
}
void T3()
{
//something
}
public static void main(String []args)
{
T1();//First T1
T2();//Second T2
T3();//Third T3
}
}
You should separate the actual tasks from how they are executed. I.e. don't extend Thread and overwrite run, instead implement Runnable as the task and don't care about the way it is executed.
That way you can design (+change later) the way you execute tasks independently from the actual implementation of a task.
E.g. Call each .run() directly if you want to execute them after each other or let some Executor handle them or even run them via new Thread manually.
If they have to wait on each other you could also use a Future. For example:
class ProcessingChainElement implements Callable<String> {
private final Future<String> previous;
public ProcessingChainElement(Future<String> previousResult) {
previous = previousResult;
}
#Override
public String call() throws Exception {
// prepare something that may take some time but does not depend on
// any previous result
Thread.sleep(500);
// get result from previous task, this blocks until it is available
result = previous.get() + " [" + System.currentTimeMillis() + "]";
return result;
}
}
And build a chain of tasks that can be executed in any way you want.
ExecutorService executor = Executors.newFixedThreadPool(3);
Future<String> result1 = executor.submit(...
Future<String> result2 = executor.submit(new ProcessingChainElement(result1));
...
Result is that each task can wait on results of a previous task but may very well run in parallel if there is anything that can be run in parallel.
Example http://ideone.com/VAg8q3 demonstrates that 3 tasks that take >= 500ms each and depend on each other could be done much quicker than actually running them in sequence.