We Keep Coding

Java parallel tasks , only executing once

This code I have is not executing tasks in parallel,
it only executes the code in this case once (whatever is in the for loop, but it should be 2) :
public class mqDirect {
public static void main(String args[]) throws Exception {
int parallelism = 2;
ExecutorService executorService =
Executors.newFixedThreadPool(parallelism);
Semaphore semaphore = new Semaphore(parallelism);
for (int i = 0; i < 1; i++) {
try {
semaphore.acquire();
// snip ... do stuff..
semaphore.release();
} catch (Throwable throwable) {
semaphore.release();
}
executorService.shutdownNow();
}
}
}

In Java the main way to make code work in parallel is to create a Thread with a new Runnable as a constructor parameter. You then need to start it.
There are many tutorials to help you get this to happen properly.
As your code stands you are merely creating an ExecutorService (and not using it), creating a Semaphore (which should be done in the thread but isn't), performing some process and then shutting down the Executor.
BTW: ShutDownNow is probably not what you want, you should just use ShutDown.

OK, So I found this good tutorial
http://programmingexamples.wikidot.com/threadpoolexecutor
And I have done something like
public class mqDirect {
int poolSize = 2;
int maxPoolSize = 2;
long keepAliveTime = 10;
ThreadPoolExecutor threadPool = null;
final ArrayBlockingQueue<Runnable> queue = new ArrayBlockingQueue<Runnable>(
5);
public mqDirect()
{
threadPool = new ThreadPoolExecutor(poolSize, maxPoolSize,
keepAliveTime, TimeUnit.SECONDS, queue);
}
public void runTask(Runnable task)
{
threadPool.execute(task);
System.out.println("Task count.." + queue.size());
}
public void shutDown()
{
threadPool.shutdown();
}
public static void main (String args[]) throws Exception
{
mqDirect mtpe = new mqDirect();
// start first one
mtpe.runTask(new Runnable()
{
public void run()
{
for (int i = 0; i < 2; i++)
{
try
{
System.out.println("First Task");
runMqTests();
Thread.sleep(1000);
} catch (InterruptedException ie)
{
}
}
}
});
// start second one
/*
* try{ Thread.sleep(500); }catch(InterruptedException
* ie){}
*/
mtpe.runTask(new Runnable()
{
public void run()
{
for (int i = 0; i < 2; i++)
{
try
{
System.out.println("Second Task");
runMqTests();
Thread.sleep(1000);
} catch (InterruptedException ie)
{
}
}
}
});
mtpe.shutDown();
// runMqTests();
}
And it works !
But the problem is , this duplicated code ... runMqtests() is the same task, is there a way to specify it to run in parallel without duplicating the code?
The example I based this off is assuming each task is different.

This code I have is not executing tasks in parallel, it only executes the code in this case once (whatever is in the for loop, but it should be 2) :
Just because you instantiate an ExecutorService instance doesn't mean that things magically run in parallel. You actually need to use that object aside from just shutting it down.
If you want the stuff in the loop to run in the threads in the service then you need to do something like:
int parallelism = 2;
ExecutorService executorService = Executors.newFixedThreadPool(parallelism);
for (int i = 0; i < parallelism; i++) {
executorService.submit(() -> {
// the code you want to be run by the threads in the exector-service
// ...
});
}
// once you have submitted all of the jobs, you can shut it down
executorService.shutdown();
// you might want to call executorService.awaitTermination(...) here
It is important to note that this will run your code in the service but there are no guarantees that it will be run "in parallel". This depends on your number of processors and the race conditions inherent with threads. For example, the first task might start up, run, and finish its code before the 2nd one starts. That's the nature of threaded programs which are by design asynchronous.
If, however, you have at least 2 cores, and the code that you submit to be run by the executor-service takes a long time to run then most likely they will be running at the same time at some point.
Lastly, as #OldCurmudgeon points out, you should call shutdown() on the service which allows current jobs already submitted to the service to run as opposed to shutdownNow() which cancels and queued jobs and also calls thread.interrupt() on any running jobs.
Hope this helps.

Multithreading arraylist of objects

My program has an arraylist of websites which I do I/O with image processing, scrape data from sites and update/insert into database. Right now it is slow because all of the I/O being done. I would like to speed this up by allowing my program to run with threads. Nothing is ever removed from the list and every website in the list is separate from each other so to me it seems okay to have instances looping through the list at the same time to speed this up.
Let's say my list is 10 websites, right now of course it's looping through position 0 through 9 until my program is done processing for all websites.
And let's say I want to have 3 threads looping through this list of 10 websites at once doing all the I/O and database updates in their own separate space at the same time but using the same list.
website.get(0) // thread1
website.get(1) // thread2
website.get(2) // thread3
Then say if thread2 reaches the end of the loop it first it comes back and works on the next position
website.get(3) // thread2
Then thread3 completes and gets the next position
website.get(4) // thread3
and then thread1 finally completes and works on the next position
website.get(5) // thread1
etc until it's done. Is this easy to set up? Is there somewhere I can find a good example of it being done? I've looked online to try to find somewhere else talking about my scenario but I haven't found it.

In my app, I use ExecutorService like this, and it works well:
Main code:
ExecutorService pool = Executors.newFixedThreadPool(3); //number of concurrent threads
for (String name : website) { //Your ArrayList
pool.submit(new DownloadTask(name, toPath));
}
pool.shutdown();
pool.awaitTermination(5, TimeUnit.SECONDS); //Wait for all the threads to finish, adjust as needed.
The actual class where you do the work:
private static class DownloadTask implements Runnable {
private String name;
private final String toPath;
public DownloadTask(String name, String toPath) {
this.name = name;
this.toPath = toPath;
}
#Override
public void run() {
//Do your parsing / downloading / etc. here.
}
}
Some cautions:
If you are using a database, you have to ensure that you don't have two threads writing to that database at the same time.
See here for more info.

As mentioned in other comments/answer you just need a thread pool executor with fixed size (say 3 as per your example) which runs 3 threads which iterate over the same list without picking up duplicate websites.
So apart from thread pool executor, you probably need to also need to correctly work out the next index in each thread to pick the element from that list in such a way that thread does not pick up same element from list and also not miss any element.
Hence i think you can use BlockingQueue instead of list which eliminates the index calculation part and guarantees that the element is correctly picked from the collection.
public class WebsitesHandler {
public static void main(String[] args) {
BlockingQueue<Object> websites = new LinkedBlockingQueue<>();
ExecutorService executorService = Executors.newFixedThreadPool(3);
Worker[] workers = new Worker[3];
for (int i = 0; i < workers.length; i++) {
workers[i] = new Worker(websites);
}
try {
executorService.invokeAll(Arrays.asList(workers));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private static class Worker implements Callable {
private BlockingQueue<Object> websites;
public Worker(BlockingQueue<Object> websites) {
this.websites = websites;
}
public String call() {
try {
Object website;
while ((website = websites.poll(1, TimeUnit.SECONDS)) != null) {
// execute the task
}
} catch (InterruptedException e) {
e.printStackTrace();
}
return "done";
}
}
}

I think you need to update yourself with latest version of java i.e Java8
And study about Streams API,That will definitely solve your problem

how to maintain a list of threads?

I have hundreds of files to process. I do each file one at a time and it takes 30 minutes.
I'm thinking I can do this processing in 10 simultaneous threads, 10 files at a time, and I might be able to do it in 3 minutes instead of 30.
My question is, what is the "correct" way to manage my 10 threads? And when one is done, create a new one to a max number of 10.
This is what I have so far ... is this the "correct" way to do it?
public class ThreadTest1 {
public static int idCounter = 0;
public class MyThread extends Thread {
private int id;
public MyThread() {
this.id = idCounter++;
}
public void run() {
// this run method represents the long-running file processing
System.out.println("I'm thread '"+this.id+"' and I'm going to sleep for 5 seconds!");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("I'm thread '"+this.id+"' and I'm done sleeping!");
}
}
public void go() {
int MAX_NUM_THREADS = 10;
List<MyThread> threads = new ArrayList<MyThread>();
// this for loop represents the 200 files that need to be processed
for (int i=0; i<200; i++) {
// if we've reached the max num of threads ...
while (threads.size() == MAX_NUM_THREADS) {
// loop through the threads until we find a dead one and remove it
for (MyThread t : threads) {
if (!t.isAlive()) {
threads.remove(t);
break;
}
}
}
// add new thread
MyThread t = new MyThread();
threads.add(t);
t.start();
}
}
public static void main(String[] args) {
new ThreadTest1().go();
}
}

You can use ExecutorService to manage you threads.
And you can add while loop to thread run method to execute file processing task repeatedly.
Also you can read about BlockingQueue usage. I think it will fit perfectly to allocate new files (tasks) between threads.

I would suggest using Camel's File component if you are open to it. The component will handle all the issues with concurrency to ensure that multiple threads don't try to process the same file. The biggest challenge with making your code multi-threaded is making sure the threads don't interact. Let a framework take care of this for you.
Example:
from("file://incoming?maxMessagesPerPoll=1&idempotent=true&moveFailed=failed&move=processed&readLock=none")
.threads(10).process()

Checking if a java Thread has been join()ed

I would like to check whether a certain thread has already been joined.
In the code below I have threads that finish at different times and I would like to check whether a thread has terminated and not yet been joined.
Is there a good way of checking this?
while(!allJoined){
allJoined=true;
for ( int i = 0; i < 10; i++ )
{
try {
if(!threadList[i].isAlive() && threadList[i].NOT_YET_JOINED() ) {
threadList[i].join(0);
System.out.println("Joined t-"+i);
} else {
allJoined = false;
}
} catch (Exception e) {
System.out.println("MASTER: Child interrupted."+e);
}
}
try {
Thread.sleep(100);
} catch (InterruptedException e) {
System.out.println("Child interrupted.");
}
}

Whether you have join()ed a thread is a part of your state, not the thread's state. Imagine: you could have several threads all trying to join a given worker-thread.
Why don't you keep a list of the threads you have already joined?
Then check that list before you join it.
Vector<Thread> joined = new Vector<Thread>();
for(int i=0;i<threadList.length;i++) {
if(threadList[i].isAlive() && !joined.contains(threadList[i])){
threadList[i].join(0);
joined.add(threadList[i]);
}
}
then use joined.size() to see if all are joined or not.

Make your Runnable objects also extend Observable. Have your main class implement Observer and register with them when you start them.
Before the threads return from run() method have them notify. This way you just join when they're ready and you know which ones you've joined. (You of course keep track of which are still running and which you've joined)

If you want to simplify this pattern, use an ExecutorService
e.g.
ExecutorService es = Executors.newFixedThreadPool(nThread);
for(int i=0;i< nTasks; i++)
es.executor(new Runnable .... );
es.shutdown();
es.awaitTermination(timeoutSeconds, TimeUnit.SECONDS);
This will create a thread pool which can be smaller than the number of tasks, and wait for it to finish efficiently.

do you need
Thread.getState()==TERMINATED

How to make the main thread wait for the other threads to complete in ThreadPoolExecutor

I am using the ThreadPoolExecutor to implement threading in my Java Application.
I have a XML which I need to parse and add each node of it to a thread to execute the completion. My implementation is like this:
parse_tp is a threadpool object created & ParseQuotesXML is the class with the run method.
try {
List children = root.getChildren();
Iterator iter = children.iterator();
//Parsing the XML
while(iter.hasNext()) {
Element child = (Element) iter.next();
ParseQuotesXML quote = new ParseQuotesXML(child, this);
parse_tp.execute(quote);
}
System.out.println("Print it after all the threads have completed");
catch(Exception ex) {
ex.printStackTrace();
}
finally {
System.out.println("Print it in the end.");
if(!parse_tp.isShutdown()) {
if(parse_tp.getActiveCount() == 0 && parse_tp.getQueue().size() == 0 ) {
parse_tp.shutdown();
} else {
try {
parse_tp.awaitTermination(30, TimeUnit.SECONDS);
} catch (InterruptedException ex) {
log.info("Exception while terminating the threadpool "+ex.getMessage());
ex.printStackTrace();
}
}
}
parse_tp.shutdown();
}
The problem is, the two print out statements are printed before the other threads exit. I want to make the main thread wait for all other threads to complete.
In normal Thread implementation I can do it using join() function but not getting a way to achieve the same in ThreadPool Executor. Also would like to ask if the code written in finally block to close the threadpool proper ?
Thanks,
Amit

A CountDownLatch is designed for this very purpose. Examples may be found here and here. When the number of threads is not known in advance, consider a Phaser, new in Java 1.7, or an UpDownLatch.

To answer your second question, I think you are doing a reasonable job trying to clean up your thread pool.
With respect to your first question, I think the method that you want to use is submit rather than execute. Rather than try to explain it all in text, here's an edited fragment from a unit test that I wrote that makes many tasks, has each of them do a fragment of the total work and then meets back at the starting point to add the results:
final AtomicInteger messagesReceived = new AtomicInteger(0);
// ThreadedListenerAdapter is the class that I'm testing
// It's not germane to the question other than as a target for a thread pool.
final ThreadedListenerAdapter<Integer> adapter =
new ThreadedListenerAdapter<Integer>(listener);
int taskCount = 10;
List<FutureTask<Integer>> taskList = new ArrayList<FutureTask<Integer>>();
for (int whichTask = 0; whichTask < taskCount; whichTask++) {
FutureTask<Integer> futureTask =
new FutureTask<Integer>(new Callable<Integer>() {
#Override
public Integer call() throws Exception {
// Does useful work that affects messagesSent
return messagesSent;
}
});
taskList.add(futureTask);
}
for (FutureTask<Integer> task : taskList) {
LocalExecutorService.getExecutorService().submit(task);
}
for (FutureTask<Integer> task : taskList) {
int result = 0;
try {
result = task.get();
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
} catch (ExecutionException ex) {
throw new RuntimeException("ExecutionException in task " + task, ex);
}
assertEquals(maxMessages, result);
}
int messagesSent = taskCount * maxMessages;
assertEquals(messagesSent, messagesReceived.intValue());
I think this fragment is similar to what you're trying to do. The key components were the submit and get methods.

First of all you can use ThreadPoolExecutor.submit() method, which returns Future instance, then after you submitted all your work items you can iterate trough those futures and call Future.get() on each of them.
Alternatively, you can prepare your runnable work items and submit them all at once using ThreadPoolExecutor.invokeAll(), which will wait until all work items completed and then you can get the execution results or exception calling the same Future.get() method.