We Keep Coding

How to create multiple threads inside of a for loop

Hi I am somewhat new to Java.
I have a method that takes in a map, and for each key value pair in the map it writes to a file.
I want to have a thread per key,value pair in the map running so that I can create multiple files at the same time. Not sure what is the proper way of doing this or how to use executor service to do get this done.
Here is a very simple example of what I'm trying to. Instead of writing all the code for writing the file Im just using system.out.println in the example:
public class CityWriter
{
public static void main(String []args)
{
LinkedHashMap<Integer, ArrayList<City>> stateNumCitiesMap = new LinkedHashMap<Integer, ArrayList<City>>();
stateNumCitiesMap = retrieveStateCitiesMap();
int numOfThreadsToExecuteAtATime = 10;
ExecutorService executor = Executors.newFixedThreadPool(numOfThreadsToExecuteAtATime);
for(Integer key : stateNumCitiesMap.keySet()) //Could have up to 50 key,values in map
{
executor.execute(writeCitiesOfStateToFile(key, StateNumCitiesMap.get(key)));
}
executor.shutdown();
}
public LinkedHashMap<Integer, ArrayList<Cities>> writeCitiesOfStateToFile(int stateNum, List<City> citiesList)
{
for(City city : citiesList)
{
System.out.println(stateNum +" "+ city);
}
}
}//end of class
My problem is that it doesn't seem like it is executing threads in parallel here. Also I don't want to run more than 10 threads at a time even though the for loop will call the executor 50 times.
please let me know what would be the most efficient way to do this.

Actually, if i understood your question well, your code does exactly what you want(of course if we omit all syntax errors in your code snippet):
It does not spawn more than 10 threads, because you have specified here Executors.newFixedThreadPool(10) how many threads you want
All your x map entries will be assigned to executor as potential job. Then executor will run each of them in parallel with all 10 threads(but no more than 10 jobs at once)
You can try this snippet out and check that several threads are doing the job in parallel:
public static void main(String[] args) {
Map<Integer, List<String>> stateNumCitiesMap = new LinkedHashMap<>();
for (int i = 0; i < 100; i++) {
stateNumCitiesMap.put(i, Collections.singletonList("ABC"));
}
ExecutorService executor = Executors.newFixedThreadPool(10);
for (Integer key : stateNumCitiesMap.keySet()) {
executor.execute(() -> writeCitiesOfStateToFile(key, stateNumCitiesMap.get(key)));
}
executor.shutdown();
}
public static void writeCitiesOfStateToFile(int stateNum, List<String> citiesList) {
for (String city : citiesList) {
System.out.println(stateNum + " " + Thread.currentThread().getName());
}
}
In case you don't want to give a job one by one to the executor, you can pass a batch of them once.
public static void main(String[] args) throws InterruptedException {
Map<Integer, List<String>> stateNumCitiesMap = new LinkedHashMap<>();
for (int i = 0; i < 100; i++) {
stateNumCitiesMap.put(i, Collections.singletonList("ABC"));
}
ExecutorService executor = Executors.newFixedThreadPool(10);
List<Callable<Void>> jobs = new ArrayList<>();
for (Integer key : stateNumCitiesMap.keySet()) {
jobs.add(() -> {
writeCitiesOfStateToFile(key, stateNumCitiesMap.get(key));
return null;
});
}
executor.invokeAll(jobs);
executor.shutdown();
}
public static void writeCitiesOfStateToFile(int stateNum, List<String> citiesList) {
for (String city : citiesList) {
System.out.println(stateNum + " " + Thread.currentThread().getName());
}
}

You can use "invokeAll" method for multiple executions and even get their results (as done or not). It will use 10 thread for them even they are 50. The results will be returned when all tasks are completed. Something like below, take it as pseudo.
Callable<int> callableTask = (fileName) -> {
// implement write to the file
return 0;
};
ExecutorService executor = Executors.newFixedThreadPool(10);
List<Callable<int>> tasksList;
for(City city : citiesList)
{
tasksList.add(callableTask(city.toString()));
}
executor.invokeAll(tasksList);

In java you need to provide the runnable interface to any object you wish to run in a thread , you are not doing this and this is what the executor is expecting.
executor.execute(() -> your function )
is actually
executor.execute(new Runnable() {
#Override
public void run() {
// your code
}
});
the method does not implement runnables , only in the run method of runnable will it be threaded
the reason is that the executor uses a kind of observer pattern and you subscribe the runnable to it , the executor then runs the run method
from java docs :
The Runnable interface should be implemented by any class whose instances are intended to be executed by a thread. The class must define a method of no arguments called run.
This interface is designed to provide a common protocol for objects that wish to execute code while they are active. For example, Runnable is implemented by class Thread. Being active simply means that a thread has been started and has not yet been stopped.
Can also make the method return a runnable itself
public static Runnable writeCitiesOfStateToFile(params) {
return () -> System.out.println(params);
}

Executor#execute may be synchronous
You said:
it doesn't seem like it is executing threads in parallel here
You did not explain the reason for that perception.
But, FYI, that may indeed be the case. You called the execute method on your ExecutorService.
for(Integer key : stateNumCitiesMap.keySet()) //Could have up to 50 key,values in map
{
executor.execute(writeCitiesOfStateToFile(key, StateNumCitiesMap.get(key)));
}
That execute method is inherited from the Executor interface, the super-interface of ExecutorService. That interface, and its method, are documented as maybe running your task asynchronously. To quote the Javadoc:
The command may execute in a new thread, in a pooled thread, or in the calling thread, at the discretion of the Executor implementation.
So you may indeed be seeing sequential non-thread synchronous execution rather than async.
From my reading of the ExecutorService methods submit, invokeAll, & invokeAny, these seem to be promising to always run asynchronously.
I do not believe such synchronous behavior is happening though, given your choice of ExecutorService implementation. Your call to Executors.newFixedThreadPool produces an object of type ThreadPoolExecutor. Looking briefly at the source code of that concrete class’ execute method, it appears to always work asynchronously (though I am not entirely sure).
Nevertheless, it would seem that we should not always assume async execution when using Executor#execute.

Run 3 threads and wait in Java

I'm trying to write a class that can only run X(Let's say 3)threads at one time. I have 8 threads that need to execute but I only want to allow 3 to run at once, then wait. Once one of the currently running threads stops, then it will start another. I'm not quite sure how to do this. My code looks like this:
public class Main {
public void start() {
for(int i=0; i<=ALLTHREADS; i++) {
MyThreadClass thread = new MyThreadClass(someParam, someParam);
thread.run();
// Continue until we have 3 running threads, wait until a new spot opens up. This is what I'm having problems with
}
}
}
public class MyThreadClass implements Runnable {
public MyThreadClass(int param1, int param2) {
// Some logic unimportant to this post
}
public void run() {
// Generic code here, the point of this is to download a large file
}
}
As you can see above most of it is stubbed out pseudo-code. I can post it if anyone would like but it's unimportant to the main question.

you should use thread pooling mechanism here to run multiple threads.
to make it easy we can go for thread pool executor in java which is very easy
create a fixed pool of 3 threads using executors method.
write a for loop for 8 iteration and call execute on each thread and it will run only 3 threads at a time.
ExecutorService executor = Executors.newFixedThreadPool(3);
for (int i = 0; i < 8; i++) {
Task task = new Task(someParam, someParam);
executor.execute(task);
}
executor.shutdown();

Unless this is homework, you can use Executors.newFixedThreadPool(3) which returns an ExecutorService with a max of 3 threads to perform Runnable tasks.

Java threadpools and runnables creating runnables

Bear with me as I'm not terribly savvy in multithreaded programming...
I'm currently building out a system that uses a ThreadPool ExecutorService for various runnables. That much is straightforward. However, I'm looking at the possibility of having the runnables themselves spawn an additional runnable based on what happens in the original runnable (ie, if success, do this, if fail, do this, etc as some tasks must be complete before others execute). It should be noted that the main thread does not need to be notified of the results of these tasks, although it might be handy for handling exceptions, ie, if an external service cannot be contacted and all threads are throwing exceptions as a result, then stop submitting tasks and periodically check on the external service until it comes back up. This isn't completely necessary, but it would be nice.
Ie, submit Task A. Task A does some things. If everything goes well, Task A will execute Task B. If something doesn't work out properly or an exception is thrown, execute Task C. Each child task may also have additional tasks, but only a few levels deep. I'd much rather do something like this than large, snarled conditionals in a single task as this approach allows for much greater flexibility.
However, I'm not certain how this would affect the thread pool. I would assume that any additional thread(s) created from within a thread in the pool would exist outside of the pool as they themselves were not submitted directly to the pool. Is this a correct assumption? If so, it's likely a bad idea (well, if not, it may not be a very good idea anyway) as it could result in a lot more threads as the original thread completes and a new task is submitted while the thread spawned from the earlier task is still going (and may last considerably longer than others).
I've also considered implementing these as Callables instead and placing a response object in the Future that is returned, then add the appropriate Callable to the thread pool based on the response. However, this would tie all actions back to the main thread, which seems an unnecessary bottleneck. I suppose I could place a Runnable into the pool that itself handles the execution of the Callable and subsequent actions, but then I get twice as many threads.
Am I on the right track here or am I completely off the rails?

I have never used this, but it can be useful for you: http://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html

There are many ways to do what you want. You need to be careful you don't end up creating too many threads.
The following is an example, you could make this more efficient with an ExecutorCompletionService and alternatively you could use Runnable's.
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class ThreadsMakeThreads {
public static void main(String[] args) {
new ThreadsMakeThreads().start();
}
public void start() {
//Create resources
ExecutorService threadPool = Executors.newCachedThreadPool();
Random random = new Random(System.currentTimeMillis());
int numberOfThreads = 5;
//Prepare threads
ArrayList<Leader> leaders = new ArrayList<Leader>();
for(int i=0; i < numberOfThreads; i++) {
leaders.add(new Leader(threadPool, random));
}
//Get the results
try {
List<Future<Integer>> results = threadPool.invokeAll(leaders);
for(Future<Integer> result : results) {
System.out.println("Result is " + result.get());
}
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
threadPool.shutdown();
}
class Leader implements Callable<Integer> {
private ExecutorService threadPool;
private Random random;
public Leader(ExecutorService threadPool, Random random) {
this.threadPool = threadPool;
this.random = random;
}
#Override
public Integer call() throws Exception {
int numberOfWorkers = random.nextInt(10);
ArrayList<Worker> workers = new ArrayList<Worker>();
for(int i=0; i < numberOfWorkers; i++) {
workers.add(new Worker(random));
}
List<Future<Integer>> tasks = threadPool.invokeAll(workers);
int result = 0;
for(Future<Integer> task : tasks) {
result += task.get();
}
return result;
}
}
class Worker implements Callable<Integer> {
private Random random;
public Worker(Random random) {
this.random = random;
}
#Override
public Integer call() throws Exception {
return random.nextInt(100);
}
}
}

Submitting tasks to the thread pool from other tasks is quite meaningful idea. But I am afraid you think of running new tasks on separate threads, that really can eat all the memory. Just set a limit to the number of threads when the pool is created, and submit new tasks to that thread pool.
This approach can be further elaborated in different directions. First, treat tasks as ordinary objects, with interface methods, and let that methods decide if they want to submit this object to the thread pool. This requires that each task knows its thread pool - pass it as a parameter at the time of creation. Even more convenient, keep reference to the thread pool as a thread local variable.
You can easily emulate functional programming: an object represents a function call, and for each parameter it has corresponding set method. When all parameters are set, the object is submitted to the thread pool.
Another direction is actor programming: task class has single set method, but it can be called multiple times, and if previous argument is not yet processed, the set method does not submit the task to the thread pool, but simply stores its argument in a queue. The run() method processes all available arguments from the queue and then returns.
All this features are implemented in the dataflow library https://github.com/rfqu/df4j. I wrote it intentionally to support task-based parallelism.

Using threads in Java

I have a portion of code dealing with threads and I want to understand its function in detail. The run method is empty in my example, but lets assume it has some operations to do on a global variable:
import java.io.File;
public class DigestThread extends Thread {
private File input;
public DigestThread(File input) {
this.input = input;
}
public void run() {
}
public static void main(String[] args) {
for (int i = 0; i < args.length; i++) {
File f = new File(args[i]);
Thread t = new DigestThread(f);
t.start();
}
}
}
After creating a thread and starting it, will it wait to finish the tasks in the run method before creating/running another thread ?
second question
if a variable has declared in run method that means it will be declared many times because every thread created will do the task in run method , is every thread handles its own varible although variable in each thread are same ?

will it waitng for finish the tasks in run method to creat another
method ?
No. That's the whole point of Multithreading. Every thread has it's own stack of execution. So, when you start a thread, it enters the run() method, and executes it in a different stack, and at the same time the main thread continues execution, in it's own stack.
After that, main thread, can spawn another thread. Thus all the threads run simultaneously, but of course, one at a time, sharing the CPU amongst them, based on the specific CPU allocation algorithm being used.
It's not possible to write down all the details about the execution process of multiple threads here. I would rather suggest to read some tutorials or search for some online resources, regarding the concept of Multithreading. Once clear in concept, move ahead with the implementation in Java.
Here's some tutorials links you can start with: -
Thread Wiki Page
SO Multithreading Wiki Page
Concurrency - Oracle Tutorial
http://www.vogella.com/articles/JavaConcurrency/article.html

Once you start a thread, it will run in a separate thread and the main() thread will continue (it may end before the child thread ends).
If you want the thread to finish before main then you can use the Thread.join method on thread to wait for it.

First, Thread is a single sequential flow of control within a program.
We can execute more than one thread in a program, but there is a life cycle, where you can find out how threads work...
Whenever we call the start() method of Thread it automatically calls the overriden method public void run() and executes the code of the run() method...
Here is a simple example of Thread in Java with ABC and XYZ thread classes
/* Write a program to print having two Thread Class 1) ABC with 1 second 2) XYZ 2 second. 10 times with Extend Constructor */
class ABCThreadConstructor extends Thread {
ABCThreadConstructor(String name) {
super(name);
}
public void run() {
try {
for(int i = 0; i < 10; i++) {
System.out.println("ABC");
Thread.sleep(1000);
}
} catch(InterruptedException ie) {
System.out.println("Interrupted Exception : "+ie);
}
}
}
class XYZThreadConstructor extends Thread {
XYZThreadConstructor(String name) {
super(name);
}
public void run() {
try{
for(int i = 0; i < 10; i++) {
System.out.println("XYZ");
Thread.sleep(2000);
}
} catch(InterruptedException ie) {
System.out.println("Interrupted Exception : "+ie);
}
}
}
class AbcXyzThreadConstructorDemo {
public static void main(String args[]) {
ABCThreadConstructor atc = new ABCThreadConstructor("ABCThreadConstructor");
System.out.println("Thread Name : " + atc.getName());
atc.start();
XYZThreadConstructor xtc = new XYZThreadConstructor("XYZThreadConstructor");
System.out.println("Thread Name : " + xtc.getName());
xtc.start();
}
}

Assuming your question is "After creating a thread and starting it, will the program wait for the thread to finish its run method before creating another thread?"
If that's the case, No the program will not wait. t.start() kicks off the thread and it gets its own chunk of memory and thread priority to run. It will do its operations and then exist accordingly. Your main thread will start the number of threads specified in args before terminating.
If you need the application to wait on the thread then use t.join(). That way the parent thread (the one that runs main) will be joined to the child thread and block until its operation is complete. In this case it sort of defeats the purpose of threading but you can store the thread ID for whatever logic you need and join() later.

How to stop the main process while the spawned threads finish execution

Here's my code
public class Main {
private static class GetData implements Runnable{
private List list;
private SqlQuery query;
GetData(SqlQuery<String> param){
this.query=param;
}
public void run(){
list = query.execute();
}
}
public static void main(String[] args){
ApplicationContext context = new ClassPathXmlApplicationContext("database.xml");
SqlQuery<String> parameter = (SqlQuery<String>) context.getBean("BEAN_NAME");
System.out.println("hello");
new Thread(new Inner(parameter)).start();
for(each element in list of inner class){
System.out.println(element.id);
}
}
}
Well my question is after i get the query from xml file, it executes but it doesnt print anything? Why?
Also, how do i ensure that after all my threads have finished execution, only then my main program moves ahead in execution, given i make another thread and run it to create another list.

Change
new Thread(new Inner(parameter)).start();
to
Thread t = new Thread(new Inner(parameter));
t.start();
and put t.join(); after your for loop.
EDIT:
For 5 or any number of threads say n
Create an array of threads like this
Thread[] tArray = new Thread[n];
for (int j = 0; j < tArray .length; j++) {
//your code to start the thread goes here
}
Once you have started them all, loop through them again at the end of the main function to join each of them to main thread.
for (int j = 0; j < tarray .length; j++) {
tArray.join()
}

If you are using one Thread and want main program waits its execution completed. You do not need to use Thread mechanism. Instead, you can add a method in main program in substance to Thread.run().
Otherwise if you want to use multiple thread you can use Thread.join method so that all other threads wait at that line until all thread execution are completed.
I also advice you to investifate countdownlatch mechanism. It can give you ready mechanism not to involve in Join/wait operations manually.

Use the join method of the thread you want to wait. Thread.join() javadoc
how it works :
The thread (lets call him A) that join another thread (called B) will stop it execution until the joined thread (B) finish and returns.
EDIT :
In fact, unless your Thread is in Daemon mode, your program won't exit.
The JVM automatically joins all running non daemon thread before exiting

new Thread(runnable).start(); executes the provided runnable asynchronously and continues the execution to the next line, so the loop executes before anything has been added to the list.
So if you want to execute the loop after the thread has finished, you will have to wait for it. The easiest way is to run the Runnable: new Inner(parameter).run();.
Now that defeats the purpose of parallel execution.
Assuming you have more than one runnable, you could use an ExecutorService (instead of using the low-level Thread API, which is more complicated to use and error-prone) to run the various tasks in parallel and collect the results when they are all completed:
ExecutorService executor = Executors.newCachedThreadPool();
executor.submit(runnable1); //first task
executor.submit(runnable2); //second task
executor.shutdown(); //stop accepting new tasks
executor.awaiTermination(Integer.MAX_VALUE, TimeUnit.SECONDS); //wait until both tasks finish
//now you can use the results of your tasks.
Finally, note that if you use threads, you will be sharing the list in your runnable across threads (the worker thread and the main thread) and you will need to use a thread safe structure to make that possible - for example, by using a CopyOnWriteArrayList:
list = new CopyOnWriteArrayList(query.execute());