I am trying to use threads to make my program run some parts in parallel and am struggling.
Goal is to process a list of links, urlList through the ImageProcessor().processLink function. I have two problems I am trying to solve:
How do I loop this so it uses N number of threads in the pool, 10 in this case? That is, I want to process 10 links at a time.
The processing function above returns a File, which I need to add to an array, fileList. When it comes to multithreading, how will I handle this?
This is what i've got so far:
ArrayList<String> urlList = new ArrayList<>(Arrays.asList(arr.split("\\r?\\n"))) ;
ArrayList<File> fileList = new ArrayList<>();
ExecutorService executor = Executors.newFixedThreadPool(10);
//process the requested files
for (int i = 0; i < urlList.size(); i++){
Future<File> value = executor.submit(new Callable<File>() {
#Override
public File call(int i) throws IOException {
return new ImageProcessor().processLink(urlList.get(i));
}
});
try {
fileList.add(value.get());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
Got it working with the following:
ArrayList<String> urlList = new ArrayList<>(Arrays.asList(arr.split("\\r?\\n"))) ;
ArrayList<File> fileList = new ArrayList<>();
ExecutorService executor = Executors.newFixedThreadPool(THREAD_SIZE);
List<Future<File>> futures = new ArrayList<>();
for (int i = 0; i < urlList.size(); i++) {
ImageProcessor proc = new ImageProcessor(urlList.get(i));
final Future<File> future = executor.submit(proc);
futures.add(future);
}
try {
executor.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
for (int i = 0; i < futures.size(); i++)
{
Future<File> result = futures.get(i);
try {
fileList.add(result.get());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executor.shutdown();
while (!executor.isTerminated()) { }
System.out.println("Finished all threads");
Related
I am starting multiple threads and the idea is to wait for only one of them(does not matter which one) to continue the main thread and ignoring the other threads. In other words the main thread starts several threads and then waits until the fastest thread joins. Any ideas how to implement such a thing?
There's several possibilities.
1. Use a CountDownLatch
The idea is to have a CountDownLatch set to 1. All tasks running on the relevant threads will end by calling countDown() on it.
The main thread will simply call await() on it, and as soon as the first task is finished, the latch will hit 0, and release the waiting main thread.
This approach works with raw Threads, as well as with an ExecutorService.
public class UseCountDownLatch {
public static void main(String[] args) {
int numberOfThreads = 5;
CountDownLatch countDownLatch = new CountDownLatch(1);
ExecutorService executorService = Executors.newFixedThreadPool(numberOfThreads);
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
executorService.submit(() -> {
try {
TimeUnit.SECONDS.sleep(print * 3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
countDownLatch.countDown();
}
});
}
executorService.shutdown();
try {
countDownLatch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
executorService.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
2. Use a CompletionService
Wrap your ExecutionService in a CompletionService, then just wait for the first result to come in, and ignore further results.
public class UseCompletionService {
public static void main(String[] args) {
int numberOfThreads = 5;
ExecutorService executorService = Executors.newFixedThreadPool(numberOfThreads);
CompletionService<Void> completionService = new ExecutorCompletionService<Void>(executorService);
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
completionService.submit(() -> {
try {
TimeUnit.SECONDS.sleep(print * 3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return null;
});
}
executorService.shutdown();
try {
completionService.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
executorService.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
3. Use CompletableFutures
The CompletableFutures API is made to make tasks composable. The trick is simply to combine the CompletableFutures for each individual task in a CompletableFuture that completes as soon as any of its components completes. Then it's just a matter of getting the result from the composed CompletableFuture.
public class UseCompletableFuture {
public static void main(String[] args) {
int numberOfThreads = 5;
CompletableFuture<?>[] tasks = new CompletableFuture<?>[numberOfThreads];
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
tasks[i] = CompletableFuture.runAsync(() -> {
try {
TimeUnit.SECONDS.sleep(print*3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
}
try {
CompletableFuture.anyOf(tasks).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
CompletableFuture.allOf(tasks).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
here is code
private static class NgrokRunner implements Runnable {
private InputStream inputStream;
private boolean doStop = false;
public NgrokRunner(InputStream inputStream) {
this.inputStream = inputStream;
}
#Override
public void run() {
// TODO Auto-generated method stub
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
try {
while((line = reader.readLine()) != null) {
System.out.println(line);
if (keepRunning()) {
continue;
} else {
System.out.println("break ----");
break;
}
}
} catch (Exception e) {
//TODO: handle exception
System.out.println("Ngrok exception");
}
}
public synchronized void doStop() {
this.doStop = true;
}
private synchronized boolean keepRunning() {
return this.doStop == false;
}
}
and i started above thread like this
ProcessBuilder processBuilder = new ProcessBuilder();
processBuilder.command("ngrok", "http","8080", "--log=stdout");
try {
Process process = processBuilder.start();
NgrokRunner runner = new NgrokRunner(process.getInputStream());
Thread ngrokThread = new Thread(runner);
ngrokThread.start();
for (int i = 0; i < 4; i++) {
System.out.println(i);
Thread.sleep(10L * 100L);
}
//System.out.println("It works");
runner.doStop();
} catch (Exception e) {
//TODO: handle exception System.out.println(e);
}
But in while loop my child thread which is reading input from ngrok , blocking and even after calling doStop() it never reached at if condition where i am checking bool flag to exit from thread.
Well can anyone suggest me logic to achieve my ideal situation.
what i want is "Run ngrok server through binary file of ngrok in a thread and close the thread/ngrok whenever i want ( like when user wants through a pause/end button )"
ok so i solved it and here is final solution
run() code :
#Override
public void run() {
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
try {
while(!Thread.interrupted()) {
if (!reader.ready()) {
try {
Thread.sleep(1000);
continue;
} catch (InterruptedException e) {
//TODO: handle exception
System.out.println("We got interrupted");
return;
}
}
line = reader.readLine();
System.out.println(line);
}
} catch (IOException e) {
//TODO: handle exception
System.out.println("Ngrok exception" + e);
}
}
Now from main thread
try {
process = processBuilder.start();
NgrokRunner runner = new NgrokRunner(process.getInputStream());
ngrokThread = new Thread(runner);
ngrokThread.start();
for (int i = 0; i < 8; i++) {
System.out.println(i);
Thread.sleep(1000);
}
ngrokThread.interrupt();
} catch (Exception e) {
//TODO: handle exception
System.out.println(e);
}
I am using the below code in a java application.It spawns 3 worker threads. What I need is that if one thread fails (throws an unhandled exception from run)other threads should continue execution. If I call executor.shutdown() as below it is terminating the other threads(no new tasks are accepted).One of the options that I was thinking was to have a counter on the ExecutionException catch block and call shutdown only when counter==numConsumers. Is there a better way to do this?
public void execute() {
int numConsumers = Integer.parseInt(configs.getSINK_NUMBER_OF_CONSUMERs());
Future<?> future=null;
log.info("Creating {} consumers",numConsumers);
List<String> topics = Arrays.asList(configs.getTOPIC_NAME());
ExecutorService executor = Executors.newFixedThreadPool(numConsumers);
final List<SinkConsumer> consumers = new ArrayList<>();
for (int i = 0; i < numConsumers; i++) {
consumer = new SinkConsumer(UUID.randomUUID().toString(),topics,configs);
consumers.add(consumer);
future = executor.submit(consumer);
}
try {
future.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} catch (ExecutionException e) {
executor.shutdown();
}
}
I have been given this problem in which if a file is currently getting read, no write operation can occur on it and vice versa, using wait() and notify(). I have tried to come up with a solution but after first read the program only does the write operation and gets stuck. Here's the code
public static boolean LOCK = false;
public synchronized void read() {
String path = "/path/to/file/working.txt";
while (LOCK == true) {
try {
System.out.println("reading paused..");
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try (BufferedReader bin = new BufferedReader(new FileReader(path))) {
LOCK = true;
String line = "";
System.out.println("reading now..");
while ((line = bin.readLine()) != null) {
System.out.println(line);
}
LOCK = false;
notify();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
public synchronized void write(String word) {
String path = "/path/to/file/working.txt";
while (LOCK == true) {
try {
System.out.println("writing paused..");
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try (PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(path, true)))) {
System.out.println("writing resumed..");
LOCK = true;
out.println(word);
LOCK = false;
notify();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
I passed an String array of fruits as test, lauching read() and write() as seperate threads and the output I'm getting is,
Writing resumed..
reading..
Apple
Writing resumed..
Writing resumed..
Writing resumed..
The output gets written completly but no read operation occurs after the first word. Please can you tell me what I'm doing wrong? Thank you.
Here's the test code,
String[] fruits = { "Apple", "Banana", "Orange", "Cherry", "Date", "ElderBerry", "Marionberry", "Blueberry", };
FileSyncDemo fileDemo = new FileSyncDemo();
Runnable r = () -> {
try {
fileDemo.read();
Thread.sleep((long) (Math.random() * 1000));
} catch (InterruptedException e) {
e.printStackTrace();
}
};
Runnable r2 = () -> {
try {
for (int i = 0; i < fruits.length; i++) {
fileDemo.write(fruits[i]);
Thread.sleep((long) (Math.random() * 1000));
}
} catch (InterruptedException e) {
e.printStackTrace();
}
};
Thread t = new Thread(r);
t.start();
Thread t2 = new Thread(r2);
t2.start();
}
I have the following code in a command line application. Once the loop completes, my app is still running. Why is it not shutting down. From the logs, I can see that endIndex has reached. But the app is still running?
Executor exec = Executors.newFixedThreadPool(4);
for (int i = startIndex; i <= endIndex; i++) {
final String spURL = urlPart + i;
Runnable requestHandler = new Runnable() {
#Override
public void run() {
try {
getImageForURL(spURL, 0);
} catch (IOException ex) {
} catch (Exception ex) {
}
}
};
exec.execute(requestHandler);
}
Try this
ExecutorService e = Executors.newFixedThreadPool(4);
...
e.shutdown();
You need to shutdown the executors after the loop (ideally in a finally block):
exec.shutdown() or exec.shutdownNow()