I have the following code in a command line application. Once the loop completes, my app is still running. Why is it not shutting down. From the logs, I can see that endIndex has reached. But the app is still running?
Executor exec = Executors.newFixedThreadPool(4);
for (int i = startIndex; i <= endIndex; i++) {
final String spURL = urlPart + i;
Runnable requestHandler = new Runnable() {
#Override
public void run() {
try {
getImageForURL(spURL, 0);
} catch (IOException ex) {
} catch (Exception ex) {
}
}
};
exec.execute(requestHandler);
}
Try this
ExecutorService e = Executors.newFixedThreadPool(4);
...
e.shutdown();
You need to shutdown the executors after the loop (ideally in a finally block):
exec.shutdown() or exec.shutdownNow()
Related
I am starting multiple threads and the idea is to wait for only one of them(does not matter which one) to continue the main thread and ignoring the other threads. In other words the main thread starts several threads and then waits until the fastest thread joins. Any ideas how to implement such a thing?
There's several possibilities.
1. Use a CountDownLatch
The idea is to have a CountDownLatch set to 1. All tasks running on the relevant threads will end by calling countDown() on it.
The main thread will simply call await() on it, and as soon as the first task is finished, the latch will hit 0, and release the waiting main thread.
This approach works with raw Threads, as well as with an ExecutorService.
public class UseCountDownLatch {
public static void main(String[] args) {
int numberOfThreads = 5;
CountDownLatch countDownLatch = new CountDownLatch(1);
ExecutorService executorService = Executors.newFixedThreadPool(numberOfThreads);
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
executorService.submit(() -> {
try {
TimeUnit.SECONDS.sleep(print * 3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
countDownLatch.countDown();
}
});
}
executorService.shutdown();
try {
countDownLatch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
executorService.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
2. Use a CompletionService
Wrap your ExecutionService in a CompletionService, then just wait for the first result to come in, and ignore further results.
public class UseCompletionService {
public static void main(String[] args) {
int numberOfThreads = 5;
ExecutorService executorService = Executors.newFixedThreadPool(numberOfThreads);
CompletionService<Void> completionService = new ExecutorCompletionService<Void>(executorService);
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
completionService.submit(() -> {
try {
TimeUnit.SECONDS.sleep(print * 3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return null;
});
}
executorService.shutdown();
try {
completionService.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
executorService.awaitTermination(1, TimeUnit.MINUTES);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
3. Use CompletableFutures
The CompletableFutures API is made to make tasks composable. The trick is simply to combine the CompletableFutures for each individual task in a CompletableFuture that completes as soon as any of its components completes. Then it's just a matter of getting the result from the composed CompletableFuture.
public class UseCompletableFuture {
public static void main(String[] args) {
int numberOfThreads = 5;
CompletableFuture<?>[] tasks = new CompletableFuture<?>[numberOfThreads];
for (int i = 0; i < numberOfThreads; i++) {
int print = i;
tasks[i] = CompletableFuture.runAsync(() -> {
try {
TimeUnit.SECONDS.sleep(print*3);
System.out.println(print);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
});
}
try {
CompletableFuture.anyOf(tasks).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
System.out.println("Continue");
try {
CompletableFuture.allOf(tasks).get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
System.out.println("Done");
}
}
I am trying to use threads to make my program run some parts in parallel and am struggling.
Goal is to process a list of links, urlList through the ImageProcessor().processLink function. I have two problems I am trying to solve:
How do I loop this so it uses N number of threads in the pool, 10 in this case? That is, I want to process 10 links at a time.
The processing function above returns a File, which I need to add to an array, fileList. When it comes to multithreading, how will I handle this?
This is what i've got so far:
ArrayList<String> urlList = new ArrayList<>(Arrays.asList(arr.split("\\r?\\n"))) ;
ArrayList<File> fileList = new ArrayList<>();
ExecutorService executor = Executors.newFixedThreadPool(10);
//process the requested files
for (int i = 0; i < urlList.size(); i++){
Future<File> value = executor.submit(new Callable<File>() {
#Override
public File call(int i) throws IOException {
return new ImageProcessor().processLink(urlList.get(i));
}
});
try {
fileList.add(value.get());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
Got it working with the following:
ArrayList<String> urlList = new ArrayList<>(Arrays.asList(arr.split("\\r?\\n"))) ;
ArrayList<File> fileList = new ArrayList<>();
ExecutorService executor = Executors.newFixedThreadPool(THREAD_SIZE);
List<Future<File>> futures = new ArrayList<>();
for (int i = 0; i < urlList.size(); i++) {
ImageProcessor proc = new ImageProcessor(urlList.get(i));
final Future<File> future = executor.submit(proc);
futures.add(future);
}
try {
executor.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
for (int i = 0; i < futures.size(); i++)
{
Future<File> result = futures.get(i);
try {
fileList.add(result.get());
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executor.shutdown();
while (!executor.isTerminated()) { }
System.out.println("Finished all threads");
I have a problem with ExecutorService, it does not execute all calls.
public class MainClass {
public static void main(String[] args) throws IOException, InvalidConnection, InterruptedException {
ExecutorService executor = Executors.newCachedThreadPool();
XMLUrlService xmlUrlService = new XMLUrlService();
LinkedBlockingQueue<String> linkedBlockingQueue = xmlUrlService.getAllXMLUrls();
System.out.println(linkedBlockingQueue.size());
for (int i = 0; i < linkedBlockingQueue.size(); i++) {
executor.execute(new XMLParser(linkedBlockingQueue));
}
executor.shutdown();
}
}
XMLUrlService class returns a list of URLs to which it then performs connections.
XMLPaser is a class that is responsible for making a connection to a given URL.
public class XMLParser implements Runnable {
private LinkedBlockingQueue queue;
public XMLParser(LinkedBlockingQueue queue) {
this.queue = queue;
}
public XMLParser(){}
public void getRates(String data) throws IOException, XMLStreamException, InvalidConnection {
URL url = new URL(data);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
InputStream inputStream = connection.getInputStream();
inputStream.close();
connection.disconnect();
}
public void run() {
try {
String data = (String) queue.take();
getRates(data);
} catch (InterruptedException | IOException ex) {
//Handle exception
} catch (InvalidConnection invalidConnection) {
invalidConnection.printStackTrace();
} catch (XMLStreamException e) {
e.printStackTrace();
}
}
}
My blocking queue contains 250 different url addresses, but the executorservice performs random number of calls (250 is only random number of elemnts to be tested). I think that when I take items from the queue, I can omit some, but I'm not sure.
There are three things you should change:
The way you execute the tasks
You are executing the tasks in a very odd way. Instead of passing linkedBlockingQueue, pass the String like so:
System.out.println(linkedBlockingQueue.size());
for (int i = 0; i < linkedBlockingQueue.size(); i++) {
String url = linkedBlockingQueue.remove()
executor.execute(new XMLParser(url));
}
The way you handle exceptions
Do not swallow exceptions - the easiest way is to use a Callable instead to get Futures so that you can later check what happened with your tasks:
ExecutorService executor = Executors.newCachedThreadPool();
LinkedBlockingQueue<String> linkedBlockingQueue = new LinkedBlockingQueue<String>();
System.out.println(linkedBlockingQueue.size());
List<Future<Void>> result = linkedBlockingQueue.stream().map(
url -> executor.submit(new XMLParser(url))
).collect(Collectors.toList());
for (Future<Void> future : result) {
try {
future.get();
} catch (Exception e) {
System.out.println("Something happened:" + e);
}
}
executor.shutdown();
Waiting for execution
After you shutdown the executor, wait until all your tasks are done.
executor.shutdown();
executor.awaitTermination(10, TimeUnit.HOURS);
I am using the below code in a java application.It spawns 3 worker threads. What I need is that if one thread fails (throws an unhandled exception from run)other threads should continue execution. If I call executor.shutdown() as below it is terminating the other threads(no new tasks are accepted).One of the options that I was thinking was to have a counter on the ExecutionException catch block and call shutdown only when counter==numConsumers. Is there a better way to do this?
public void execute() {
int numConsumers = Integer.parseInt(configs.getSINK_NUMBER_OF_CONSUMERs());
Future<?> future=null;
log.info("Creating {} consumers",numConsumers);
List<String> topics = Arrays.asList(configs.getTOPIC_NAME());
ExecutorService executor = Executors.newFixedThreadPool(numConsumers);
final List<SinkConsumer> consumers = new ArrayList<>();
for (int i = 0; i < numConsumers; i++) {
consumer = new SinkConsumer(UUID.randomUUID().toString(),topics,configs);
consumers.add(consumer);
future = executor.submit(consumer);
}
try {
future.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} catch (ExecutionException e) {
executor.shutdown();
}
}
I'm pretty new to Multithreading in java but am totally stumped about why this isn't behaving as I want it to.
I have a Producer-Consumer wherein I have
private void produceConsume() {
try {
Thread producer = new Thread(new Runnable() {
public void run() {
try {
produce();
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
completedProduce = true;
}
}
private void produce() throws InterruptedException {
synchronized (this) {
while (queue.size() == capacity) {
wait();
}
try(InputStream is = new FileInputStream(file)) {
queue.add("hello");
} catch (IOException e) {
LOG.error("Error doing stream stuff: " + e.getMessage(), e);
}
notify();
}
}
});
producer.start();
List<Thread> consumers = new ArrayList<>();
for (int i = 0; i < noOfThreads; i++) {
Thread consumer = new Thread(new Runnable() {
#Override
public void run() {
try {
consume();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void consume() throws InterruptedException {
while (queue.size() > 0 || !completedProduce) {
synchronized (this) {
while (queue.size() == 0 && !completedProduce) {
wait();
}
String s = queue.poll();
System.out.println(s);
}
notify();
}
}
}
});
consumer.start();
consumers.add(consumer);
}
for (Thread t : consumers) {
t.join();
}
producer.join();
} catch (Exception e) {
LOG.error("InterruptedException e: " + e.getMessage(), e);
} finally {
LOG.info("We are done with this file!");
}
}
Now, I've noticed that all functionality changes based on where I put my producer.join() statement. For example, if I put producer.join() right after producer.start() then everything works - but the number of threads has no impact on runtime. This makes sense as I'm slowed down drastically by how long it takes to produce and so the longest task wins out.
However, if I put producer.join() where it is in the example provided (I do the join when I do the join for the consumers) then everything just stops running before the producer actually finishes. As in, the program stalls after the first thing is consumed, waiting for something, but the thread never dies.
How do I make it so that things run correctly and nothing stalls waiting for another process to finish?
Thanks in advance,