We Keep Coding

Best approach to manage parallel threads with a set of ip addresses using ExecutorService

I have a set of 255 ip addresses to manage (x.x.x.1 -> x.x.x.255).
In Java if I check connection from my java mobile app with only one array of IP with a setTimeout(200) I could wait too much till finish all 255 ip addresses. On the other hand if I connect to at least one of those ip address I have some other things to do.
So my goal to reduce wait time on check if connection test works or fails is to split in a group of 15 parallel threads working at the same time where inside each of them I check 17 ip addresses.
In this way I made a class that implements Runnable where I execute something like:
HttpURLConnection con;
for(i=(currentThreadNumber*17)+1;i<(currentThreadNumber*17)+17;i++) {
String ipToCheckConnection = maskIP+"."+i;
String loginURL = "http://" + ipToLogin + "/....";
try {
URL obj = new URL(loginURL);
con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setConnectTimeout(100);
con.connect();
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
con.disconnect();
do something else with this IP....
}
} catch (java.net.SocketTimeoutException e) {
continue;
} catch (java.io.IOException e) {
return;
}
}
In my main function inside a button click event I implement:
ExecutorService executorService = Executors.newFixedThreadPool(15);
then I tried various way to execute parallel jobs calling the runnable class waiting all threads are finished before to continue and exit from button click event, like using CountDownLatch , .awaitTermination... I tried also to use .invokeAll but it doesn't work with runnable but with callable... but I encounter the issue how to pass to runnable class current thread count like 0,1,2,3,..so I can pass to the for(...) inside runnable...
What would be best approach to pass this current thread count like 0,1,2,3,.. to the runnable class?
Or is there a better different way than using ExecutorService which I read everywhere it's the simplest way to work parallel threads with?...
Thanks!
Cheers

You can do something as follows:
initialize the CountDownLatch and ExecutorService with the number of desired workers
on the loop of assigning work to the workers pass the current task number
each worker should also call countDownLatch.countDown(); after having terminated the work to signal that to the other workers that might be waiting
the main thread should call countDownLatch.await(); to wait for the remaining workers to finish their work.
The code could look like the following:
public class WaitForAllToEnd {
public static void main(String[] args) throws InterruptedException {
final int total_threads = 15;
CountDownLatch countDownLatch = new CountDownLatch(total_threads);
ExecutorService executor = Executors.newFixedThreadPool(total_threads);
for(int i = 0; i < total_threads; i++){
final int thread_id = i;
executor.execute(parallelWork(thread_id, countDownLatch));
}
countDownLatch.await();
System.out.println("Exit");
executor.shutdown();
}
private static Runnable parallelWork(int thread_id, CountDownLatch countDownLatch) {
return () -> {
try {
// Some code logic
} catch (InterruptedException e) {
// Do Something
}
// Some code logic
countDownLatch.countDown();
};
}
}

How to cancel Libreoffice UNO API Library process of convertion of ms office files to PDF in case it is over maximum permitted time

I am using UNO API Library (Soffice) from Libreoffice 6.0 to convert ms office formats to PDF, the Soffice process serves multiple sumultanious requests in server mode.
Usually the convertion is fast, but while converting some large files, e.g. xlsx or pptx, the Soffice process uses 100% CPU and convertion takes up to a few minutes.
This is unacceptable, because during this time concurrent requests are not treated.
To handle this situation I tried to use java.util.concurrent to execute some subtasks as threads with timeout control via future interface. But it works good only if hanging occured on original ms office document load stage of convertion.
If process of convertion has already started, even though timeout exception occures, Soffice process does not quit 100% load at once, but contimue to convert document to pdf.
Program execution pauses trying to dispose loaded document.
SOffice process is started under linux via command:
Runtime.getRuntime().exec("/usr/lib64/libreoffice/program/soffice, --nologo, --nodefault, --norestore, --nocrashreport, --nolockcheck, --accept=socket,host=localhost,port=8100;urp;");
Code for convertion ms office file to pdf in simplified form is:
public void convertFile(){
xRemoteContext = BootstrapSocketConnector.bootstrap(oooPath);
xRemoteServiceManager = xRemoteContext.getServiceManager();
Object desktop = null;
desktop = xRemoteServiceManager.createInstanceWithContext("com.sun.star.frame.Desktop", xRemoteContext);
xComponentLoader = (XComponentLoader) UnoRuntime.queryInterface(XComponentLoader.class, desktop);
File mfile = new File(workingDir + myTemplate);
String sUrl = pathToURL(workingDir + myTemplate);
PropertyValue[] propertiesIn;
propertiesIn = new PropertyValue[2];
propertiesIn[0] = property("ReadOnly", Boolean.TRUE);
propertiesIn[1] = property("Hidden", Boolean.TRUE);
XComponent xComp = null;
try {
//xComp = xComponentLoader.loadComponentFromURL(sUrl, "_blank", 0, propertiesIn);
//The same via timeout control
xComp = callLibreLoad(sUrl, propertiesIn);
}
catch (TimeoutException ex) {
if(xComp!= null)
xComp.dispose();
...
}
// save as a PDF
XStorable xStorable = (XStorable) UnoRuntime.queryInterface(XStorable.class, xComp);
PropertyValue[] propertiesOut = new PropertyValue[2];
propertiesOut[0] = property("FilterName", formatfilter);
propertiesOut[1] = property("Overwrite", Boolean.TRUE);
String myResult = workingDir + fileNameOut;
try {
//xStorable.storeToURL(pathToURL(myResult), propertiesOut);
//The same via timeout control
callLibreStore(xStorable,pathToURL(myResult), propertiesOut);
}
catch (TimeoutException ex) {
if(xComp!= null)
xComp.dispose();
...
}
if(xComp!= null)
xComp.dispose();
}
Functions callLibreLoad and callLibreStore use Future interface for timeout control:
private XComponent callLibreLoad(String sUrl, PropertyValue[] propertiesIn) throws Exception {
XComponent result = null;
ExecutorService executor = Executors.newCachedThreadPool();
Callable<Object> task = new Callable<Object>() {
public Object call() throws IllegalArgumentException, com.sun.star.io.IOException {
return xComponentLoader.loadComponentFromURL(sUrl, "_blank", 0, propertiesIn);
}
};
Future<Object> future = executor.submit(task);
try {
result = (XComponent) future.get(maxTimeout, TimeUnit.SECONDS);
}
finally
{ future.cancel(true);}
return result;
}
private void callLibreStore(XStorable xStorable, String sUrl, PropertyValue[] propertiesOut) throws Exception {
Integer result = null;
ExecutorService executor = Executors.newCachedThreadPool();
Runnable task = new Runnable() {
public void run() {
try {
xStorable.storeToURL(sUrl, propertiesOut);
} catch (com.sun.star.io.IOException e) {
log.error(e);
}
}
};
Future future = executor.submit(task);
try {
future.get(maxTimeout, TimeUnit.SECONDS);
}
finally {
future.cancel(true); // may or may not desire this
}
}
So, when timeout exception take place in function callLibreLoad, SOffice process is restored to working state at once.
But when timeout take place later, in function callLibreStore, even after timeout happens and convertion thread was intrerrupted, SOffice process stays in 100% load state for more than a minute trying to dispose loaded document, executing code xComp.dispose().
At this period the stack trace of java thread with SOffice process contains following:
State: WAITING on com.sun.star.lib.uno.environments.remote.JobQueue#30af74b8
Total blocked: 455 Total waited: 1 967
Stack trace:
java.lang.Object.wait(Native Method)
com.sun.star.lib.uno.environments.remote.JobQueue.removeJob(JobQueue.java:207)
com.sun.star.lib.uno.environments.remote.JobQueue.enter(JobQueue.java:316)
com.sun.star.lib.uno.environments.remote.JobQueue.enter(JobQueue.java:289)
com.sun.star.lib.uno.environments.remote.JavaThreadPool.enter(JavaThreadPool.java:81)
com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge.sendRequest(java_remote_bridge.java:618)
com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.request(ProxyFactory.java:145)
com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.invoke(ProxyFactory.java:129)
com.sun.proxy.$Proxy211.close(Unknown Source)
com.componentplus.prom.libreoffice.LibreOfficeStationary.closeDocument(LibreOfficeStationary.java:425)
com.componentplus.prom.libreoffice.LibreOfficeStationary.convertFile(LibreOfficeStationary.java:393)
...
How is it possible to force Soffice to cancel convertion to pdf in case it takes more than maximum permited time.

One possibility might be to save the Process instance returned by Runtime.getRuntime().exec, e.g. in a variable myProc, and then call myProc.destroy() to kill the process when needed.

Idiomatic way to feed multithreaded output to a single thread in modern Java?

I have this code which works fine. This question is focused on code maintainability, writing less code to accomplish the same goal:
Queue<IncomingItem[]> queue = new ConcurrentLinkedQueue<>();
IncomingItem[] EOF = new IncomingItem[0];
ForkJoinPool.commonPool().submit(() -> {
IncomingItem[] next;
while((next = queue.poll()) != EOF) {
if(next == null) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
continue;
}
dao.batchInsert(next);
}
});
ds.reload(queue::add);
queue.add(EOF);
ForkJoinPool.commonPool().awaitQuiescence(1, TimeUnit.HOURS);
The purpose is to have the output from ds.reload, which is a heavily multi-threaded method, feed into the dao.batchInsert method, which is non-threadsafe non-thread-shareable (e.g. it is a Hibernate based DAO), while never blocking the ds.reload method, as would happen if the dao.batchInsert method were made synchronized.
This code was Java 8 compatible. Is there anything that has come out in newer Java releases which would enable a more elegant solution?

Using thread pool here is a needless complication. Use dedicated thread. Instead of polling and sleeping, use blocking queue.
BlockingQueue<IncomingItem[]> queue = new LinkedBlockingQueue<>();
IncomingItem[] EOF = new IncomingItem[0];
Thread thread = new Thread(() -> {
IncomingItem[] next;
while((next = queue.take()) != EOF) {
dao.batchInsert(next);
}
});
thread.start();
ds.reload(queue::put);
queue.add(EOF);
thread.join();

Thread safety with ExecutorService and CountDownLatch

I have cycle, where i download image, I need to load for example 10 images and merge them in one image. In my interest what images will all loaded. This is how i do that.
I have executor for limit thread count, and i have CountDownLatch barrier which waiting until all images will be loaded.
CountDownLatch barrier = new CountDownLatch(images.size());
private static ExecutorService executorService = Executors.newFixedThreadPool(MAX_THREAD_POOL);
for (Image image : images) {
executorService.execute(new ImageRunnable(image, barrier));
}
barrier.await();
In ImageRunnable i download image like this. From google static map.
String url ="my url"
try {
URL target = new URL(url);
ImageIO.read(target);
barrier.countDown();
//exaggerated logic
} catch (IOException e) {
System.out.println("Can not load image, " + e);
}
Other people said to me that i can get case when all threads in executor will be busy and my algorithm never ends because he will wait until all threads get barrier.await() point (deadlock). How said to me it's will happen when ImageIO.read(target) called and connection will established but HTTP session never be closed (response from server do not come back). This can happen? I thought in this case i get some exception and bad thread will interrupted. Exactly that happens when I start my cycle but on third image i close internet connection by firewall. On output I get broken image like network was closed and image not loaded to end. Am I wrong?

The concern is you may throw an exception and never count down your latch.
I would consider doing this:
String url ="my url"
try {
URL target = new URL(url);
ImageIO.read(target);
} catch (IOException e) {
System.out.println("Can not load image, " + e);
throw e;
} finally {
barrier.countDown();
}
Throw the exception to let the world know you've run into a problem and may not be able to complete (you know you can't recover from it) but at the very least let the barrier get lowered. I'd rather have to deal with an exception than a deadlock.

Just to flesh out my comment:
CompletionService<Image> service = new ExecutorCompletionService<Image>(
Executors.newFixedThreadPool(nThreads));
for (Image image : images) {
service.submit(new ImageRunnable(image), image);
}
try {
for (int i = 0; i < images.size(); i++) {
service.take();
}
} catch (InterruptedException e) {
// someone wants this thread to cancel peacefully; either exit the thread
// or at a bare minimum do this to pass the interruption up
Thread.currentThread().interrupt();
}
There. That's it.
If you're concerned about enforcing timeouts on the HTTP connection, my quick and dirty research suggests something like...
URL target = // whatever;
URLConnection connection = target.openConnection();
connection.setReadTimeout(timeoutInMilliseconds);
InputStream stream;
try {
stream = connection.getInputStream();
return ImageIO.read(stream);
} finally {
if (stream != null) { stream.close(); }
}

Apart from moving barrier.countDown() to finally block as suggested by #corsiKa, make sure your code ever finishes. Set some timeout on reading URL and on await():
barrier.await(1, TimeUnit.MINUTES);

Issue with wait - notify implementation

I am working on Java multithreading , where I am starting 4 threads after assigning 4 different files to them , to be uploaded to the server.
My objective is , when one thread completes file upload , I need to start another thread assigning a new file to it.
After each file upload , I receive a notification from the server.
// The code for adding the first set of files
for (int count = 0; count < 4; count++) {
if (it.hasNext()) {
File current = new File((String) it.next());
try {
Thread t = new Thread(this, current );
t.start();
t.sleep(100);
} catch (Exception e) {
}
}
}
Now , I am assigning another thread with a file & keeping the thread in a wait state .
When a previous thread notifies , the current thread should start upload.
if (tempThreadCounter == 4 ) {
if (it.hasNext()) {
File current = new File((String) it.next());
try {
Thread t = new Thread(this, current);
t.start();
synchronized (this) {
t.wait();
}
tempThreadCounter++;
} catch (Exception e) {
}
}
}
On the final statement on the run method , I am adding the following statement.
public void run (){
// Performing different operations
//Final statement of the run method below
synchronized (this) {
this.notifyAll();
}
}
Currently , all the 5 threads are starting uploading at the same time.
It should be that the first 4 threads should start uploading & the fifth thread should start only when it it notified by any thread that it had completed its operation.
Any suggestions on the incorrect Thread implementation.

You can use ExecutorService with newFixedThreadPool and specify a concurrency of 1. But really, then why do you need multiple threads? One thread doing all the uploads so the user interface remains responsive should be enough.
ExecutorService exec = Executors.newFixedThreadPool(1); //1 thread at a time
for (int count = 0; count < 4; count++) {
if (it.hasNext()) {
File current = new File((String) it.next());
exec.execute(new Runnable() {
#Override
public void run() {
upload(current);
}
});
}
}
exec.shutdown();
exec.awaitTermination(900, TimeUnit.SECONDS);

Throw it all away and use java.util.concurrent.Executor.

you can join on the thread instead of waiting on it
try {
t.join();
}catch(InterruptedException e){
throw new RuntimeException(e);
}
tempThreadCounter++;