We Keep Coding

Reading container logs with 'follow' flag, thread is blocked forever

Given backend, which processes user requests of reading container logs(with follow option). Following approach is used:
Future<?> f = threadPool.submit(() -> {
try {
while (logStream.hasNext()) {
LogMessage msg = logStream.next();
String text = StandardCharsets.UTF_8.decode(msg.content()).toString();
emitter.send(SseEmitter.event().data(text).name(msg.stream().name()));
}
emitter.complete();
} catch (Exception ex) {
emitter.completeWithError(ex);
}
});
Where threadPool is just a Executors.newCachedThreadPool() and emitter is Spring's SseEmitter.
The problem is: when user no longer wants to read logs, he just closes the connection, but this thread is still running(execution is blocked in logStream.hasNext() which is calling InputStream.read(..)).
As far as I understand, hasNext() will never return false(at least while container is running properly), so this loop is endless, and we need to stop it somehow. Possible solution I tried:
emitter.onCompletion(() -> {
f.cancel(true);
});
was not successful. No InterruptedException was thrown.
Question: Is there any way of unblocking thread? Or maybe there is another approach of solving this problem (i.e. have a possibility to stop waiting for logs)?

How to read lines from a CSV to use in multiple threads

Suppose I have a CSV file with hundreds of lines with two random keywords as cells I'd like to Google search and have the first result on the page printed to the console or stored in some array. In the case of this example, I imagine I would successfully do this reading one line at a time using something like the following:
CSVReader reader = new CSVReader(new FileReader(FILE_PATH));
String [] nextLine;
while ((nextLine = reader.readNext())) !=null) {
driver.get("http://google.com/");
driver.findElement(By.name("q").click();
driver.findElement(By.name("q").clear();
driver.findElement(By.name("q").sendKeys(nextLine[0] + " " + nextLine[1]);
System.out.println(driver.findElement(By.xpath(XPATH_TO_1ST));
}
How would I go about having 5 or however many threads of chromedriver through selenium process the CSV file as fast as possible? I've been able to get 5 lines done at a time implementing Runnable on a class that does this and starting 5 threads, but I would like to know if there is a solution where as soon as one thread is complete, it processes the next available or unprocessed line, as opposed to waiting for the 5 searches to process, then going on to the next 5 lines. Would appreciate any suggested reading or tips on cracking this!

This is a pure java response, rather than specifically a selenium response.
You want to partition the data. A crude but effective partitioner can be made by reading a row from the CSV file and putting it in a Queue. Afterwards, run as many threads as you can profitably use to simply pull the next entry off of the queue and process it.

If you want to do 5 (or more) threads at the same time, you would need to start 5 instances of WebDriver as it is not thread safe. As for updating the CSV, you would need to synchronize writes to that for each thread to prevent corruption to the file itself, or you could batch up updates at some threshold and write several lines at once.
See this Can Selenium use multi threading in one browser?
Update:
How about this? It ensures the web driver is not re-used between threads.
CSVReader reader = new CSVReader(new FileReader(FILE_PATH));
// number to do at same time
int concurrencyCount = 5;
ExecutorService executorService = Executors.newFixedThreadPool(concurrencyCount);
CompletionService<Boolean> completionService = new ExecutorCompletionService<Boolean>(executorService);
String[] nextLine;
// ensure we use a distinct WebDriver instance per thread
final LinkedBlockingQueue<WebDriver> webDrivers = new LinkedBlockingQueue<WebDriver>();
for (int i=0; i<concurrencyCount; i++) {
webDrivers.offer(new ChromeDriver());
}
int count = 0;
while ((nextLine = reader.readNext()) != null) {
final String [] line = nextLine;
completionService.submit(new Callable<Boolean>() {
public Boolean call() {
try {
// take a webdriver from the queue to use
final WebDriver driver = webDrivers.take();
driver.get("http://google.com/");
driver.findElement(By.name("q")).click();
driver.findElement(By.name("q")).clear();
driver.findElement(By.name("q")).sendKeys(line[0] + " " + line[1]);
System.out.println(line[1]);
line[2] = driver.findElement(By.xpath(XPATH_TO_1ST)).getText();
// put webdriver back on the queue
webDrivers.offer(driver);
return true;
} catch (InterruptedException e) {
e.printStackTrace();
return false;
}
}
});
count++;
}
boolean errors = false;
while(count-- > 0) {
Future<Boolean> resultFuture = completionService.take();
try {
Boolean result = resultFuture.get();
} catch(Exception e) {
e.printStackTrace();
errors = true;
}
}
System.out.println("done, errors=" + errors);
for (WebDriver webDriver : webDrivers) {
webDriver.close();
}
executorService.shutdown();

You can create Callable for each row and give it to the ExecutorService. It takes care of the execution of the tasks and manages the worker threads for you. Carefully choose the thread pool size for optimal execution time.
More information about thread pool size can be found here

Monitoring Thread Execution

I have below code where first I am creating an on demand db connection and then sharing it with multiple threads. This is working fine. Now, I wanted to know if it is possible to track whether all the threads using this database connection have finished execution so that I can close the database connection. Any guidance on how I can achieve this will be helpful.
Connection connection = null;
Properties connectionProperties = getProperties();
for (int reqIdx = 0; reqIdx < requests.length(); reqIdx++) {
connection = DBonnection.getConnection(connectionProperties);
ConnectorRunner connectorRunner = null;
try {
connectorRunner = new ConnectorRunner(someConnector);
connectorRunner.setDBConnection(connection);
} catch (Exception e) {
e.printStackTrace();
}
executorService.execute(connectorRunner);
}

The easiest way is using a CountDownLatch from the standard JDK facilities. In your main thread do
CountDownLatch doneSignal = new CountDownLatch(requests.length());
for (Request req : requests) {
ConnectorRunner connectorRunner = new ConnectorRunner(doneSignal);
connectorRunner.setConnection(DBonnection.getConnection());
executorService.execute(connectorRunner);
}
doneSignal.await();
DBonnection.dispose();
The ConnectorRunner must simply call doneSignal.countDown() when it's done.

Apart from the above comments, if you are looking to do something when all your threads are finished, you can take either of the below approach.
ExecutorService es = Executors.newCachedThreadPool();
for(int i=0;i<5;i++)
es.execute(new Runnable() { /* your task */ });
es.shutdown();
boolean finshed = es.awaitTermination(1, TimeUnit.MINUTES);
// all tasks have finished or the time has been reached.
OR
for (Thread thread : threads) {
thread.join();
}
Please note that second approach will block the current thread.

Thread to process multiple rest calls

I am trying to process around 1000 files using below code:
ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
Runnable worker = null;
for (File file : files) {
if (file.isFile()) {
worker = new FileProcessThread(file, connectionVo);
executor.execute(worker);
file.deleteOnExit();
}
}
while (!executor.isTerminated()) {
System.out.println("Still running");
}
executor.shutdown();
System.out.println("Finished all threads");
This code creates multiple threads. Each thread has multiple rest calls inside.
These rest apis are for processing input file. Each thread also logs each transaction event which occurs while processing.
But result of these threads execution is not consistent.
For few threads it works perfectly fine.Picks the file.Logs correct
transactions and move processed file to proper directory.
But for for some threads it shows some unpredictable behavior such as it logs file process event of one thread into other.
Steps in each thread :
Create transaction - rest call
Log event in transaction for process start - rest call
Gives file to other module for file conversion - rest call which internally
creates one more thread which is synchronized
Once file is processed it is moved to other - in the same code directory
I want consistent performance out of these threads. Any help will be appreciated.
Code inside run :
long transactionID = 0l;
long connectionId = connectionVo.getConnectionId();
try {
transactionID = beginTransaction.getTransactionId();
FileInputStream processedFileData;
processedFileData = new FileInputStream(file);
Response response = Service.postMessage(stream2file,
connectionId, 0, transactionID);
if (response.getStatus() != 200) {
writToDirectory(stream2file, userError, file.getName(), transactionID);
}
} else {
String userArchive = getUserArchive();
if (checkDirectory(userArchive, transactionID)) {
writToDirectory(stream2file, userArchive, file.getName(), transactionID);
}
}
file.delete();
} catch (FileNotFoundException e) {
}

I suggest you use Java 8 to do the multi-threading as it is much cleaner.
files.parallelStream()
.filter(File::isFile)
.forEach(f -> new FileProcessThread(file, connectionVo).run());
Your task deletes the file when finished successfully.
This will only pass each file to one task.
BTW Don't call your tasks xxxThread unless they are actually a Thread, and avoid ever sub-classing a Thread.

Java runtime.exec() running multiple process using Threadpool

In my program I have a list of n test scripts , I need to iterate the list and run 3 test scripts in parallel. To achieve this task I created a Threadpool of size 3. My implementation is like below for the thread pool
ExecutorService executor = Executors.newFixedThreadPool(3);
for (int threadpoolCount = 0; threadpoolCount < classNames.size(); threadpoolCount++) {
Runnable worker = new ProcessRunnable(classNames.get(threadpoolCount));
executor.execute(worker);
}
executor.shutdown();
while (!executor.isTerminated()) {
}
System.out.println("Finished all threads");
below is my thread implementation where in i execute a batch file with maven commands in it
public void run() {
try {
System.out.println(testname);
System.out.println("Task ID : " + this.testname + " performed by " + Thread.currentThread().getName());
Process p = Runtime.getRuntime().exec("cmd /C Junit_runner.bat" + " " + testname);
p.waitFor();
Thread.sleep(5000);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Here is what I get in my console (I am not starting the command prompt and running it in background)
com.selenium.test.testname1
Task ID : com.selenium.test.testname1
performed by pool-1-thread-1
com.selenium.test.testname1
Task ID : com.selenium.test.testname2
performed by pool-1-thread-2
com.selenium.test.testname1
Task ID : com.selenium.test.testname3
performed by pool-1-thread-3
The execution pauses here and it didn't do anything , I am not sure what's happening behind. I also cross checked that the batch file its working fine.

The process takes long to execute and so your control is not returning back.
public abstract int waitFor() throws InterruptedException
Causes the current thread to wait, if necessary, until the process represented by this Process object has terminated. This method returns immediately if the subprocess has already terminated. If the subprocess has not yet terminated, the calling thread will be blocked until the subprocess exits.
As waitFor() is a blocking call all the 3 threads are stuck at this line.
NOTE: You don't need Thread.sleep(5000); as waitFor() is itself blocking in nature.
Try executing some other command and see if the control returns.
Also instead of:
while (!executor.isTerminated()) {
}
You can use ExecutorService#awaitTermination()

Read the p.getInputStream() and p.getErrorStream() and write it to your console instead of thread.sleep(), you will get an indication of what the threads are doing.