We Keep Coding

How to apply multithreading in java to find a word in a directory(main directory and the word given) and recursively its subdirectories

I am quite new on Stack Overflow and a beginner in Java so please forgive me if I have asked this question in an improper way.
PROBLEM
I have an assignment which tells me to make use of multi-threading to search files for a given word, which might be present in any file of type .txt and .html, on any-level in the given directory (So basically the entire directory). The absolute file path of the file has to be displayed on the console if the file contains the given word.
WHAT HAVE I TRIED
So I thought of dividing the task into 2 sections, Searching and Multithreading respectively,
I was able to get the Searching part( File_search.java ). This file has given satisfactory results by searching through the directory and finding all the files in it for the given word.
File_search.java
public class File_search{
String fin_output = "";
public String searchInTextFiles(File dir,String search_word) {
File[] a = dir.listFiles();
for(File f : a){
if(f.isDirectory()) {
searchInTextFiles(f,search_word);
}
else if(f.getName().endsWith(".txt") || f.getName().endsWith(".html") || f.getName().endsWith(".htm") ) {
try {
searchInFile(f,search_word);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
return fin_output;
}
public void searchInFile(File f,String search_word) throws FileNotFoundException {
final Scanner sc = new Scanner(f);
while(sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if(lineFromFile.contains(search_word)) {
fin_output += "FILE : "+f.getAbsolutePath().toString()+"\n";
}
}
}
Now, I want to be able to use multiple threads to execute the task File_search.java using ThreadPoolExecuter service. I'm not sure If I can do it using Runnable ,Callable or by using a Thread class or by any other method?
Can you please help me with the code to do the multi-threading part? Thanks :)

I agree to the comment of #chrylis -cautiouslyoptimistic, but for the purpose of understanding below will help you.
One simpler approach could be to do the traversal of directories in the main Thread, I mean the logic which you have added in function searchInTextFiles and do the searching logic as you did in function searchInFile in a Threadpool of size let's say 10.
Below sample code will help you to understand it better.
public class Traverser {
private List<Future<String>> futureList = new ArrayList<Future<String>>();
private ExecutorService executorService;
public Traverser() {
executorService = Executors.newFixedThreadPool(10);
}
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("Started");
long start = System.currentTimeMillis();
Traverser traverser = new Traverser();
traverser.searchInTextFiles(new File("Some Directory Path"), "Some Text");
for (Future<String> future : traverser.futureList) {
System.out.println(future.get());
}
traverser.executorService.shutdown();
while(!traverser.executorService.isTerminated()) {
System.out.println("Not terminated yet, sleeping");
Thread.sleep(1000);
}
long end = System.currentTimeMillis();
System.out.println("Time taken :" + (end - start));
}
public void searchInTextFiles(File dir,String searchWord) {
File[] filesList = dir.listFiles();
for(File file : filesList){
if(file.isDirectory()) {
searchInTextFiles(file,searchWord);
}
else if(file.getName().endsWith(".txt") || file.getName().endsWith(".html") || file.getName().endsWith(".htm") ) {
try {
futureList.add(executorService.submit(new SearcherTask(file,searchWord)));
} catch (Exception e) {
e.printStackTrace();
}
}
}
}}
public class SearcherTask implements Callable<String> {
private File inputFile;
private String searchWord;
public SearcherTask(File inputFile, String searchWord) {
this.inputFile = inputFile;
this.searchWord = searchWord;
}
#Override
public String call() throws Exception {
StringBuilder result = new StringBuilder();
Scanner sc = null;
try {
sc = new Scanner(inputFile);
while (sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if (lineFromFile.contains(searchWord)) {
result.append("FILE : " + inputFile.getAbsolutePath().toString() + "\n");
}
}
} catch (Exception e) {
//log error
throw e;
} finally {
sc.close();
}
return result.toString();
}}

Block actions till list becomes non-empty with java.util.concurrent

I need your help. I should use the java.util.concurrent package in my exercise but I don't know how to do it. The question is only about the get method.
public String getInput() {
if (inputList.isEmpty()) return null;
String input = inputList.get(0);
inputList.remove(0);
return input;
}
How do I need to write the code to wait till the given list (variable: inputList) becomes non-empty?
Greetings

you could try using the LinkedBlockingDeque class from the java.util.concurrent
package which implements the BlockingDequeinterface.
it lets you add items to the BlockingDeque and the take* methods block until there is an element available and remove it after fetching. Have a look at the Javadoc
Here is an example:
public class Queue {
BlockingDeque<String> inputList = new LinkedBlockingDeque<>();
public String getInput() {
try {
System.out.println("waiting on queue");
String input = inputList.takeFirst();
System.out.println("taken " + input);
return input;
} catch (InterruptedException e) {
e.printStackTrace();
return null;
}
}
public static void main(String[] args) {
Queue queue = new Queue();
new Thread(() -> {
try {
Thread.sleep(4000);
queue.inputList.add("string");
System.out.println("added string");
Thread.sleep(2000);
queue.inputList.add("string1");
System.out.println("added string 1");
} catch (InterruptedException e) {
e.printStackTrace();
}
}).start();
for (int i = 0; i < 2; i++){
queue.getInput();
}
}
}

Why does Java file write consume CPU?

I am writing data to file using a queue on a separate thread, but the process consumes around 25% of CPU, as shown in this test main.
Is there something I can do to resolve this issue?
Perhaps I should be using flush() somewhere?
The test shows the main method start and run the queue thread and then send created data to it. The queue thread writes the data to a BufferedWriter which handles writing the data to a file.
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.logging.Level;
import java.util.logging.Logger;
import uk.co.moonsit.utils.timing.Time;
public class OutputFloatQueueReceiver extends Thread {
private static final Logger LOG = Logger.getLogger(OutputFloatQueueReceiver.class.getName());
private ConcurrentLinkedQueue<List<Float>> queue = null;
private boolean running = true;
private final BufferedWriter outputWriter;
private int ctr = 0;
private final int LIMIT = 1000;
public OutputFloatQueueReceiver(String outputFile, String header, ConcurrentLinkedQueue<List<Float>> q) throws IOException {
queue = q;
File f = new File(outputFile);
FileWriter fstream = null;
if (!f.exists()) {
try {
f.getParentFile().mkdirs();
if (!f.createNewFile()) {
throw new IOException("Exception when trying to create file " + f.getAbsolutePath());
}
fstream = new FileWriter(outputFile, false);
} catch (IOException ex) {
//Logger.getLogger(ControlHierarchy.class.getName()).log(Level.SEVERE, null, ex);
throw new IOException("Exception when trying to create file " + f.getAbsolutePath());
}
}
fstream = new FileWriter(outputFile, true);
outputWriter = new BufferedWriter(fstream);
outputWriter.append(header);
}
public synchronized void setRunning(boolean running) {
this.running = running;
}
#Override
public void run() {
while (running) {
while (queue.peek() != null) {
if (ctr++ % LIMIT == 0) {
LOG.log(Level.INFO, "Output Queue size = {0} '{'ctr={1}'}'", new Object[]{queue.size(), ctr});
}
List<Float> list = queue.poll();
if (list == null) {
continue;
}
try {
StringBuilder sbline = new StringBuilder();
Time t = new Time(list.get(0));
sbline.append(t.HMSS()).append(",");
for (Float f : list) {
sbline.append(f).append(",");
}
sbline.append("\n");
outputWriter.write(sbline.toString());
} catch (IOException ex) {
LOG.info(ex.toString());
break;
}
}
}
if (outputWriter != null) {
try {
outputWriter.close();
LOG.info("Closed outputWriter");
} catch (IOException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
public static void main(String[] args) {
try {
String outputFile = "c:\\tmp\\qtest.csv";
File f = new File(outputFile);
f.delete();
StringBuilder header = new StringBuilder();
header.append("1,2,3,4,5,6,7,8,9");
header.append("\n");
ConcurrentLinkedQueue<List<Float>> outputQueue = null;
OutputFloatQueueReceiver outputQueueReceiver = null;
outputQueue = new ConcurrentLinkedQueue<>();
outputQueueReceiver = new OutputFloatQueueReceiver(outputFile, header.toString(), outputQueue);
outputQueueReceiver.start();
for (int i = 1; i < 100000; i++) {
List<Float> list = new ArrayList<>();
//list.set(0, (float) i); // causes exception
list.add((float) i);
for (int j = 1; j < 10; j++) {
list.add((float) j);
}
outputQueue.add(list);
}
try {
Thread.sleep(5000);
} catch (InterruptedException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
outputQueueReceiver.setRunning(false);
} catch (IOException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
}
}

This code is the reason while your code is using so much CPU:
while (running) {
while (queue.peek() != null) {
// logging
List<Float> list = queue.poll();
if (list == null) {
continue;
}
// do stuff with list
}
}
Basically, your code is busy-waiting, repeatedly "peeking" until a queue entry becomes available. It is probably spinning there in a tight loop.
You should replace your queue class with a BlockingQueue, and simply use take() ... like this:
while (running) {
List<Float> list = queue.take();
// do stuff with list
}
The take() call block indefinitely, only returning once there is an element available, and returning that element as the result. If blocking indefinitely is a problem, you could either use poll(...) with a timeout, or you could arrange that some other thread interrupts the thread that is blocked.

Is it possible to know the newly added symbol in a HashSet?

I have this piece of code inside my application which runs continuously .
When ever a symbol is added , this below Thread gets fired up and executes two different tasks ( currently the task is represented as sys out for simplicity )
For the first time everything runs fine , but from the second time , the task is being repeated for all the symbols present inside the allSymbolsSet .
The issue i am facing here is that i want to run the task only for the new symbol added . (For example if the allSymbolsSet consists of 3 symbols initially and when a new symbol is added to it , it runs that task for all the 4 symbols , whereas i want it to execute it only for the newly added symbol )
This is my code
package com;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import java.util.concurrent.PriorityBlockingQueue;
public class TaskerThread extends Thread {
private PriorityBlockingQueue<String> priorityBlocking = new PriorityBlockingQueue<String>();
private Set<String> allSymbolsSet = new HashSet<String>();
public void addSymbols(String str) {
if (str != null) {
priorityBlocking.add(str);
}
}
public void run() {
while (true) {
try {
boolean added = false;
while (priorityBlocking.peek() != null) {
added = true;
String symbol = priorityBlocking.poll();
allSymbolsSet.add(symbol);
try {
System.out.println("Symbol From priorityBlocking"+ " " + symbol);
} catch (Exception e) {
e.printStackTrace();
}
}
Iterator<String> ite = allSymbolsSet.iterator();
if (added) {
while (ite.hasNext()) {
String symbol = ite.next();
if (symbol != null && symbol.trim().length() > 0) {
try {
System.out.println("Symbol From allSymbolsSet"+ " " + symbol);
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
Thread.sleep(2000);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public static void main(String args[]) {
try {
TaskerThread qT = new TaskerThread();
qT.start();
qT.addSymbols("SymbolTest");
Thread.sleep(110);
qT.addSymbols("Symbo2222222");
} catch (Exception e) {
e.printStackTrace();
}
}
}

add() method returns false if the Object being added was ignored because it was already present

A simple solution would be to have two hashsets - set1, holding all symbols, set2 containing newly added symbols. Add new symbols to set2, in your thread's run, when the execution is complete, add new symbol to set1 and remove it from set2. How about that?

Well, of course it runs for all elements in the set, you are iterating over them!
package com;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import java.util.concurrent.PriorityBlockingQueue;
public class TaskerThread extends Thread {
private final PriorityBlockingQueue<String> priorityBlocking = new PriorityBlockingQueue<String>();
private final Set<String> allSymbolsSet = new Collections.synchronizedSet(new HashSet<String>());
public void addSymbols(String str) {
if (str != null) {
priorityBlocking.add(str);
}
}
public void run() {
while (true) {
try {
while (true) {
final String symbol = priorityBlocking.take();
if (allSymbolsSet.add(symbol)) {
doSomething(symbol); // do whatever you want with the symbol
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public static void main(String args[]) {
try {
TaskerThread qT = new TaskerThread();
qT.start();
qT.addSymbols("SymbolTest");
} catch (Exception e) {
e.printStackTrace();
}
}
}
This should do what you were looking for. Take better care of possible exceptions, namely InterruptedException.

How to organize multithreadings work with queue?

I'm begginer and keep yourself in hends.
I need to do organize multithreadings find in files:
User input where find(path) and what find(word);
First thread finds .txt files in folder and add result to queue;
When queue has some file => Second thread start find in this file
what need to find(word).
If was finded success would show path this
file + how offen times this word meets in file.
Qestions:
Can we use ArrayList (or exist any alternatives) for queue which works with few threads?
How to do if queue is empty, Second thread don't start but waits when First finded need file?
Need we use synchronized for this task and inherited MultiThreadingSearch(or better to use composition)?
Code:
import java.util.*;
import java.io.*;
class ArrayListOfFiles {
private Node first, last;
private class Node {
String item;
Node next;
}
public boolean isEmpty() {
return first == null;
}
public synchronized void enqueue(String item) {
Node oldlast = last;
last = new Node();
last.item = item;
last.next = null;
if (isEmpty())
first = last;
else
oldlast.next = last;
}
public synchronized String dequeue() {
String item = first.item;
first = first.next;
if (isEmpty())
last = null;
return item;
}
}
class FolderScan extends MultiThreadingSearch implements Runnable {
FolderScan(String path, String whatFind) {
super(path, whatFind);
}
#Override
public void run() {
findFiles(path);
}
ArrayListOfFiles findFiles(String path) {
File root = new File(path);
File[] list = root.listFiles();
for (File titleName : list) {
if (titleName.isDirectory()) {
findFiles(titleName.getAbsolutePath());
} else {
if (titleName.getName().toLowerCase().endsWith((".txt"))) {
textFiles.enqueue(titleName.getName());
}
}
}
return textFiles;
}
}
class FileScan extends MultiThreadingSearch implements Runnable {
Scanner scanner = new Scanner((Readable) textFiles);
Set<String> words = new HashSet<String>();
int matches = 0;
FileScan(String file, String whatFind) {
super(file, whatFind);
Thread wordFind = new Thread();
wordFind.start();
}
#Override
public void run() {
while (scanner.hasNext()) {
String word = scanner.next();
words.add(word);
}
if (words.contains(this.whatFind)) {
System.out.println("File:" + this.path);
matches++;
}
System.out.println(matches);
}
}
public class MultiThreadingSearch {
String path;
String whatFind;
ArrayListOfFiles textFiles;
MultiThreadingSearch(String path, String whatFind) {
this.path = path;
this.whatFind = whatFind;
this.textFiles = new ArrayListOfFiles();
Thread pathFind = new Thread(new FolderScan(path, whatFind));
// pathFind.start();
if (!textFiles.isEmpty()) {
#SuppressWarnings("unused")
FileScan fileScan = new FileScan(textFiles.dequeue(), whatFind);
}
}
// ask user about input
public static void askUserPathAndWord() {
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(System.in));
String path;
String whatFind;
try {
System.out.println("Please, enter a Path and Word"
+ "(which you want to find):");
System.out.println("Please enter a Path:");
path = bufferedReader.readLine();
System.out.println("Please enter a Word:");
whatFind = bufferedReader.readLine();
if (path != null && whatFind != null) {
new MultiThreadingSearch(path, whatFind);
System.out.println("Thank you!");
} else {
System.out.println("You did not enter anything");
}
} catch (IOException | RuntimeException e) {
System.out.println("Wrong input!");
e.printStackTrace();
}
}
public static void main(String[] args) {
askUserPathAndWord();
}
}
I got Exception in thread "main" java.lang.StackOverflowError from this code.
How able to solve this task?
Thanks,
Nazar.

Check BlockingQueue it does exactly what you need. Thread can block until some other thread add new item to queue.
As to how decompose you system. I'd do following:
Create class for searching txt files in path. It implements Runnable. You pass path and queue to it. And it searches path for txt files and adds them to the queu.
Create class for searching file content. It implements Runnable. You pass whatFind and queue to it and it takes new file from queue and checks it's content.
Something like:
BlockingQueue<File> queue = new LinkedBlockingQueue<File>();
String path = ...;
String whatFind = ...;
FolderScan folderScan = new FolderScan(path, queue);
FileScan fileScan = new FileScan(whatFind, queue);
Executor executor = Executors.newCachecThreadPool();
executor.execute(folderScan);
executor.execute(fileScan);
If you want FileScan to wait until FolderScan adds something to the queue you can use take method:
BlockingQueue<File> queue;
File toProcess = queue.take(); // this line blocks current thread (FileScan) until someone adds new item to the queue.

After changes:
package task;
import java.util.concurrent.*;
import java.util.*;
import java.io.*;
class FolderScan implements Runnable {
private String path;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
}
public FolderScan() { }
#Override
public void run() {
findFiles(path);
queue.add(endOfWorkFile);
latch.countDown();
}
private void findFiles(String path) {
try {
File root = new File(path);
File[] list = root.listFiles();
for (File currentFile : list) {
if (currentFile.isDirectory()) {
findFiles(currentFile.getAbsolutePath());
} else {
if (currentFile.getName().toLowerCase().endsWith((".txt"))) {
queue.put(currentFile);
}
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public class FileScan implements Runnable {
private String whatFind;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
public FileScan(String whatFind, BlockingQueue<File> queue,
CountDownLatch latch, File endOfWorkFile) {
this.whatFind = whatFind;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
}
public FileScan() { }
Set<String> words = new HashSet<String>();
int matches = 0;
#Override
public void run() {
while (true) {
try {
File file;
file = queue.take();
if (file == endOfWorkFile) {
break;
}
scan(file);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
latch.countDown();
}
private void scan(File file) {
Scanner scanner = null;
try {
scanner = new Scanner(file);
} catch (FileNotFoundException e) {
System.out.println("FileNotFoundException.");
e.printStackTrace();
}
while (scanner.hasNext()) {
String word = scanner.next();
words.add(word);
}
if (words.contains(this.whatFind)) {
matches++;
}
String myStr = String.format("File: %s and the number of matches "
+ "is = %d", file.getAbsolutePath(), matches);
System.out.println(myStr);
matches = 0;
}
// ask user about input
public void askUserPathAndWord() {
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(System.in));
String path;
String whatFind;
BlockingQueue<File> queue = new LinkedBlockingQueue<File>();
try {
System.out.println("Please, enter a Path and Word"
+ "(which you want to find):");
System.out.println("Please enter a Path:");
path = bufferedReader.readLine();
System.out.println("Please enter a Word:");
whatFind = bufferedReader.readLine();
if (path != null && whatFind != null) {
File endOfWorkFile = new File("GameOver.tmp");
CountDownLatch latch = new CountDownLatch(2);
FolderScan folderScan = new FolderScan(path, queue, latch,
endOfWorkFile);
FileScan fileScan = new FileScan(whatFind, queue, latch,
endOfWorkFile);
Executor executor = Executors.newCachedThreadPool();
executor.execute(folderScan);
executor.execute(fileScan);
latch.await();
System.out.println("Thank you!");
} else {
System.out.println("You did not enter anything");
}
} catch (IOException | RuntimeException e) {
System.out.println("Wrong input!");
e.printStackTrace();
} catch (InterruptedException e) {
System.out.println("Interrupted.");
e.printStackTrace();
}
}
/**
* #param args
*/
public static void main(String[] args) {
new FileScan().askUserPathAndWord();
}
}

This may not sound too constructive, but you can either fix that code or read something like this first and then throw away your code.
Stackoverflow usually results from a recursion running deeper than expected. Make sure there is some condition in you recursive method that stops recursion.