We Keep Coding

How to apply multithreading in java to find a word in a directory(main directory and the word given) and recursively its subdirectories

I am quite new on Stack Overflow and a beginner in Java so please forgive me if I have asked this question in an improper way.
PROBLEM
I have an assignment which tells me to make use of multi-threading to search files for a given word, which might be present in any file of type .txt and .html, on any-level in the given directory (So basically the entire directory). The absolute file path of the file has to be displayed on the console if the file contains the given word.
WHAT HAVE I TRIED
So I thought of dividing the task into 2 sections, Searching and Multithreading respectively,
I was able to get the Searching part( File_search.java ). This file has given satisfactory results by searching through the directory and finding all the files in it for the given word.
File_search.java
public class File_search{
String fin_output = "";
public String searchInTextFiles(File dir,String search_word) {
File[] a = dir.listFiles();
for(File f : a){
if(f.isDirectory()) {
searchInTextFiles(f,search_word);
}
else if(f.getName().endsWith(".txt") || f.getName().endsWith(".html") || f.getName().endsWith(".htm") ) {
try {
searchInFile(f,search_word);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
return fin_output;
}
public void searchInFile(File f,String search_word) throws FileNotFoundException {
final Scanner sc = new Scanner(f);
while(sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if(lineFromFile.contains(search_word)) {
fin_output += "FILE : "+f.getAbsolutePath().toString()+"\n";
}
}
}
Now, I want to be able to use multiple threads to execute the task File_search.java using ThreadPoolExecuter service. I'm not sure If I can do it using Runnable ,Callable or by using a Thread class or by any other method?
Can you please help me with the code to do the multi-threading part? Thanks :)

I agree to the comment of #chrylis -cautiouslyoptimistic, but for the purpose of understanding below will help you.
One simpler approach could be to do the traversal of directories in the main Thread, I mean the logic which you have added in function searchInTextFiles and do the searching logic as you did in function searchInFile in a Threadpool of size let's say 10.
Below sample code will help you to understand it better.
public class Traverser {
private List<Future<String>> futureList = new ArrayList<Future<String>>();
private ExecutorService executorService;
public Traverser() {
executorService = Executors.newFixedThreadPool(10);
}
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("Started");
long start = System.currentTimeMillis();
Traverser traverser = new Traverser();
traverser.searchInTextFiles(new File("Some Directory Path"), "Some Text");
for (Future<String> future : traverser.futureList) {
System.out.println(future.get());
}
traverser.executorService.shutdown();
while(!traverser.executorService.isTerminated()) {
System.out.println("Not terminated yet, sleeping");
Thread.sleep(1000);
}
long end = System.currentTimeMillis();
System.out.println("Time taken :" + (end - start));
}
public void searchInTextFiles(File dir,String searchWord) {
File[] filesList = dir.listFiles();
for(File file : filesList){
if(file.isDirectory()) {
searchInTextFiles(file,searchWord);
}
else if(file.getName().endsWith(".txt") || file.getName().endsWith(".html") || file.getName().endsWith(".htm") ) {
try {
futureList.add(executorService.submit(new SearcherTask(file,searchWord)));
} catch (Exception e) {
e.printStackTrace();
}
}
}
}}
public class SearcherTask implements Callable<String> {
private File inputFile;
private String searchWord;
public SearcherTask(File inputFile, String searchWord) {
this.inputFile = inputFile;
this.searchWord = searchWord;
}
#Override
public String call() throws Exception {
StringBuilder result = new StringBuilder();
Scanner sc = null;
try {
sc = new Scanner(inputFile);
while (sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if (lineFromFile.contains(searchWord)) {
result.append("FILE : " + inputFile.getAbsolutePath().toString() + "\n");
}
}
} catch (Exception e) {
//log error
throw e;
} finally {
sc.close();
}
return result.toString();
}}

Concurrent method works much longer than simple method. Why?

I wrote simple method for reading one file and writing into 2 files. Also I wrote this method with 3 threads, where the first thread reads line-by-line file, the second and the third write readed line into own file. My parallel version of method works in 30 times longer than simple sequential method. Please, help to understand what I do wrong, and how I should do this method to not shooting in my leg)))
private static class Tee {
private BufferedReader reader;
private PrintWriter fWriter;
private PrintWriter sWriter;
volatile boolean done;
String buffer;
volatile int readCount;
volatile int firstWriteCount;
volatile int secondWriteCount;
public Tee(BufferedReader reader, PrintWriter fWriter, PrintWriter sWriter) {
this.reader = reader;
this.fWriter = fWriter;
this.sWriter = sWriter;
}
public void teeWhileInsideCo() throws InterruptedException {
Thread reader = new Thread(new LineReader());
Thread fWriter = new Thread(new LineWriter(this.fWriter, 0));
Thread sWriter = new Thread(new LineWriter(this.sWriter, 1));
reader.start();
fWriter.start();
sWriter.start();
reader.join();
fWriter.join();
sWriter.join();
}
private class LineReader implements Runnable {
#Override
public void run() {
String line;
while (true) {
try {
line = reader.readLine();
} catch (IOException e) {
line = null;
}
while (readCount != firstWriteCount || readCount != secondWriteCount) {
}
if (line == null) {
done = true;
break;
}
buffer = line;
readCount++;
}
}
}
private class LineWriter implements Runnable {
private PrintWriter writer;
private int number;
public LineWriter(PrintWriter writer, int n) {
System.out.println(Tee.this);
this.writer = writer;
this.number = n;
}
#Override
public void run() {
while (true) {
while (!done && ((number == 0 ? firstWriteCount : secondWriteCount) == readCount)) {
}
if (done) {
break;
}
writer.println(buffer);
if (number == 0) {
firstWriteCount++;
} else {
secondWriteCount++;
}
}
writer.flush();
}
}
}

Read file with multiple threads

Is it possible to read a text file by running several threads, so that received line contains information about the thread that read this line?
For now, i can read with one thread:
public class Test {
public static void main(String[] args) throws InterruptedException {
Deque<String> deque = new LinkedList<>();
for (int i = 0; i < 4; i++) {
new Thread(new SubReadThread(deque)).start();
}
new Thread(new WriteThread(deque)).start();
}
}
class SubReadThread implements Runnable {
private final Deque<String> deque;
public SubReadThread(Deque<String> deque) {
this.deque = deque;
}
#Override
public void run() {
try {
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("list.txt"), "UTF8"));
String line = null;
String newLine;
while (true) {
synchronized (deque) {
if (deque.size() < 1) {
line = br.readLine();
newLine = "#" + (Thread.currentThread().getId() - 9) + " " + line;
deque.addLast(newLine);
deque.notify();
} else {
deque.wait();
}
if (line == null) {
break;
}
}
}
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
}
}
class WriteThread implements Runnable {
private final Deque<String> deque;
private List<String> list = new ArrayList<>();
public WriteThread(Deque<String> deque) {
this.deque = deque;
}
#Override
public void run() {
String line;
while (true) {
synchronized (deque) {
if (deque.size() > 0) {
if ((line = deque.pollFirst()).contains("null")) {
break;
} else {
list.add(line);
deque.notifyAll();
}
} else {
try {
deque.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
for(String s : list) {
System.out.println(s);
}
}
}
And expected output something like this:
#3 line1
#1 line2
#4 line3
#2 line4
...............
UPDATE All that was needed to work properly, move BufferedReader to main method and pass its object to the constructor.

Java BlockingQueue messages not processed not processed in full

I am trying to process few million records from a text file (i.e. reading the file sequentially using one thread, and trying to process the retrieved lines using multiple threads). A method call after 'queue.take();' is only executing for the number of times equal to initial capacity allocated to BlockingQueue (100 in this example), and then the process doesn't pickup anymore records.
Could you please help in debugging or identify the issue?
Main Method:
############
final int threadCount = 10;
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(100);
ExecutorService service = Executors.newFixedThreadPool(threadCount);
for (int i = 0; i < (threadCount - 1); i++) {
service.submit(new EvaluateLine(queue));
}
service.submit(new ProcessFile(queue)).get();
service.shutdownNow();
service.awaitTermination(365, TimeUnit.DAYS);
EvaluateLine:
#############
private final BlockingQueue<String> queue;
public EvaluateLine(BlockingQueue<String> queue){
this.queue = queue;
}
#Override
public void run() {
String line;
while(true) {
try {
line = queue.take();
SyncOutput.Process(line);
} catch (InterruptedException ex) {
break;
}
}
}
ProcessFile:
############
private final BlockingQueue<String> queue;
public ProcessFile(BlockingQueue<String> queue) {
this.queue = queue;
}
#Override
public void run() {
Path path = Paths.get("C:\\Desktop\\testdata.txt");
BufferedReader br = null;
try {
br =Files.newBufferedReader(path, StandardCharsets.UTF_8);
for (String line; (line = br.readLine()) != null; ) {
queue.put(line);
}
} catch(IOException e){
e.printStackTrace();
} catch(InterruptedException e){
e.printStackTrace();
} finally{
try {
if (br != null) br.close();
}catch(IOException e){
e.printStackTrace();
}
}
Edit:
SyncOutput
##########
class SyncOutput{
public static void ProcessExpression(String inputLine) {
evalExpression(inputLine);
}
public static double evalExpression(String s){
Expression e = new ExpressionBuilder(s)
.build();
return e.evaluate();
}
}

The following code segment(using Expression evaluation library: exp4j) is blocking the multi-thread execution, not sure why. But I have replaced this code block with a different library(parsii) and now everything looks fine.
public static void ProcessExpression(String inputLine) {
evalExpression(inputLine);
}
public static double evalExpression(String s){
Expression e = new ExpressionBuilder(s)
.build();
return e.evaluate();
}

Refactoring scanning folder class (add AllwaysPassesBlocker())

I need create a new constructor in FolderScan that takes a list of "Checkers". And all these "Checkers" always return true (schoud write new Chekers List that just return true.)
But problem is that I don't know how do this and not decompose structure of program.
Code (FolderScan and each Cheker):
class FolderScan implements Runnable {
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
checkers = new ArrayList<Checker>(Arrays.asList(
new ExtentionChecking(), new ProbeContentTypeCheking(),
new EncodingChecking() ));
}
#Override
public void run() {
try {
findFiles(path);
queue.put(endOfWorkFile);
latch.countDown();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void findFiles(String path) {
try {
File root = new File(path);
File[] list = root.listFiles();
for (File currentFile : list) {
boolean checksPassed = true;
if (currentFile.isDirectory()) {
findFiles(currentFile.getAbsolutePath());
} else {
for (Checker currentChecker : checkers) {
if (!currentChecker.check(currentFile)) {
checksPassed = false;
break;
}
}
if (checksPassed) {
queue.put(currentFile);
}
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
private String path;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
private List<Checker> checkers;
}
class ExtentionChecking implements Checker {
#Override
public boolean check(File currentFile) {
fileName = currentFile.getName().toLowerCase();
Set<String> extensions = new HashSet<String>(Arrays.asList(".txt",
".pdf", ".doc", ".docx", ".html", ".htm", ".xml", ".djvu",
".djv", ".rar", ".rtf", ".tmp"));
if (extensions.contains(fileName.substring(fileName.lastIndexOf(".")))) {
return true;
}
return false;
}
private String fileName;
}
class EncodingChecking implements Checker {
#Override
public boolean check(File currentFile) {
return detectEncoding(currentFile);
}
public static boolean detectEncoding(File file) {
detector = new CharsetDetector();
// validate input
if (null == file) {
throw new IllegalArgumentException("input file can't be null");
}
if (file.isDirectory()) {
throw new IllegalArgumentException(
"input file refers to a directory");
}
// read input file
byte[] buffer;
try {
buffer = readUTFHeaderBytes(file);
} catch (IOException e) {
throw new IllegalArgumentException(
"Can't read input file, error = " + e.getLocalizedMessage());
}
if(detector.setText(buffer) != null){
return true;
}
return false;
}
private static byte[] readUTFHeaderBytes(File input) throws IOException {
// read data
FileInputStream fileInputStream = new FileInputStream(input);
try {
byte firstBytes[] = new byte[50];
int count = fileInputStream.read(firstBytes);
if (count < 5) {
throw new IOException("Poor file!");
}
return firstBytes;
} finally {
fileInputStream.close();
}
}
private static CharsetDetector detector;
}
class ProbeContentTypeCheking implements Checker {
#Override
public boolean check(File currentFile) {
String mimeType = null;
try {
Path path = Paths.get(currentFile.getAbsolutePath());
byte[] data = Files.readAllBytes(path);
MagicMatch match = Magic.getMagicMatch(data);
mimeType = match.getMimeType();
} catch (MagicParseException | MagicMatchNotFoundException
| MagicException | IOException e) {
e.printStackTrace();
}
if (null != mimeType) {
return true;
}
return false;
}
}
Question:
How do refactor this code - after this able to make new
AllwaysPassesBlocker() and all Checers return true?

A checker that always returns true would be
class UncriticalChecker implements Checker {
#Override
public boolean check(File currentFile) {
return true;
}
}
There's no point adding such a checker to the list of checkers, though. You might as well leave the list empty.
I don't quite see why the checkers should be constructed in the constructor of the FolderScan. It seems more natural to pass them to the constructor as an argument.
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile, List<Checker> checkers) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
this.checkers = checkers;
}
Then, when you initialize the FolderScan, pass it the checkers
List<Checker> checkers = new ArrayList<Checker>(Arrays.asList(
new ExtentionChecking(), new ProbeContentTypeCheking(),
new EncodingChecking() ));
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, checkers);
Or, if you wish to create a FolderScan that returns all files, you pass it an empty list.
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, Collections.emptyList());
EDIT:
I now understand that you wish to test the class. Then the UncriticalChecker makes sense. If you want to test the code with a checker that always says yes, pass it to the constructor:
List<Checker> checkers = Collections.singletonList(new UncriticalChecker());
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, checkers);