We Keep Coding

How to apply multithreading in java to find a word in a directory(main directory and the word given) and recursively its subdirectories

I am quite new on Stack Overflow and a beginner in Java so please forgive me if I have asked this question in an improper way.
PROBLEM
I have an assignment which tells me to make use of multi-threading to search files for a given word, which might be present in any file of type .txt and .html, on any-level in the given directory (So basically the entire directory). The absolute file path of the file has to be displayed on the console if the file contains the given word.
WHAT HAVE I TRIED
So I thought of dividing the task into 2 sections, Searching and Multithreading respectively,
I was able to get the Searching part( File_search.java ). This file has given satisfactory results by searching through the directory and finding all the files in it for the given word.
File_search.java
public class File_search{
String fin_output = "";
public String searchInTextFiles(File dir,String search_word) {
File[] a = dir.listFiles();
for(File f : a){
if(f.isDirectory()) {
searchInTextFiles(f,search_word);
}
else if(f.getName().endsWith(".txt") || f.getName().endsWith(".html") || f.getName().endsWith(".htm") ) {
try {
searchInFile(f,search_word);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
return fin_output;
}
public void searchInFile(File f,String search_word) throws FileNotFoundException {
final Scanner sc = new Scanner(f);
while(sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if(lineFromFile.contains(search_word)) {
fin_output += "FILE : "+f.getAbsolutePath().toString()+"\n";
}
}
}
Now, I want to be able to use multiple threads to execute the task File_search.java using ThreadPoolExecuter service. I'm not sure If I can do it using Runnable ,Callable or by using a Thread class or by any other method?
Can you please help me with the code to do the multi-threading part? Thanks :)

I agree to the comment of #chrylis -cautiouslyoptimistic, but for the purpose of understanding below will help you.
One simpler approach could be to do the traversal of directories in the main Thread, I mean the logic which you have added in function searchInTextFiles and do the searching logic as you did in function searchInFile in a Threadpool of size let's say 10.
Below sample code will help you to understand it better.
public class Traverser {
private List<Future<String>> futureList = new ArrayList<Future<String>>();
private ExecutorService executorService;
public Traverser() {
executorService = Executors.newFixedThreadPool(10);
}
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("Started");
long start = System.currentTimeMillis();
Traverser traverser = new Traverser();
traverser.searchInTextFiles(new File("Some Directory Path"), "Some Text");
for (Future<String> future : traverser.futureList) {
System.out.println(future.get());
}
traverser.executorService.shutdown();
while(!traverser.executorService.isTerminated()) {
System.out.println("Not terminated yet, sleeping");
Thread.sleep(1000);
}
long end = System.currentTimeMillis();
System.out.println("Time taken :" + (end - start));
}
public void searchInTextFiles(File dir,String searchWord) {
File[] filesList = dir.listFiles();
for(File file : filesList){
if(file.isDirectory()) {
searchInTextFiles(file,searchWord);
}
else if(file.getName().endsWith(".txt") || file.getName().endsWith(".html") || file.getName().endsWith(".htm") ) {
try {
futureList.add(executorService.submit(new SearcherTask(file,searchWord)));
} catch (Exception e) {
e.printStackTrace();
}
}
}
}}
public class SearcherTask implements Callable<String> {
private File inputFile;
private String searchWord;
public SearcherTask(File inputFile, String searchWord) {
this.inputFile = inputFile;
this.searchWord = searchWord;
}
#Override
public String call() throws Exception {
StringBuilder result = new StringBuilder();
Scanner sc = null;
try {
sc = new Scanner(inputFile);
while (sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if (lineFromFile.contains(searchWord)) {
result.append("FILE : " + inputFile.getAbsolutePath().toString() + "\n");
}
}
} catch (Exception e) {
//log error
throw e;
} finally {
sc.close();
}
return result.toString();
}}

Apache Camel: Download Multiple Files at once using SFTP component

on sftp i have several files with following xyz names:
40_20200313_0cd6963f-bf5b-4eb0-b310-255a23ed778e_p.dat
123_20200313_0cd6963f-bf5b-4eb0-b310-255a23ed778e_p.dat
etc.
I want camel to download all files at once as currently it is downloading file one by one.
Following is camel route and query:
private static String regex() {
return "(22|23|24|25|26|28|29|32|35|40|41|46|52|70|85|88|123)_(?:.*)_p.dat";
}
private static String sftpComponent() {
return "sftp://transit.ergogroup.no/Eyeshare/From_Eyeshare_Test"
+ "?username=Eyeshare_test"
+ "&password=epw3ePOugG" // Stored on wildfly server
+ "&download=true" //Shall be read chunk by chunk to avoid heap space issues. Earlier download=true was used: Harpreet
+ "&useList=true"
+ "&stepwise=false"
+ "&disconnect=true"
+ "&passiveMode=true"
+ "&reconnectDelay=10000"
// + "&bridgeErrorHandler=true"
+ "&delay=300000"
//+ "&fileName=" + sftpFileName
// + "&include=kiki\\.txt"
// + "&include=40_*_p\\.dat"sss
+ "&include="+regex()
+ "&preMove=$simple{file:onlyname}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.processing"
+ "&move=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.success"
+ "&moveFailed=$simple{file:onlyname.noext}.$simple{date:now:yyyy-MM-dd'T'hh-mm-ss}.failed";
// + "&idempotentRepository=#infinispan"
// + "&readLockRemoveOnCommit=true";
}
from(sftpComponent()).log("CHU").to(archiveReceivedFile())
Code appears fine but output is not. Anyone kindly suggest

Here some example of aggregator:
from("file:///somePath/consume/?maxMessagesPerPoll=2&delay=5000")
.aggregate(constant(true), new ZipAggregationStrategy()).completion(exchange -> exchange.getProperty("CamelBatchComplete", Boolean.class))
.to("file:///somePath/produce/")
Here maxMessagesPerPoll defining how many files will be archived. But if number of them in folder is lower then maxMessagesPerPoll value it will wait for missing files for complete archive. Here example of ZipAggregationStrategy:
private static class ZipAggregationStrategy implements AggregationStrategy {
private ZipOutputStream zipOutputStream;
private ByteArrayOutputStream out;
#Override
public Exchange aggregate(final Exchange oldExchange, final Exchange newExchange) {
try {
if (oldExchange == null) {
out = new ByteArrayOutputStream();
zipOutputStream = new ZipOutputStream(out);
}
createEntry(newExchange);
return newExchange;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private void createEntry(final Exchange exchange) throws Exception {
final ZipEntry zipEntry = new ZipEntry(exchange.getIn().getHeader(Exchange.FILE_NAME, String.class));
zipOutputStream.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
try (InputStream body = exchange.getIn().getBody(InputStream.class)) {
while ((length = body.read(bytes)) >= 0) {
zipOutputStream.write(bytes, 0, length);
}
}
}
#Override
public void onCompletion(final Exchange exchange) {
try {
zipOutputStream.close();
exchange.getIn().setBody(new ByteArrayInputStream(out.toByteArray()));
exchange.getIn().setHeader(Exchange.FILE_NAME, "someArchive.zip");
}catch (Exception e){
throw new RuntimeException(e);
} finally {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
It's in-memory example. You can improve it for example with using temporary file. And you can always create your own completion predicate based on your logic.
UPD: i think link for documentation is temporary unavailable

How to verify file is finished being written before attempting to process it?

Hard to give a good title to my problem but it is as follows. First I'm doing this on windows and it could possibly be used on a linux box also so I'd need the fix to work on both systems. I am monitoring a directory for new files. I basically looking at the directory's files and comparing them over and over and only processing the new files. Problem is I keep getting an error where the file isn't finished being written before I attempt to process.
public class LiveDetectionsProvider extends DetectionsProvider {
protected LiveDetectionsProvider.MonitorDirectory monitorDirectory = null;
protected TimeModel timeModel = null;
private ArrayList<String> loaded = new ArrayList();
private File topLayerFolder = null;
public LiveDetectionsProvider(String directory, String id) {
super(directory, id);
timeModel = super.timeModel;
}
/**
* Initialize the data provider.
*/
public void initialize() {
try {
topLayerFolder = new File(directory);
File[] dir = topLayerFolder.listFiles();
for (File file : dir) {
loaded.add(file.getName());
}
monitorDirectory = new MonitorDirectory();
monitorDirectory.execute();
}
catch (Exception ex) {
Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Failed to read detection\n{0}", ex.getMessage());
}
super.initialize();
}
/**
* Un-initialize the data provider.
*/
public void uninitialize() {
super.uninitialize();
if (monitorDirectory != null) {
monitorDirectory.continuing = false;
}
}
/**
* The class that is used to load the detection points in a background
* thread.
*/
protected class MonitorDirectory extends SwingWorker<Void, Void> {
public boolean continuing = true;
/**
* The executor service thread pool.
*/
private ExecutorService executor = null;
/**
* The completion service that reports the completed threads.
*/
private CompletionService<Object> completionService = null;
#Override
protected Void doInBackground() throws Exception {
int count = 0;
executor = Executors.newFixedThreadPool(1);
completionService = new ExecutorCompletionService<>(executor);
while (continuing && topLayerFolder != null) {
File[] dir = topLayerFolder.listFiles();
Thread.sleep(10);
ArrayList<File> filesToLoad = new ArrayList();
for (File file : dir) {
if (!loaded.contains(file.getName())) {
long filesize = 0;
boolean cont = true;
while (cont) {
if (file.length() == filesize) {
cont = false;
Thread.sleep(3);
filesToLoad.add(file);
}
else {
filesize = file.length();
Thread.sleep(3);
}
}
Thread.sleep(3);
}
}
for (File file : filesToLoad) {
timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), true);
completionService.submit(Executors.callable(new ReadDetection(file, false)));
while (completionService.take() == null) {
Thread.sleep(2);
}
loaded.add(file.getName());
count++;
Logger.getLogger(LiveDetectionsProvider.class.getName()).log(Level.SEVERE, "Detection Message Count:" + count);
}
detectionsModel.fireStateChanged(DetectionsModel.CHANGE_EVENT_DETECTIONS);
timeModel.setLoadingData(LiveDetectionsProvider.this.hashCode(), false);
}
return null;
}
}
}
The file is processed at the line with
completionService.submit(Executors.callable(new ReadDetection(file, false)));
The file at this point still hasnt finished being written and thus fails. I've tried sleeping my thread to slow it down, and I've tried verifying the file size hasn't changed. My test case for this is I'm unzipping a tar file which contains tons of 1,000 KB files.

Usually I solve this issue by create a temporary file while the file is being written. Once finish I rename the file and only the renamed file can be process.

Use a "flag file": once file.txt is "finished", indicate this by creating a file.flg - consuming process should wait for .flg to appear.

First yes it compiles look a the solution I posted in relation to the code In my question. You substitute
For(File file: dir){
while(!file.renameTo(file)){
Thread.sleep(1)
}
// In my code I check to see if the file name is already in the list which
// contains files that have been previously loaded if its not I add it to a list
// of files to be processed
}
in for
for (File file : dir) {
if (!loaded.contains(file.getName())) {
long filesize = 0;
boolean cont = true;
while (cont) {
if (file.length() == filesize) {
cont = false;
Thread.sleep(3);
filesToLoad.add(file);
}
else {
filesize = file.length();
Thread.sleep(3);
}
}
Thread.sleep(3);
}
}
sorry I forgot to put comment tags // in on the line that said do what every you need to do here.
What it does is it looks at each file in the directory and checks to see if you can rename it if the rename fails it sleeps and continues checking till it sucessfully is able to rename at which point you can do what you need to with the file which in my case was everything after the for loop that was replaced. I'm curious why my awnser was viewed as sub par deleted and locked. This solution does work and solved my problem and would anyone else who's having the same issue attempting to process a file that still being written or copied to a directory thats being monitored for changes.

Where to provide try catch block

I am trying to copy files from windows server1 to another windows server2 and not sure where to put the try catch block. I want to inform the user whenver windows server1 or windows server2 shuts down while copying process is ongoing either throught a popup or displaying in a textArea and here is my swingworker code. Thanks in advance
class CopyTask extends SwingWorker<Void, Integer>
{
private File source;
private File target;
private long totalBytes = 0;
private long copiedBytes = 0;
public CopyTask(File src, File dest)
{
this.source = src;
this.target = dest;
progressAll.setValue(0);
progressCurrent.setValue(0);
}
#Override
public Void doInBackground() throws Exception
{
ta.append("Retrieving info ... ");
retrieveTotalBytes(source);
ta.append("Done!\n");
copyFiles(source, target);
return null;
}
#Override
public void process(List<Integer> chunks)
{
for(int i : chunks)
{
progressCurrent.setValue(i);
}
}
#Override
public void done()
{
setProgress(100);
}
private void retrieveTotalBytes(File sourceFile)
{
File[] files = sourceFile.listFiles();
for(File file : files)
{
if(file.isDirectory()) retrieveTotalBytes(file);
else totalBytes += file.length();
}
}
private void copyFiles(File sourceFile, File targetFile) throws IOException
{
if(sourceFile.isDirectory())
{
if(!targetFile.exists()) targetFile.mkdirs();
String[] filePaths = sourceFile.list();
for(String filePath : filePaths)
{
File srcFile = new File(sourceFile, filePath);
File destFile = new File(targetFile, filePath);
copyFiles(srcFile, destFile);
}
}
else
{
ta.append("Copying " + sourceFile.getAbsolutePath() + " to " + targetFile.getAbsolutePath() ); //appends to textarea
bis = new BufferedInputStream(new FileInputStream(sourceFile));
bos = new BufferedOutputStream(new FileOutputStream(targetFile));
long fileBytes = sourceFile.length();
long soFar = 0;
int theByte;
while((theByte = bis.read()) != -1)
{
bos.write(theByte);
setProgress((int) (copiedBytes++ * 100 / totalBytes));
publish((int) (soFar++ * 100 / fileBytes));
}
bis.close();
bos.close();
publish(100);
}
}

Where is the line where the exception can happen? That's the first place I locate any exception.
Generally, if your modules are small, you can wrap the try around all the real code in the module and catch the exceptions at the end, especially if the exception is fatal. Then you can log the exception and return an error message/status to the user.
However, the strategy is different if the exception is not fatal. In this case you'll have to handle it right where the connection exception is thrown so you can seamlessly resume when the connection returns. Of course, this is a little more work.
EDIT - you probably want bis.close() and bos.close() inside a finally block to ensure they get closed. It may be pedantic but it seems prudent.

SVN Commit Log Issue

I am currently working on a project about calculations.I have done the main part of my project,Also integrated SVN Commit function to my code (using .ini file to read the specific address etc. )
I can easily Commit the files, what I am trying is I want to implement the real-time log to my console. Is there any way to implement the log to the console ? Not the general log but the commit log which should be real time.
I am using eclipse for mac, I've heard about SVNKit but I am really poor about SVN.
Thanks in advance for any information
--- EDIT ---
This is the code for reading the svn commands from .ini file
public static String iniSVNOkut(String deger, String getObje, String fetchObje){
Ini uzantilariAlIni = null;
try
{
String uzantiAyarlari = "Uzantilar.ini";
try
{
uzantilariAlIni = new Ini(new FileReader(uzantiAyarlari));
}
catch (InvalidFileFormatException e)
{
System.err.print("Hata InvalidFileFormat : " + e.getMessage() + "\n" );
e.printStackTrace();
}
catch(FileNotFoundException e)
{
System.err.print("Hata FileNotFoundException : " + e.getMessage() + "\n" );
e.printStackTrace();
}
catch (IOException e)
{
System.err.print("Hata IOException : " + e.getMessage() + "\n" );
e.printStackTrace();
}
return deger = uzantilariAlIni.get(getObje).fetch(fetchObje);
}
finally
{
}
}
This is what .ini includes
[svnAdresi]
svnAdresiniAl = svn co http://svn.svnkit.com/repos/svnkit/trunk/ /Users/sample/Documents/workspace/SatirHesaplaGUI/svnTestMAC
This is how I call it
String svnAdresi;
svnAdresi = IniFonksiyon.iniSVNOkut(svnAdresi, "svnAdresi", "svnAdresiniAl");
Runtime cmdCalistir = Runtime.getRuntime();
try
{
Process islem = cmdCalistir.exec(svnAdresi);
}
catch (Exception e)
{
e.printStackTrace();
}

If I understand your question correctly, you want to read the Subversion commit log into your console application.
The easiest way is to use SVNKit.
Here's how I did it.
private static List<SVNLogEntry> logEntryList;
/*
* Gets the Subversion log records for the directory
*/
LogHandler handler = new LogHandler();
String[] paths = { directory };
try {
repository.log(paths, latestRevision, 1L, false, true, handler);
} catch (SVNException svne) {
if (svne.getMessage().contains("not found")) {
logEntryList = new ArrayList<SVNLogEntry>();
} else {
CobolSupportLog.logError(
"Error while fetching the repository history: "
+ svne.getMessage(), svne);
return false;
}
}
logEntryList = handler.getLogEntries();
directory - string pointing to a particular directory or module
latestRevision - largest revision number from Subversion. Placing the latestRevision second in the log method invocation returns the log records in most recent order.
If you want the log records in sequential order, from 1 to latestRevision, then the 1L would be placed second, and the latestRevision would be placed third.
repository - Subversion repository that you've already authenticated.
Here's LogHandler.
public class LogHandler implements ISVNLogEntryHandler {
protected static final int REVISION_LIMIT = 5;
protected List<SVNLogEntry> logEntryList;
public LogHandler() {
logEntryList = new ArrayList<SVNLogEntry>();
}
public void handleLogEntry(SVNLogEntry logEntry) throws SVNException {
logEntryList.add(logEntry);
}
public List<SVNLogEntry> getLogEntries() {
if (logEntryList.size() <= REVISION_LIMIT) {
return logEntryList;
} else {
return logEntryList.subList(0, REVISION_LIMIT);
}
}
}