I am quite new on Stack Overflow and a beginner in Java so please forgive me if I have asked this question in an improper way.
PROBLEM
I have an assignment which tells me to make use of multi-threading to search files for a given word, which might be present in any file of type .txt and .html, on any-level in the given directory (So basically the entire directory). The absolute file path of the file has to be displayed on the console if the file contains the given word.
WHAT HAVE I TRIED
So I thought of dividing the task into 2 sections, Searching and Multithreading respectively,
I was able to get the Searching part( File_search.java ). This file has given satisfactory results by searching through the directory and finding all the files in it for the given word.
File_search.java
public class File_search{
String fin_output = "";
public String searchInTextFiles(File dir,String search_word) {
File[] a = dir.listFiles();
for(File f : a){
if(f.isDirectory()) {
searchInTextFiles(f,search_word);
}
else if(f.getName().endsWith(".txt") || f.getName().endsWith(".html") || f.getName().endsWith(".htm") ) {
try {
searchInFile(f,search_word);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
return fin_output;
}
public void searchInFile(File f,String search_word) throws FileNotFoundException {
final Scanner sc = new Scanner(f);
while(sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if(lineFromFile.contains(search_word)) {
fin_output += "FILE : "+f.getAbsolutePath().toString()+"\n";
}
}
}
Now, I want to be able to use multiple threads to execute the task File_search.java using ThreadPoolExecuter service. I'm not sure If I can do it using Runnable ,Callable or by using a Thread class or by any other method?
Can you please help me with the code to do the multi-threading part? Thanks :)
I agree to the comment of #chrylis -cautiouslyoptimistic, but for the purpose of understanding below will help you.
One simpler approach could be to do the traversal of directories in the main Thread, I mean the logic which you have added in function searchInTextFiles and do the searching logic as you did in function searchInFile in a Threadpool of size let's say 10.
Below sample code will help you to understand it better.
public class Traverser {
private List<Future<String>> futureList = new ArrayList<Future<String>>();
private ExecutorService executorService;
public Traverser() {
executorService = Executors.newFixedThreadPool(10);
}
public static void main(String[] args) throws InterruptedException, ExecutionException {
System.out.println("Started");
long start = System.currentTimeMillis();
Traverser traverser = new Traverser();
traverser.searchInTextFiles(new File("Some Directory Path"), "Some Text");
for (Future<String> future : traverser.futureList) {
System.out.println(future.get());
}
traverser.executorService.shutdown();
while(!traverser.executorService.isTerminated()) {
System.out.println("Not terminated yet, sleeping");
Thread.sleep(1000);
}
long end = System.currentTimeMillis();
System.out.println("Time taken :" + (end - start));
}
public void searchInTextFiles(File dir,String searchWord) {
File[] filesList = dir.listFiles();
for(File file : filesList){
if(file.isDirectory()) {
searchInTextFiles(file,searchWord);
}
else if(file.getName().endsWith(".txt") || file.getName().endsWith(".html") || file.getName().endsWith(".htm") ) {
try {
futureList.add(executorService.submit(new SearcherTask(file,searchWord)));
} catch (Exception e) {
e.printStackTrace();
}
}
}
}}
public class SearcherTask implements Callable<String> {
private File inputFile;
private String searchWord;
public SearcherTask(File inputFile, String searchWord) {
this.inputFile = inputFile;
this.searchWord = searchWord;
}
#Override
public String call() throws Exception {
StringBuilder result = new StringBuilder();
Scanner sc = null;
try {
sc = new Scanner(inputFile);
while (sc.hasNextLine()) {
final String lineFromFile = sc.nextLine();
if (lineFromFile.contains(searchWord)) {
result.append("FILE : " + inputFile.getAbsolutePath().toString() + "\n");
}
}
} catch (Exception e) {
//log error
throw e;
} finally {
sc.close();
}
return result.toString();
}}
I have written a very simple Java program to copy a file passed as an argument to the /tmp directory. The program produces several Java exceptions.
public class CopyFile {
public static void main(String[] args) throws IOException {
String fqp2File = "";
if (new File(args[0]).isFile()) {
fqp2File = args[0];
}
else {
System.out.println("Passed argument is not a file");
}
copy(fqp2File, "/tmp");
}
private static boolean copy(String from, String to) throws IOException{
Path src = Paths.get(from);
Path dest = Paths.get(to);
try {
Files.copy(src, dest, StandardCopyOption.REPLACE_EXISTING);
return true;
} catch (IOException ioe) {
System.err.format("I/O Error when copying file");
ioe.printStackTrace();
return false;
}
}
}
When I run this program I get these errors:
java -jar CopyFile.jar /home/downloads/dfA485MVSZ.ncr.pwgsc.gc.ca.1531160874.13500750
I/O Error when copying filejava.nio.file.FileSystemException: /tmp:
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:103)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:114)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:119)
at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:578)
at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:265)
at java.nio.file.Files.copy(Files.java:1285)
at ca.gc.ssc.gems.esnap.cipo.CopyFile.copy(CopyFile.java:39)
at ca.gc.ssc.gems.esnap.cipo.CopyFile.main(CopyFile.java:31)
To test your code I used C:/tmp/test.txt; as your args[0]. I fixed the issue by giving the output a filename to write to shown below:
Path dest = Paths.get(to);
to
Path dest = Paths.get(to, "test2.txt");
And it now successfully copied the file into that name, you can modify the filename however you want or add logic to change filename automatically.
I try to delete a directory using java,here is my code
public static void delDirectory(String path) throws IOException {
Path p = Paths.get(path);
delHelp(p);
}
private static void delHelp(Path p) throws IOException {
if (!p.toFile().exists()) {
return;
} else if(p.toFile().isFile()){
log.debug("delete file:" + p.toAbsolutePath().toString());
Files.delete(p);
}else if(p.toFile().isDirectory()){
for(Path subPath:Files.newDirectoryStream(p)){
delHelp(subPath);
}
log.debug("delete directory:"+p.toAbsolutePath().toString());
Files.delete(p);
}
}
On unix-like system, it works out. On windows, the code Files.delete(p) actually move the directory to the trash can, so when delete the parent directory the code will throw exception: Exception in thread "main" java.nio.file.DirectoryNotEmptyException
Any idea about this os-dependent behavior? How can I work around this?
The actual problem is that you are not closing the DirectoryStream, which is causing the DirectoryNotEmptyException when you try to delete the directory.
From the Javadoc:
When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed so as to free any resources held for the open directory.
So you can either call close() on it when you are done with it, or use it in try-with-resources:
private static void delHelp(Path p) throws IOException {
if (!p.toFile().exists()) {
return;
} else if(p.toFile().isFile()){
Files.delete(p);
} else if(p.toFile().isDirectory()){
try (DirectoryStream<Path> ds = Files.newDirectoryStream(p)) {
for (Path subPath : ds){
delHelp(subPath);
}
}
Files.delete(p);
}
}
please first of all add this Jar into your project first.
Find below code works perfectly as per your requirement too.
i.e. work on window machine and should be not goes to trash/recycle-bin
public static void main(String[] args) {
try {
delDirectory("E:\\RecursiveDataContainDirectoryName");
} catch (Exception e) {
e.printStackTrace();
}
}
public static void delDirectory(String path) throws IOException {
Path p = Paths.get(path);
FileDeleteStrategy.FORCE.delete(p.toFile());
}
I need to write a recursive algorithm to display the contents of a directory in a computer's file system but I am very new to Java. Does anyone have any code or a good tutorial on how to access a directory in a file system with Java??
You can use the JFileChooser class, check this example.
Optionally you can also execute native commands like DIR , lsusing java , here is an example
This took me way too long to write and test, but here's something that should work.
Note: You can pass in either a string or file.
Note 2: This is a naive implementation. Not only is it single-threaded, but it does not check to see if files are links, and could get stuck in an endless loop due to this.
Note 3: The lines immediately after comments can be replaced with your own implementation.
import java.io.*;
public class DirectoryRecurser {
public static void parseFile(String filePath) throws FileNotFoundException {
File file = new File(filePath);
if (file.exists()) {
parseFile(file);
} else {
throw new FileNotFoundException(file.getPath());
}
}
public static void parseFile(File file) throws FileNotFoundException {
if (file.isDirectory()) {
for(File child : file.listFiles()) {
parseFile(child);
}
} else if (file.exists()) {
// Process file here
System.out.println(file.getPath());
} else {
throw new FileNotFoundException(file.getPath());
}
}
}
Which could then be called something like this (using a Windows path, because this Workstation is using Windows):
public static void main(String[] args) {
try {
DirectoryRecurser.parseFile("D:\\raisin");
} catch (FileNotFoundException e) {
// Error handling here
System.out.println("File not found: " + e.getMessage());
}
}
In my case, this prints out:
File not found: D:\raisin
because said directory is just one I made up. Otherwise, it prints out the path to each file.
Check out Apache Commons VFS: http://commons.apache.org/vfs/
Sample:
// Locate the Jar file
FileSystemManager fsManager = VFS.getManager();
FileObject jarFile = fsManager.resolveFile( "jar:lib/aJarFile.jar" );
// List the children of the Jar file
FileObject[] children = jarFile.getChildren();
System.out.println( "Children of " + jarFile.getName().getURI() );
for ( int i = 0; i < children.length; i++ )
{
System.out.println( children[ i ].getName().getBaseName() );
}
If you need to access files on a network drive, check out JCIFS: http://jcifs.samba.org/
check this out buddy
http://java2s.com/Code/Java/File-Input-Output/Traversingallfilesanddirectoriesunderdir.htm
public class Main {
public static void main(String[] argv) throws Exception {
}
public static void visitAllDirsAndFiles(File dir) {
System.out.println(dir);
if (dir.isDirectory()) {
String[] children = dir.list();
for (int i = 0; i < children.length; i++) {
visitAllDirsAndFiles(new File(dir, children[i]));
}
}
}
}
For each file you need to check if it is a directory. If it is, you need to recurse. Here is some untested code, which should help:
public void listFiles(File f){
System.out.println(f.getAbsolutePath());
if(f.isDirectory()){
for (File i : f.listFiles()){
listFiles(i);
}
}
}
I have this function that prints the name of all the files in a directory recursively. The problem is that my code is very slow because it has to access a remote network device with every iteration.
My plan is to first load all the files from the directory recursively and then after that go through all files with the regex to filter out all the files I don't want. Is there a better solution?
public static printFnames(String sDir) {
File[] faFiles = new File(sDir).listFiles();
for (File file : faFiles) {
if (file.getName().matches("^(.*?)")) {
System.out.println(file.getAbsolutePath());
}
if (file.isDirectory()) {
printFnames(file.getAbsolutePath());
}
}
}
This is just a test. Later on I'm not going to use the code like this; instead I'm going to add the path and modification date of every file which matches an advanced regex to an array.
Assuming this is actual production code you'll be writing, then I suggest using the solution to this sort of thing that's already been solved - Apache Commons IO, specifically FileUtils.listFiles(). It handles nested directories, filters (based on name, modification time, etc).
For example, for your regex:
Collection files = FileUtils.listFiles(
dir,
new RegexFileFilter("^(.*?)"),
DirectoryFileFilter.DIRECTORY
);
This will recursively search for files matching the ^(.*?) regex, returning the results as a collection.
It's worth noting that this will be no faster than rolling your own code, it's doing the same thing - trawling a filesystem in Java is just slow. The difference is, the Apache Commons version will have no bugs in it.
In Java 8, it's a 1-liner via Files.find() with an arbitrarily large depth (eg 999) and BasicFileAttributes of isRegularFile()
public static printFnames(String sDir) {
Files.find(Paths.get(sDir), 999, (p, bfa) -> bfa.isRegularFile()).forEach(System.out::println);
}
To add more filtering, enhance the lambda, for example all jpg files modified in the last 24 hours:
(p, bfa) -> bfa.isRegularFile()
&& p.getFileName().toString().matches(".*\\.jpg")
&& bfa.lastModifiedTime().toMillis() > System.currentMillis() - 86400000
This is a very simple recursive method to get all files from a given root.
It uses the Java 7 NIO Path class.
private List<String> getFileNames(List<String> fileNames, Path dir) {
try(DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) {
for (Path path : stream) {
if(path.toFile().isDirectory()) {
getFileNames(fileNames, path);
} else {
fileNames.add(path.toAbsolutePath().toString());
System.out.println(path.getFileName());
}
}
} catch(IOException e) {
e.printStackTrace();
}
return fileNames;
}
With Java 7, a faster way to walk through a directory tree was introduced with the Paths and Files functionality. They're much faster than the "old" File way.
This would be the code to walk through and check path names with a regular expression:
public final void test() throws IOException, InterruptedException {
final Path rootDir = Paths.get("path to your directory where the walk starts");
// Walk thru mainDir directory
Files.walkFileTree(rootDir, new FileVisitor<Path>() {
// First (minor) speed up. Compile regular expression pattern only one time.
private Pattern pattern = Pattern.compile("^(.*?)");
#Override
public FileVisitResult preVisitDirectory(Path path,
BasicFileAttributes atts) throws IOException {
boolean matches = pattern.matcher(path.toString()).matches();
// TODO: Put here your business logic when matches equals true/false
return (matches)? FileVisitResult.CONTINUE:FileVisitResult.SKIP_SUBTREE;
}
#Override
public FileVisitResult visitFile(Path path, BasicFileAttributes mainAtts)
throws IOException {
boolean matches = pattern.matcher(path.toString()).matches();
// TODO: Put here your business logic when matches equals true/false
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult postVisitDirectory(Path path,
IOException exc) throws IOException {
// TODO Auto-generated method stub
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path path, IOException exc)
throws IOException {
exc.printStackTrace();
// If the root directory has failed it makes no sense to continue
return path.equals(rootDir)? FileVisitResult.TERMINATE:FileVisitResult.CONTINUE;
}
});
}
The fast way to get the content of a directory using Java 7 NIO:
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.FileSystems;
import java.nio.file.Path;
...
Path dir = FileSystems.getDefault().getPath(filePath);
DirectoryStream<Path> stream = Files.newDirectoryStream(dir);
for (Path path : stream) {
System.out.println(path.getFileName());
}
stream.close();
Java's interface for reading filesystem folder contents is not very performant (as you've discovered). JDK 7 fixes this with a completely new interface for this sort of thing, which should bring native level performance to these sorts of operations.
The core issue is that Java makes a native system call for every single file. On a low latency interface, this is not that big of a deal - but on a network with even moderate latency, it really adds up. If you profile your algorithm above, you'll find that the bulk of the time is spent in the pesky isDirectory() call - that's because you are incurring a round trip for every single call to isDirectory(). Most modern OSes can provide this sort of information when the list of files/folders was originally requested (as opposed to querying each individual file path for it's properties).
If you can't wait for JDK7, one strategy for addressing this latency is to go multi-threaded and use an ExecutorService with a maximum # of threads to perform your recursion. It's not great (you have to deal with locking of your output data structures), but it'll be a heck of a lot faster than doing this single threaded.
In all of your discussions about this sort of thing, I highly recommend that you compare against the best you could do using native code (or even a command line script that does roughly the same thing). Saying that it takes an hour to traverse a network structure doesn't really mean that much. Telling us that you can do it native in 7 second, but it takes an hour in Java will get people's attention.
This will work just fine and it’s recursive.
File root = new File("ROOT PATH");
for (File file : root.listFiles())
{
getFilesRecursive(file);
}
private static void getFilesRecursive(File pFile)
{
for(File files : pFile.listFiles())
{
if(files.isDirectory())
{
getFilesRecursive(files);
}
else
{
// Do your thing
//
// You can either save in HashMap and
// use it as per your requirement
}
}
}
I personally like this version of FileUtils. Here's an example that finds all mp3s or flacs in a directory or any of its subdirectories:
String[] types = {"mp3", "flac"};
Collection<File> files2 = FileUtils.listFiles(/path/to/your/dir, types , true);
This will work fine
public void displayAll(File path){
if(path.isFile()){
System.out.println(path.getName());
}else{
System.out.println(path.getName());
File files[] = path.listFiles();
for(File dirOrFile: files){
displayAll(dirOrFile);
}
}
}
Java 8
public static void main(String[] args) throws IOException {
Path start = Paths.get("C:\\data\\");
try (Stream<Path> stream = Files.walk(start, Integer.MAX_VALUE)) {
List<String> collect = stream
.map(String::valueOf)
.sorted()
.collect(Collectors.toList());
collect.forEach(System.out::println);
}
}
public class GetFilesRecursive {
public static List <String> getFilesRecursively(File dir){
List <String> ls = new ArrayList<String>();
for (File fObj : dir.listFiles()) {
if(fObj.isDirectory()) {
ls.add(String.valueOf(fObj));
ls.addAll(getFilesRecursively(fObj));
} else {
ls.add(String.valueOf(fObj));
}
}
return ls;
}
public static List <String> getListOfFiles(String fullPathDir) {
List <String> ls = new ArrayList<String> ();
File f = new File(fullPathDir);
if (f.exists()) {
if(f.isDirectory()) {
ls.add(String.valueOf(f));
ls.addAll(getFilesRecursively(f));
}
} else {
ls.add(fullPathDir);
}
return ls;
}
public static void main(String[] args) {
List <String> ls = getListOfFiles("/Users/srinivasab/Documents");
for (String file:ls) {
System.out.println(file);
}
System.out.println(ls.size());
}
}
This function will probably list all the file name and its path from its directory and its subdirectories.
public void listFile(String pathname) {
File f = new File(pathname);
File[] listfiles = f.listFiles();
for (int i = 0; i < listfiles.length; i++) {
if (listfiles[i].isDirectory()) {
File[] internalFile = listfiles[i].listFiles();
for (int j = 0; j < internalFile.length; j++) {
System.out.println(internalFile[j]);
if (internalFile[j].isDirectory()) {
String name = internalFile[j].getAbsolutePath();
listFile(name);
}
}
} else {
System.out.println(listfiles[i]);
}
}
}
it feels like it's stupid access the
filesystem and get the contents for
every subdirectory instead of getting
everything at once.
Your feeling is wrong. That's how filesystems work. There is no faster way (except when you have to do this repeatedly or for different patterns, you can cache all the file paths in memory, but then you have to deal with cache invalidation i.e. what happens when files are added/removed/renamed while the app runs).
Just so you know isDirectory() is quite a slow method. I'm finding it quite slow in my file browser. I'll be looking into a library to replace it with native code.
Another optimized code
import java.io.File;
import java.util.ArrayList;
import java.util.List;
public class GetFilesRecursive {
public static List <String> getFilesRecursively(File dir){
List <String> ls = new ArrayList<String>();
if (dir.isDirectory())
for (File fObj : dir.listFiles()) {
if(fObj.isDirectory()) {
ls.add(String.valueOf(fObj));
ls.addAll(getFilesRecursively(fObj));
} else {
ls.add(String.valueOf(fObj));
}
}
else
ls.add(String.valueOf(dir));
return ls;
}
public static void main(String[] args) {
List <String> ls = getFilesRecursively(new File("/Users/srinivasab/Documents"));
for (String file:ls) {
System.out.println(file);
}
System.out.println(ls.size());
}
}
One more example of listing files and directories using Java 8 filter
public static void main(String[] args) {
System.out.println("Files!!");
try {
Files.walk(Paths.get("."))
.filter(Files::isRegularFile)
.filter(c ->
c.getFileName().toString().substring(c.getFileName().toString().length()-4).contains(".jpg")
||
c.getFileName().toString().substring(c.getFileName().toString().length()-5).contains(".jpeg")
)
.forEach(System.out::println);
} catch (IOException e) {
System.out.println("No jpeg or jpg files");
}
System.out.println("\nDirectories!!\n");
try {
Files.walk(Paths.get("."))
.filter(Files::isDirectory)
.forEach(System.out::println);
} catch (IOException e) {
System.out.println("No Jpeg files");
}
}
Test folder
I tested some method with 60,000 files in 284 folders on Windows 11:
public class App {
public static void main(String[] args) throws Exception {
Path path = Paths.get("E:\\书籍");
// 1.walkFileTree
long start1 = System.currentTimeMillis();
Files.walkFileTree(path, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {
// if(pathMatcher.matches(file))
// files.add(file.toFile());
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
// System.out.println(dir.getFileName());
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path file, IOException e) {
return FileVisitResult.CONTINUE;
}
});
long end1 = System.currentTimeMillis();
// 2. newDirectoryStream
long start2 = System.currentTimeMillis();
search(path.toFile());
long end2 = System.currentTimeMillis();
// 3. listFiles
long start3 = System.currentTimeMillis();
getFileNames(path);
long end3 = System.currentTimeMillis();
System.out.println("\r执行耗时:" + (end1 - start1));
System.out.println("\r执行耗时:" + (end2 - start2));
System.out.println("\r执行耗时:" + (end3 - start3));
}
private static void getFileNames(Path dir) {
try(DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) {
for (Path path : stream) {
if(Files.isDirectory(path)) {
getFileNames(path);
}
}
} catch(IOException e) {
e.printStackTrace();
}
}
public static void search(File file) {
Queue<File> q = new LinkedList<>();
q.offer(file);
while (!q.isEmpty()) {
try {
for (File childfile : q.poll().listFiles()) {
// System.out.println(childfile.getName());
if (childfile.isDirectory()) {
q.offer(childfile);
}
}
} catch (Exception e) {
}
}
}
}
Result (milliseconds)：
walkFileTree
listFiles
newDirectoryStream
68
451
493
64
464
482
61
478
457
67
477
488
59
474
466
Known performance issues：
From Kevin Day's answer:
If you profile your algorithm above, you'll find that the bulk of the time is spent in the pesky isDirectory() call - that's because you are incurring a round trip for every single call to isDirectory().
listfiles() will create new File Object for every entry
In Guava you don't have to wait for a Collection to be returned to you, but can actually iterate over the files. It is easy to imagine a IDoSomethingWithThisFile interface in the signature of the below function:
public static void collectFilesInDir(File dir) {
TreeTraverser<File> traverser = Files.fileTreeTraverser();
FluentIterable<File> filesInPostOrder = traverser.preOrderTraversal(dir);
for (File f: filesInPostOrder)
System.out.printf("File: %s\n", f.getPath());
}
TreeTraverser also allows you to between various traversal styles.
import java.io.*;
public class MultiFolderReading {
public void checkNoOfFiles (String filename) throws IOException {
File dir = new File(filename);
File files[] = dir.listFiles(); // Files array stores the list of files
for(int i=0; i<files.length; i++)
{
if(files[i].isFile()) // Check whether files[i] is file or directory
{
System.out.println("File::" + files[i].getName());
System.out.println();
}
else if(files[i].isDirectory())
{
System.out.println("Directory::" + files[i].getName());
System.out.println();
checkNoOfFiles(files[i].getAbsolutePath());
}
}
}
public static void main(String[] args) throws IOException {
MultiFolderReading mf = new MultiFolderReading();
String str = "E:\\file";
mf.checkNoOfFiles(str);
}
}
The more efficient way I found in dealing with millions of folders and files is to capture a directory listing through a DOS command in some file and parse it.
Once you have parsed the data then you can do analysis and compute statistics.