I tried to write code for counting files of certain type on my computer.
I tested both one thread solution and multi-threads asynch solution, and it seems like the one thread is working faster. Is anything wrong with my code? and if not, why isn't it working faster?
The code below:
AsynchFileCounter - The asynchronized version.
ExtensionFilter - The file filter to list only directories and files with the extension specified
BasicFileCounter - The one thread version.
public class AsynchFileCounter {
public int countFiles(String path, String extension) throws InterruptedException, ExecutionException {
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
return countFilesRecursive(f, filter);
}
private int countFilesRecursive(File f, ExtensionFilter filter) throws InterruptedException, ExecutionException {
return CompletableFuture.supplyAsync(() -> f.listFiles(filter))
.thenApplyAsync(files -> {
int count = 0;
for (File file : files) {
if(file.isFile())
count++;
else
try {
count += countFilesRecursive(file, filter);
} catch (Exception e) {
e.printStackTrace();
}
}
return count;
}).get();
}
}
public class ExtensionFilter implements FileFilter {
private String extension;
private boolean allowDirectories;
public ExtensionFilter(String extension, boolean allowDirectories) {
if(extension.startsWith("."))
extension = extension.substring(1);
this.extension = extension;
this.allowDirectories = allowDirectories;
}
#Override
public boolean accept(File pathname) {
if(pathname.isFile() && pathname.getName().endsWith("." + extension))
return true;
if(allowDirectories) {
if(pathname.isDirectory())
return true;
}
return false;
}
}
public class BasicFileCounter {
public int countFiles(String path, String extension) {
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
return countFilesRecursive(f, filter);
}
private int countFilesRecursive(File f, ExtensionFilter filter) {
int count = 0;
File [] ar = f.listFiles(filter);
for (File file : ar) {
if(file.isFile())
count++;
else
count += countFilesRecursive(file, filter);
}
return count;
}
}
You have to spawn multiple asynchronous jobs and must not wait immediately for their completion:
public int countFiles(String path, String extension) {
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
return countFilesRecursive(f, filter).join();
}
private CompletableFuture<Integer> countFilesRecursive(File f, FileFilter filter) {
return CompletableFuture.supplyAsync(() -> f.listFiles(filter))
.thenCompose(files -> {
if(files == null) return CompletableFuture.completedFuture(0);
int count = 0;
CompletableFuture<Integer> fileCount = new CompletableFuture<>(), all=fileCount;
for (File file : files) {
if(file.isFile())
count++;
else
all = countFilesRecursive(file, filter).thenCombine(all, Integer::sum);
}
fileCount.complete(count);
return all;
});
}
Note that File.listFiles may return null.
This code will count all files of a directory immediately but launch a new asynchronous job for sub-directories. The results of the sub-directory jobs are combined via thenCombine, to sum their results. For simplification, we create another CompletableFuture, fileCount to represent the locally counted files. thenCompose returns a future which will be completed with the result of the future returned by the specified function, so the caller can use join() to wait for the final result of the entire operation.
For I/O operations, it may help to use a different thread pool, as the default ForkJoinPool is configured to utilize the CPU cores rather the I/O bandwidth:
public int countFiles(String path, String extension) {
ExecutorService es = Executors.newFixedThreadPool(30);
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
int count = countFilesRecursive(f, filter, es).join();
es.shutdown();
return count;
}
private CompletableFuture<Integer> countFilesRecursive(File f,FileFilter filter,Executor e){
return CompletableFuture.supplyAsync(() -> f.listFiles(filter), e)
.thenCompose(files -> {
if(files == null) return CompletableFuture.completedFuture(0);
int count = 0;
CompletableFuture<Integer> fileCount = new CompletableFuture<>(), all=fileCount;
for (File file : files) {
if(file.isFile())
count++;
else
all = countFilesRecursive(file, filter,e).thenCombine(all,Integer::sum);
}
fileCount.complete(count);
return all;
});
}
There is no best number of threads, this depends on the actual execution environment and would be subject to measuring and tuning. When the application is supposed to run in different environments, this should be a configurable parameter.
But consider that you might be using the wrong tool for the job. An alternative are Fork/Join tasks, which support interacting with the thread pool to determine the current saturation, so once all worker threads are busy, it will proceed scanning locally with an ordinary recursion rather than submitting more asynchronous jobs:
public int countFiles(String path, String extension) {
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
return POOL.invoke(new FileCountTask(f, filter));
}
private static final int TARGET_SURPLUS = 3, TARGET_PARALLELISM = 30;
private static final ForkJoinPool POOL = new ForkJoinPool(TARGET_PARALLELISM);
static final class FileCountTask extends RecursiveTask<Integer> {
private final File path;
private final FileFilter filter;
public FileCountTask(File file, FileFilter ff) {
this.path = file;
this.filter = ff;
}
#Override
protected Integer compute() {
return scan(path, filter);
}
private static int scan(File directory, FileFilter filter) {
File[] fileList = directory.listFiles(filter);
if(fileList == null || fileList.length == 0) return 0;
List<FileCountTask> recursiveTasks = new ArrayList<>();
int count = 0;
for(File file: fileList) {
if(file.isFile()) count++;
else {
if(getSurplusQueuedTaskCount() < TARGET_SURPLUS) {
FileCountTask task = new FileCountTask(file, filter);
recursiveTasks.add(task);
task.fork();
}
else count += scan(file, filter);
}
}
for(int ix = recursiveTasks.size() - 1; ix >= 0; ix--) {
FileCountTask task = recursiveTasks.get(ix);
if(task.tryUnfork()) task.complete(scan(task.path, task.filter));
}
for(FileCountTask task: recursiveTasks) {
count += task.join();
}
return count;
}
}
I figured it out. since I am adding up the results in this line:
count += countFilesRecursive(file, filter);
and using get() to receive the result, I am actually waiting for the result, instead of really parallelising the code.
This is my current code, which actually runs much faster than the one thread code. However, I could not figure out an elegant way of knowing when the parallel method is done.
I would love to hear how should I solve that?
Here's the ugly way I am using:
public class AsynchFileCounter {
private LongAdder count;
public int countFiles(String path, String extension) {
count = new LongAdder();
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
countFilesRecursive(f, filter);
// ******** The way I check whether The function is done **************** //
int prev = 0;
int cur = 0;
do {
prev = cur;
try {
Thread.sleep(50);
} catch (InterruptedException e) {}
cur = (int)count.sum();
} while(cur>prev);
// ******************************************************************** //
return count.intValue();
}
private void countFilesRecursive(File f, ExtensionFilter filter) {
CompletableFuture.supplyAsync(() -> f.listFiles(filter))
.thenAcceptAsync(files -> {
for (File file : files) {
if(file.isFile())
count.increment();
else
countFilesRecursive(file, filter);
}
});
}
}
I did some changes to the code:
I use AtomicInteger to count the files instead of the LongAdder.
After reading Holger's answer, I decided to count directories being processed. When the number goes down to zero, the work is done. So I added a lock and a condition to let the main thread know when the work is done.
I added a check whether the file.listFiles() returns a null. I ran the code on windows and it never did (I had an empty directory, and it returned an empty array), but since it is using native code, it might return null on other OS.
public class AsynchFileCounter {
private AtomicInteger count;
private AtomicInteger countDirectories;
private ReentrantLock lock;
private Condition noMoreDirectories;
public int countFiles(String path, String extension) {
count = new AtomicInteger();
countDirectories = new AtomicInteger();
lock = new ReentrantLock();
noMoreDirectories = lock.newCondition();
ExtensionFilter filter = new ExtensionFilter(extension, true);
File f = new File(path);
countFilesRecursive(f, filter);
lock.lock();
try {
noMoreDirectories.await();
} catch (InterruptedException e) {}
finally {
lock.unlock();
}
return count.intValue();
}
private void countFilesRecursive(File f, ExtensionFilter filter) {
countDirectories.getAndIncrement();
CompletableFuture.supplyAsync(() -> f.listFiles(filter))
.thenAcceptAsync(files -> countFiles(filter, files));
}
private void countFiles(ExtensionFilter filter, File[] files) {
if(files != null) {
for (File file : files) {
if(file.isFile())
count.incrementAndGet();
else
countFilesRecursive(file, filter);
}
}
int currentCount = countDirectories.decrementAndGet();
if(currentCount == 0) {
lock.lock();
try {
noMoreDirectories.signal();
}
finally {
lock.unlock();
}
}
}
}
I extended FileSystemView and overwrote every method in this class. The model looks like this:
public class RemoteSystemFilesView extends FileSystemView {
private IDirectoryService directoryService;
public RemoteSystemFilesView(IDirectoryService aDirectoryService){
this.directoryService = aDirectoryService;
}
....
}
The directoryService object returns directories from the remote UNIX server. Then, I create JFileChooser.
JFileChooser fc = new JFileChooser(new RemoteSystemFilesView(new DirectoryService()));
int returnVal = fc.showOpenDialog(this);
The dialog shows remote dirs and files correctly, but then I doubleClick on one of the displayed folders, I expect to navigate into that folder, but instead folder path appears in the field "File name" and that's it. I can't go to any other directory except root (/). Should I implement something else also in JFileChooser, not just in FileSystemView?
The problem might be that your FileSystemView is actually returning plain java.io.File objects.
Instead try to return a VirtualFile wrapper object that extends java.io.File and returns true for the public boolean exists() and wraps returns VirtualFile instead of java.io.File for all the necessary methods.
This is an example of a VirtualFileSystem that I developed. It uses java.nio.Path because my code is mainly based in them. I hope it gives you a good starting point for understanding how to modify your code.
private static class VirtualFileSystemView extends FileSystemView {
final Path base;
final Set<Path> choices;
private VirtualFileSystemView(final Path base,
final Set<Path> choices) {
this.base = base;
this.choices = choices;
}
#Override
protected File createFileSystemRoot(File f) {
return new VirtualFile(f);
}
#Override
public boolean isComputerNode(File dir) {
return false;
}
#Override
public boolean isFloppyDrive(File dir) {
return false;
}
#Override
public boolean isDrive(File dir) {
return false;
}
#Override
public Icon getSystemIcon(File f) {
return null;
}
#Override
public String getSystemTypeDescription(File f) {
return f.toPath().toString();
}
#Override
public String getSystemDisplayName(File f) {
return f.getName();
}
#Override
public File getParentDirectory(final File dir) {
return new VirtualFile(dir.getParentFile());
}
#Override
public File[] getFiles(final File dir, boolean useFileHiding) {
final List<File> files = new ArrayList<>(choices.size());
choices.stream()
.filter((path) -> (path.getParent().equals(dir.toPath()))).
forEach((path) -> {
files.add(new VirtualFile(path.toFile()));
});
return files.toArray(new File[files.size()]);
}
#Override
public File createFileObject(final String path) {
return new VirtualFile(path);
}
#Override
public File createFileObject(final File dir, final String filename) {
Path fileObject;
if (dir != null) {
fileObject = Paths.get(dir.toPath().toString(), filename);
} else {
fileObject = Paths.get(filename);
}
return new VirtualFile(fileObject.toFile());
}
#Override
public File getDefaultDirectory() {
return new VirtualFile(base.toFile());
}
#Override
public File getHomeDirectory() {
return new VirtualFile(base.toFile());
}
#Override
public File[] getRoots() {
final List<File> files = new ArrayList<>(choices.size());
files.add(new VirtualFile(base.toFile()));
return files.toArray(new File[files.size()]);
}
#Override
public boolean isFileSystemRoot(final File dir) {
boolean isRoot = dir.toPath().getParent() == null;
return isRoot;
}
#Override
public boolean isHiddenFile(final File f) {
return false;
}
#Override
public boolean isFileSystem(final File f) {
return !isFileSystemRoot(f);
}
#Override
public File getChild(final File parent, final String fileName) {
return new VirtualFile(parent, fileName);
}
#Override
public boolean isParent(final File folder, final File file) {
return file.toPath().getParent().equals(folder.toPath());
}
#Override
public Boolean isTraversable(final File f) {
boolean isTraversable = false;
for (final Path path : choices) {
if (path.startsWith(f.toPath())) {
isTraversable = true;
break;
}
}
return isTraversable;
}
#Override
public boolean isRoot(final File f) {
boolean isRoot = false;
for (final Path path : choices) {
if (path.getParent().equals(f.toPath())) {
isRoot = true;
}
}
return isRoot;
}
#Override
public File createNewFolder(final File containingDir) throws IOException {
return new VirtualFile(containingDir);
}
private class VirtualFile extends File {
private static final long serialVersionUID = -1752685357864733168L;
private VirtualFile(final File file) {
super(file.toString());
}
private VirtualFile(String pathname) {
super(pathname);
}
private VirtualFile(String parent, String child) {
super(parent, child);
}
private VirtualFile(File parent, String child) {
super(parent, child);
}
#Override
public boolean exists() {
return true;
}
#Override
public boolean isDirectory() {
return VirtualFileSystemView.this.isTraversable(this);
}
#Override
public File getCanonicalFile() throws IOException {
return new VirtualFile(super.getCanonicalFile());
}
#Override
public File getAbsoluteFile() {
return new VirtualFile(super.getAbsoluteFile());
}
#Override
public File getParentFile() {
File parent = super.getParentFile();
if (parent != null) {
parent = new VirtualFile(super.getParentFile());
}
return parent;
}
}
}
How do I recursively list all files under a directory in Java? Does the framework provide any utility?
I saw a lot of hacky implementations. But none from the framework or nio
Java 8 provides a nice stream to process all files in a tree.
try (Stream<Path> stream = Files.walk(Paths.get(path))) {
stream.filter(Files::isRegularFile)
.forEach(System.out::println);
}
This provides a natural way to traverse files. Since it's a stream you can do all nice stream operations on the result such as limit, grouping, mapping, exit early etc.
UPDATE: I might point out there is also Files.find which takes a BiPredicate that could be more efficient if you need to check file attributes.
Files.find(Paths.get(path),
Integer.MAX_VALUE,
(filePath, fileAttr) -> fileAttr.isRegularFile())
.forEach(System.out::println);
Note that while the JavaDoc eludes that this method could be more efficient than Files.walk it is effectively identical, the difference in performance can be observed if you are also retrieving file attributes within your filter. In the end, if you need to filter on attributes use Files.find, otherwise use Files.walk, mostly because there are overloads and it's more convenient.
TESTS: As requested I've provided a performance comparison of many of the answers. Check out the Github project which contains results and a test case.
FileUtils have iterateFiles and listFiles methods. Give them a try. (from commons-io)
Edit: You can check here for a benchmark of different approaches. It seems that the commons-io approach is slow, so pick some of the faster ones from here (if it matters)
// Ready to run
import java.io.File;
public class Filewalker {
public void walk( String path ) {
File root = new File( path );
File[] list = root.listFiles();
if (list == null) return;
for ( File f : list ) {
if ( f.isDirectory() ) {
walk( f.getAbsolutePath() );
System.out.println( "Dir:" + f.getAbsoluteFile() );
}
else {
System.out.println( "File:" + f.getAbsoluteFile() );
}
}
}
public static void main(String[] args) {
Filewalker fw = new Filewalker();
fw.walk("c:\\" );
}
}
Java 7 will have has Files.walkFileTree:
If you provide a starting point and a file visitor, it will invoke various methods on the file visitor as it walks through the file in the file tree. We expect people to use this if they are developing a recursive copy, a recursive move, a recursive delete, or a recursive operation that sets permissions or performs another operation on each of the files.
There is now an entire Oracle tutorial on this question.
No external libraries needed.
Returns a Collection so you can do whatever you want with it after the call.
public static Collection<File> listFileTree(File dir) {
Set<File> fileTree = new HashSet<File>();
if(dir==null||dir.listFiles()==null){
return fileTree;
}
for (File entry : dir.listFiles()) {
if (entry.isFile()) fileTree.add(entry);
else fileTree.addAll(listFileTree(entry));
}
return fileTree;
}
I would go with something like:
public void list(File file) {
System.out.println(file.getName());
File[] children = file.listFiles();
for (File child : children) {
list(child);
}
}
The System.out.println is just there to indicate to do something with the file. there is no need to differentiate between files and directories, since a normal file will simply have zero children.
I prefer using a queue over recursion for this kind of simple traversion:
List<File> allFiles = new ArrayList<File>();
Queue<File> dirs = new LinkedList<File>();
dirs.add(new File("/start/dir/"));
while (!dirs.isEmpty()) {
for (File f : dirs.poll().listFiles()) {
if (f.isDirectory()) {
dirs.add(f);
} else if (f.isFile()) {
allFiles.add(f);
}
}
}
just write it yourself using simple recursion:
public List<File> addFiles(List<File> files, File dir)
{
if (files == null)
files = new LinkedList<File>();
if (!dir.isDirectory())
{
files.add(dir);
return files;
}
for (File file : dir.listFiles())
addFiles(files, file);
return files;
}
With Java 7 you can use the following class:
import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
public class MyFileIterator extends SimpleFileVisitor<Path>
{
public MyFileIterator(String path) throws Exception
{
Files.walkFileTree(Paths.get(path), this);
}
#Override
public FileVisitResult visitFile(Path file,
BasicFileAttributes attributes) throws IOException
{
System.out.println("File: " + file);
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult preVisitDirectory(Path dir,
BasicFileAttributes attributes) throws IOException
{
System.out.println("Dir: " + dir);
return FileVisitResult.CONTINUE;
}
}
This code is ready to run
public static void main(String... args) {
File[] files = new File("D:/").listFiles();
if (files != null)
getFiles(files);
}
public static void getFiles(File[] files) {
for (File file : files) {
if (file.isDirectory()) {
getFiles(file.listFiles());
} else {
System.out.println("File: " + file);
}
}
}
I think this should do the work:
File dir = new File(dirname);
String[] files = dir.list();
This way you have files and dirs. Now use recursion and do the same for dirs (File class has isDirectory() method).
In Java 8, we can now use the Files utility to walk a file tree. Very simple.
Files.walk(root.toPath())
.filter(path -> !Files.isDirectory(path))
.forEach(path -> System.out.println(path));
Apart from the recursive traversal one can use a Visitor based approach as well.
Below code is uses Visitor based approach for the traversal.It is expected that the input to the program is the root directory to traverse.
public interface Visitor {
void visit(DirElement d);
void visit(FileElement f);
}
public abstract class Element {
protected File rootPath;
abstract void accept(Visitor v);
#Override
public String toString() {
return rootPath.getAbsolutePath();
}
}
public class FileElement extends Element {
FileElement(final String path) {
rootPath = new File(path);
}
#Override
void accept(final Visitor v) {
v.visit(this);
}
}
public class DirElement extends Element implements Iterable<Element> {
private final List<Element> elemList;
DirElement(final String path) {
elemList = new ArrayList<Element>();
rootPath = new File(path);
for (File f : rootPath.listFiles()) {
if (f.isDirectory()) {
elemList.add(new DirElement(f.getAbsolutePath()));
} else if (f.isFile()) {
elemList.add(new FileElement(f.getAbsolutePath()));
}
}
}
#Override
void accept(final Visitor v) {
v.visit(this);
}
public Iterator<Element> iterator() {
return elemList.iterator();
}
}
public class ElementWalker {
private final String rootDir;
ElementWalker(final String dir) {
rootDir = dir;
}
private void traverse() {
Element d = new DirElement(rootDir);
d.accept(new Walker());
}
public static void main(final String[] args) {
ElementWalker t = new ElementWalker("C:\\temp");
t.traverse();
}
private class Walker implements Visitor {
public void visit(final DirElement d) {
System.out.println(d);
for(Element e:d) {
e.accept(this);
}
}
public void visit(final FileElement f) {
System.out.println(f);
}
}
}
You can use below code to get a list of files of specific folder or directory recursively.
public static void main(String args[]) {
recusiveList("D:");
}
public static void recursiveList(String path) {
File f = new File(path);
File[] fl = f.listFiles();
for (int i = 0; i < fl.length; i++) {
if (fl[i].isDirectory() && !fl[i].isHidden()) {
System.out.println(fl[i].getAbsolutePath());
recusiveList(fl[i].getAbsolutePath());
} else {
System.out.println(fl[i].getName());
}
}
}
I came up with this for printing all the files/file names recursively.
private static void printAllFiles(String filePath,File folder) {
if(filePath==null) {
return;
}
File[] files = folder.listFiles();
for(File element : files) {
if(element.isDirectory()) {
printAllFiles(filePath,element);
} else {
System.out.println(" FileName "+ element.getName());
}
}
}
Lists all files with provided extensions,with option to scan
subfolders (recursive)
public static ArrayList<File> listFileTree(File dir,boolean recursive) {
if (null == dir || !dir.isDirectory()) {
return new ArrayList<>();
}
final Set<File> fileTree = new HashSet<File>();
FileFilter fileFilter = new FileFilter() {
private final String[] acceptedExtensions = new String[]{"jpg", "png", "webp", "jpeg"};
#Override
public boolean accept(File file) {
if (file.isDirectory()) {
return true;
}
for (String extension : acceptedExtensions) {
if (file.getName().toLowerCase().endsWith(extension)) {
return true;
}
}
return false;
}
};
File[] listed = dir.listFiles(fileFilter);
if(listed!=null){
for (File entry : listed) {
if (entry.isFile()) {
fileTree.add(entry);
} else if(recursive){
fileTree.addAll(listFileTree(entry,true));
}
}
}
return new ArrayList<>(fileTree);
}
List<Path> filePaths = Files
.find(Paths.get(dir), Integer.MAX_VALUE, (filePath, fileAttr) -> fileAttr.isRegularFile() || fileAttr.isDirectory())
.collect(Collectors.toList());
filePaths will have files and folder list which can be iterated and proceed further.
Non-recursive BFS with a single list (particular example is searching for *.eml files):
final FileFilter filter = new FileFilter() {
#Override
public boolean accept(File file) {
return file.isDirectory() || file.getName().endsWith(".eml");
}
};
// BFS recursive search
List<File> queue = new LinkedList<File>();
queue.addAll(Arrays.asList(dir.listFiles(filter)));
for (ListIterator<File> itr = queue.listIterator(); itr.hasNext();) {
File file = itr.next();
if (file.isDirectory()) {
itr.remove();
for (File f: file.listFiles(filter)) itr.add(f);
}
}
My version (of course I could have used the built in walk in Java 8 ;-) ):
public static List<File> findFilesIn(File rootDir, Predicate<File> predicate) {
ArrayList<File> collected = new ArrayList<>();
walk(rootDir, predicate, collected);
return collected;
}
private static void walk(File dir, Predicate<File> filterFunction, List<File> collected) {
Stream.of(listOnlyWhenDirectory(dir))
.forEach(file -> walk(file, filterFunction, addAndReturn(collected, file, filterFunction)));
}
private static File[] listOnlyWhenDirectory(File dir) {
return dir.isDirectory() ? dir.listFiles() : new File[]{};
}
private static List<File> addAndReturn(List<File> files, File toAdd, Predicate<File> filterFunction) {
if (filterFunction.test(toAdd)) {
files.add(toAdd);
}
return files;
}
Here a simple but perfectly working solution using recursion:
public static List<Path> listFiles(String rootDirectory)
{
List<Path> files = new ArrayList<>();
listFiles(rootDirectory, files);
return files;
}
private static void listFiles(String path, List<Path> collectedFiles)
{
File root = new File(path);
File[] files = root.listFiles();
if (files == null)
{
return;
}
for (File file : files)
{
if (file.isDirectory())
{
listFiles(file.getAbsolutePath(), collectedFiles);
} else
{
collectedFiles.add(file.toPath());
}
}
}
private void fillFilesRecursively(File file, List<File> resultFiles) {
if (file.isFile()) {
resultFiles.add(file);
} else {
for (File child : file.listFiles()) {
fillFilesRecursively(child, resultFiles);
}
}
}
Kotlin has FileTreeWalk for this purpose. For example:
dataDir.walkTopDown().filter { !it.isDirectory }.joinToString("\n") {
"${it.toRelativeString(dataDir)}: ${it.length()}"
}
Will produce a text list of all the non-directory files under a given root, one file per line with the path relative to the root and length.
The accepted answer is great, however it breaks down when you want to do IO inside the lambda.
Here is what you can do if your action declares IOExceptions.
You can treat the filtered stream as an Iterable, and then do your action in a regular for-each loop. This way, you don't have to handle exceptions inside a lambda.
try (Stream<Path> pathStream = Files.walk(Paths.get(path))
.filter(Files::isRegularFile)) {
for (Path file : (Iterable<Path>) pathStream::iterator) {
// something that throws IOException
Files.copy(file, System.out);
}
}
Found that trick here: https://stackoverflow.com/a/32668807/1207791
Another way you can do even if someone already provide Java 8 walk.
This one will provide you all files recursively
private Stream<File> files(File file) {
return file.isDirectory()
? Arrays.stream(file.listFiles()).flatMap(this::files)
: Stream.of(file);
}
public static String getExten(String path) {
int i = path.lastIndexOf('.');
if (i > 0) {
return path.substring(i);
}
else return "";
}
public static List<String> GetAllFiles(String path, List<String>fileList){
File file = new File(path);
File[] files = file.listFiles();
for(File folder:files) {
if(extensions.contains(getExten(folder.getPath()))) {
fileList.add(folder.getPath());
}
}
File[] direcs = file.listFiles(File::isDirectory);
for(File dir:direcs) {
GetAllFiles(dir.getPath(),fileList);
}
return fileList;
}
This is a simple recursive function that should give you all the files. extensions is a list of string that contains only those extensions which are accepted. Example extensions = [".txt",".docx"] etc.
The accepted answer is poor because it can result in a resource leak.
Files.walk is backed by DirectoryStreams.
The returned stream encapsulates one or more DirectoryStreams. If timely disposal of file system resources is required, the try-with-resources construct should be used to ensure that the stream's close method is invoked after the stream operations are completed. Operating on a closed stream will result in an IllegalStateException.
A DirectoryStream must be closed as specified in it's javadoc:
A DirectoryStream is opened upon creation and is closed by invoking the close method. Closing a directory stream releases any resources associated with the stream. Failure to close the stream may result in a resource leak. The try-with-resources statement provides a useful construct to ensure that the stream is closed:
Path dir = ...
try (DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) {
for (Path entry: stream) {
...
}
}
As a result, the true answer is:
try (Stream<Path> stream = Files.walk(Paths.get(path))) {
// Do something with the stream.
stream.filter(Files::isRegularFile)
.forEach(System.out::println);
}
Example outputs *.csv files in directory recursive searching Subdirectories using Files.find() from java.nio:
String path = "C:/Daten/ibiss/ferret/";
logger.debug("Path:" + path);
try (Stream<Path> fileList = Files.find(Paths.get(path), Integer.MAX_VALUE,
(filePath, fileAttr) -> fileAttr.isRegularFile() && filePath.toString().endsWith("csv"))) {
List<String> someThingNew = fileList.sorted().map(String::valueOf).collect(Collectors.toList());
for (String t : someThingNew) {
t.toString();
logger.debug("Filename:" + t);
}
}
Posting this example, as I had trouble understanding howto pass the filename parameter in the #1 example given by Bryan, using foreach on Stream-result -
Hope this helps.
base on #Michael answer, add check whether listFiles return null
static Stream<File> files(File file) {
return file.isDirectory()
? Optional.ofNullable(file.listFiles()).map(Stream::of).orElseGet(Stream::empty).flatMap(MainActivity::files)
: Stream.of(file);
}
or use Lightweight-Stream-API, which support Android5 & Android6
static Stream<File> files(File f) {
return f.isDirectory() ? Stream.ofNullable(f.listFiles()).flatMap(MainActivity::files) : Stream.of(f);
}
Based on stacker answer. Here is a solution working in JSP without any external libraries so you can put it almost anywhere on your server:
<!DOCTYPE html>
<%# page session="false" %>
<%# page import="java.util.*" %>
<%# page import="java.io.*" %>
<%# page contentType="text/html; charset=UTF-8" %>
<%!
public List<String> files = new ArrayList<String>();
/**
Fills files array with all sub-files.
*/
public void walk( File root ) {
File[] list = root.listFiles();
if (list == null) return;
for ( File f : list ) {
if ( f.isDirectory() ) {
walk( f );
}
else {
files.add(f.getAbsolutePath());
}
}
}
%>
<%
files.clear();
File jsp = new File(request.getRealPath(request.getServletPath()));
File dir = jsp.getParentFile();
walk(dir);
String prefixPath = dir.getAbsolutePath() + "/";
%>
Then you just do something like:
<ul>
<% for (String file : files) { %>
<% if (file.matches(".+\\.(apk|ipa|mobileprovision)")) { %>
<li><%=file.replace(prefixPath, "")%></li>
<% } %>
<% } %>
</ul>
import java.io.File;
public class Main {
public static void main(String[] args) {
loopFiles(new File("C:\\Users\\serge.klimkovitch\\Documents"));
}
private static void loopFiles(File element) {
if (element.isDirectory()) {
for (File currentFile : element.listFiles()) {
loopFiles(currentFile);
System.out.println(currentFile);
}
}
}
}
How can I get list all the files within a folder recursively in Java?
Not sure how you want to represent the tree? Anyway here's an example which scans the entire subtree using recursion. Files and directories are treated alike. Note that File.listFiles() returns null for non-directories.
public static void main(String[] args) {
Collection<File> all = new ArrayList<File>();
addTree(new File("."), all);
System.out.println(all);
}
static void addTree(File file, Collection<File> all) {
File[] children = file.listFiles();
if (children != null) {
for (File child : children) {
all.add(child);
addTree(child, all);
}
}
}
Java 7 offers a couple of improvements. For example, DirectoryStream provides one result at a time - the caller no longer has to wait for all I/O operations to complete before acting. This allows incremental GUI updates, early cancellation, etc.
static void addTree(Path directory, Collection<Path> all)
throws IOException {
try (DirectoryStream<Path> ds = Files.newDirectoryStream(directory)) {
for (Path child : ds) {
all.add(child);
if (Files.isDirectory(child)) {
addTree(child, all);
}
}
}
}
Note that the dreaded null return value has been replaced by IOException.
Java 7 also offers a tree walker:
static void addTree(Path directory, final Collection<Path> all)
throws IOException {
Files.walkFileTree(directory, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs)
throws IOException {
all.add(file);
return FileVisitResult.CONTINUE;
}
});
}
import java.io.File;
public class Test {
public static void main( String [] args ) {
File actual = new File(".");
for( File f : actual.listFiles()){
System.out.println( f.getName() );
}
}
}
It displays indistinctly files and folders.
See the methods in File class to order them or avoid directory print etc.
http://java.sun.com/javase/6/docs/api/java/io/File.html
You can also use the FileFilter interface to filter out what you want. It is best used when you create an anonymous class that implements it:
import java.io.File;
import java.io.FileFilter;
public class ListFiles {
public File[] findDirectories(File root) {
return root.listFiles(new FileFilter() {
public boolean accept(File f) {
return f.isDirectory();
}});
}
public File[] findFiles(File root) {
return root.listFiles(new FileFilter() {
public boolean accept(File f) {
return f.isFile();
}});
}
}
public static void directory(File dir) {
File[] files = dir.listFiles();
for (File file : files) {
System.out.println(file.getAbsolutePath());
if (file.listFiles() != null)
directory(file);
}
}
Here dir is Directory to be scanned. e.g. c:\
Visualizing the tree structure was the most convenient way for me :
public static void main(String[] args) throws IOException {
printTree(0, new File("START/FROM/DIR"));
}
static void printTree(int depth, File file) throws IOException {
StringBuilder indent = new StringBuilder();
String name = file.getName();
for (int i = 0; i < depth; i++) {
indent.append(".");
}
//Pretty print for directories
if (file.isDirectory()) {
System.out.println(indent.toString() + "|");
if(isPrintName(name)){
System.out.println(indent.toString() + "*" + file.getName() + "*");
}
}
//Print file name
else if(isPrintName(name)) {
System.out.println(indent.toString() + file.getName());
}
//Recurse children
if (file.isDirectory()) {
File[] files = file.listFiles();
for (int i = 0; i < files.length; i++){
printTree(depth + 4, files[i]);
}
}
}
//Exclude some file names
static boolean isPrintName(String name){
if (name.charAt(0) == '.') {
return false;
}
if (name.contains("svn")) {
return false;
}
//.
//. Some more exclusions
//.
return true;
}
In JDK7, "more NIO features" should have methods to apply the visitor pattern over a file tree or just the immediate contents of a directory - no need to find all the files in a potentially huge directory before iterating over them.