Use Java to build an index of files in Windows 10 - java

I am writing a desktop application in Java to quickly find files. I have used the exec command in Java to run powershell to do this, as Java's os.walk method seems to be much slower. Right now it takes about 5 minutes to generate a text file that lists the contents of all files on my computer (a total of around 440,000 files).
This is fine, but the problem I have is that I have no way of updating this list of files. So if I change a few files in my file system and want to update my file list, I can't do so quickly (i.e. incrementally). Instead, I have to generate the file list all over from scratch.
I know you can use git-bash to create a locate database (using updatedb). Now this is an awesome solution, but the application I'm trying to create may be used by people who don't have that installed. So I'd like to do it using default apps provided with Windows (i.e. powershell, or natively in Java). I am trying to make this app easy to use, so I don't want the user to have to install a bunch of other dependencies.

The following code shows how to use Java and avoid Powershell altogether. It builds an array in memory and writes it to a text file (467,000 files listed) all in under 30 seconds!
Run the following code in Main or wherever you want. It calls the createFileList method.
List<Path> pathsArrayList = new ArrayList<>();
Path rootPath_obj;
rootPath_obj = Paths.get(this.configMap.get("root_path"));
createFileList(rootPath_obj);
Here's the tree stream traversal code:
public void createFileList(Path path_in) throws IOException, AccessDeniedException {
try (DirectoryStream<Path> mystream = Files.newDirectoryStream(path_in)) {
for (Path entry : mystream) {
if (Files.isDirectory(entry)) {
createFileList(entry);
}
pathsArrayList.add(entry);
}
}
catch (AccessDeniedException ex) {
// Do nothing, just move on to the next file
}
}
Now write the file to save for later. This is the listing of all files within the root path tree.
System.out.println("Writing database...");
try (FileWriter writer = new FileWriter(this.configMap.get("db_path"))) {
for(Path pth: pathsArrayList){
writer.write(pth.toString() + System.lineSeparator());
}
}
System.out.println("...database written.");

Related

How to change the user.dir in java 8? [duplicate]

How can I change the current working directory from within a Java program? Everything I've been able to find about the issue claims that you simply can't do it, but I can't believe that that's really the case.
I have a piece of code that opens a file using a hard-coded relative file path from the directory it's normally started in, and I just want to be able to use that code from within a different Java program without having to start it from within a particular directory. It seems like you should just be able to call System.setProperty( "user.dir", "/path/to/dir" ), but as far as I can figure out, calling that line just silently fails and does nothing.
I would understand if Java didn't allow you to do this, if it weren't for the fact that it allows you to get the current working directory, and even allows you to open files using relative file paths....
There is no reliable way to do this in pure Java. Setting the user.dir property via System.setProperty() or java -Duser.dir=... does seem to affect subsequent creations of Files, but not e.g. FileOutputStreams.
The File(String parent, String child) constructor can help if you build up your directory path separately from your file path, allowing easier swapping.
An alternative is to set up a script to run Java from a different directory, or use JNI native code as suggested below.
The relevant OpenJDK bug was closed in 2008 as "will not fix".
If you run your legacy program with ProcessBuilder, you will be able to specify its working directory.
There is a way to do this using the system property "user.dir". The key part to understand is that getAbsoluteFile() must be called (as shown below) or else relative paths will be resolved against the default "user.dir" value.
import java.io.*;
public class FileUtils
{
public static boolean setCurrentDirectory(String directory_name)
{
boolean result = false; // Boolean indicating whether directory was set
File directory; // Desired current working directory
directory = new File(directory_name).getAbsoluteFile();
if (directory.exists() || directory.mkdirs())
{
result = (System.setProperty("user.dir", directory.getAbsolutePath()) != null);
}
return result;
}
public static PrintWriter openOutputFile(String file_name)
{
PrintWriter output = null; // File to open for writing
try
{
output = new PrintWriter(new File(file_name).getAbsoluteFile());
}
catch (Exception exception) {}
return output;
}
public static void main(String[] args) throws Exception
{
FileUtils.openOutputFile("DefaultDirectoryFile.txt");
FileUtils.setCurrentDirectory("NewCurrentDirectory");
FileUtils.openOutputFile("CurrentDirectoryFile.txt");
}
}
It is possible to change the PWD, using JNA/JNI to make calls to libc. The JRuby guys have a handy java library for making POSIX calls called jnr-posix. Here's the maven info
As mentioned you can't change the CWD of the JVM but if you were to launch another process using Runtime.exec() you can use the overloaded method that lets you specify the working directory. This is not really for running your Java program in another directory but for many cases when one needs to launch another program like a Perl script for example, you can specify the working directory of that script while leaving the working dir of the JVM unchanged.
See Runtime.exec javadocs
Specifically,
public Process exec(String[] cmdarray,String[] envp, File dir) throws IOException
where dir is the working directory to run the subprocess in
If I understand correctly, a Java program starts with a copy of the current environment variables. Any changes via System.setProperty(String, String) are modifying the copy, not the original environment variables. Not that this provides a thorough reason as to why Sun chose this behavior, but perhaps it sheds a little light...
The working directory is a operating system feature (set when the process starts).
Why don't you just pass your own System property (-Dsomeprop=/my/path) and use that in your code as the parent of your File:
File f = new File ( System.getProperty("someprop"), myFilename)
The smarter/easier thing to do here is to just change your code so that instead of opening the file assuming that it exists in the current working directory (I assume you are doing something like new File("blah.txt"), just build the path to the file yourself.
Let the user pass in the base directory, read it from a config file, fall back to user.dir if the other properties can't be found, etc. But it's a whole lot easier to improve the logic in your program than it is to change how environment variables work.
I have tried to invoke
String oldDir = System.setProperty("user.dir", currdir.getAbsolutePath());
It seems to work. But
File myFile = new File("localpath.ext");
InputStream openit = new FileInputStream(myFile);
throws a FileNotFoundException though
myFile.getAbsolutePath()
shows the correct path.
I have read this. I think the problem is:
Java knows the current directory with the new setting.
But the file handling is done by the operation system. It does not know the new set current directory, unfortunately.
The solution may be:
File myFile = new File(System.getPropety("user.dir"), "localpath.ext");
It creates a file Object as absolute one with the current directory which is known by the JVM. But that code should be existing in a used class, it needs changing of reused codes.
~~~~JcHartmut
You can use
new File("relative/path").getAbsoluteFile()
after
System.setProperty("user.dir", "/some/directory")
System.setProperty("user.dir", "C:/OtherProject");
File file = new File("data/data.csv").getAbsoluteFile();
System.out.println(file.getPath());
Will print
C:\OtherProject\data\data.csv
You can change the process's actual working directory using JNI or JNA.
With JNI, you can use native functions to set the directory. The POSIX method is chdir(). On Windows, you can use SetCurrentDirectory().
With JNA, you can wrap the native functions in Java binders.
For Windows:
private static interface MyKernel32 extends Library {
public MyKernel32 INSTANCE = (MyKernel32) Native.loadLibrary("Kernel32", MyKernel32.class);
/** BOOL SetCurrentDirectory( LPCTSTR lpPathName ); */
int SetCurrentDirectoryW(char[] pathName);
}
For POSIX systems:
private interface MyCLibrary extends Library {
MyCLibrary INSTANCE = (MyCLibrary) Native.loadLibrary("c", MyCLibrary.class);
/** int chdir(const char *path); */
int chdir( String path );
}
The other possible answer to this question may depend on the reason you are opening the file. Is this a property file or a file that has some configuration related to your application?
If this is the case you may consider trying to load the file through the classpath loader, this way you can load any file Java has access to.
If you run your commands in a shell you can write something like "java -cp" and add any directories you want separated by ":" if java doesnt find something in one directory it will go try and find them in the other directories, that is what I do.
Use FileSystemView
private FileSystemView fileSystemView;
fileSystemView = FileSystemView.getFileSystemView();
currentDirectory = new File(".");
//listing currentDirectory
File[] filesAndDirs = fileSystemView.getFiles(currentDirectory, false);
fileList = new ArrayList<File>();
dirList = new ArrayList<File>();
for (File file : filesAndDirs) {
if (file.isDirectory())
dirList.add(file);
else
fileList.add(file);
}
Collections.sort(dirList);
if (!fileSystemView.isFileSystemRoot(currentDirectory))
dirList.add(0, new File(".."));
Collections.sort(fileList);
//change
currentDirectory = fileSystemView.getParentDirectory(currentDirectory);

Get current path of executed file

I try to write and read to the file in my java project file called Books.txt.
The problem is that I can access the file only if partialPath has full path to the file.
Here is the code:
public <T> List<T> readFromFile(String fileName) {
private String partialPath = "\\HW3\\src\\java\\repos\\";
try {
String path = partialPath + fileName;
FileInputStream fi = new FileInputStream(path);
ObjectInputStream oi = new ObjectInputStream(fi);
// Read objects
List<T> items = (List<T>) oi.readObject();
oi.close();
fi.close();
return items;
} catch (IOException | ClassNotFoundException e) {
}
}
If I set relative path as above I get exception file not found.
My question is how can I set full path to the current directory programmatically?
Here is a code snippet of the Drombler Commons - Client Startup code I wrote, to determine the location of the executable jar. Replace DromblerClientStarter with your main class.
This should work at least when you're running your application as an executable JAR file.
/**
* The jar URI prefix "jar:"
*/
private static final String FULL_JAR_URI_PREFIX = "jar:";
/**
* Length of the jar URI prefix "jar:"
*/
private static final int FULL_JAR_URI_PREFIX_LENGTH = 4;
private Path determineMainJarPath() throws URISyntaxException {
Class<DromblerClientStarter> type = DromblerClientStarter.class;
String jarResourceURIString = type.getResource("/" + type.getName().replace(".", "/") + ".class").toURI().
toString();
int endOfJarPathIndex = jarResourceURIString.indexOf("!/");
String mainJarURIString = endOfJarPathIndex >= 0 ? jarResourceURIString.substring(0, endOfJarPathIndex)
: jarResourceURIString;
if (mainJarURIString.startsWith(FULL_JAR_URI_PREFIX)) {
mainJarURIString = mainJarURIString.substring(FULL_JAR_URI_PREFIX_LENGTH);
}
Path mainJarPath = Paths.get(URI.create(mainJarURIString));
return mainJarPath;
}
Depending on where you bundle Books.txt in your application distribution package, you can use this mainJarPath to determine the path of Books.txt.
I also feel that files created (and later possibly modified and or deleted) by your running Java application is usually better to be placed in a location of the file system that is away from your java application installed home directory. An example might be the 'C:\ProgramData\ApplicationNameFiles\' for the Windows operating system or something similar for other OS platforms. In my opinion, at least for me, I feel it provides less chance of corruption to essential application files due to a poorly maintained drive or, accidental deletion by a User that opens up a File Explorer and decides to take it upon him/her self to clean their system of so called unnecessary files, and other not so obvious reasons.
Because Java can run on almost any platform and such data file locations are platform specific the User should be allowed to select the location to where these files can be created and manipulated from. This location then can be saved as a Property. Indeed, slightly more work but IMHO I feel it may be well worth it.
It is obviously much easier to create a directory (folder) within the install home directory of your JAR file when it's first started and then store and manipulate your application's created data files from there. Definitely much easier to find but then again...that would be a matter of opinion and it wouldn't be mine. Never-the-less if you're bent on doing it this way then your Java application's Install Utility should definitely know where that install path would be, it is therefore just a matter of storing that location somewhere.
No Install Utility? Well then your Java application will definitely need a means to know from where your JAR file is running from and the following code is one way to do that:
public String applicationPath(Class mainStartupClassName) {
try {
String path = mainStartupClassName.getProtectionDomain().getCodeSource().getLocation().getPath();
String pathDecoded = URLDecoder.decode(path, "UTF-8");
pathDecoded = pathDecoded.trim().replace("/", File.separator);
if (pathDecoded.startsWith(File.separator)) {
pathDecoded = pathDecoded.substring(1);
}
return pathDecoded;
}
catch (UnsupportedEncodingException ex) {
Logger.getLogger("applicationPath() Method").log(Level.SEVERE, null, ex);
}
return null;
}
And here is how you would use this method:
String appPath = applicationPath(MyMainStartupClassName.class);
Do keep in mind that if this method is run from within your IDE it will most likely not return the path to your JAR file but instead point to a folder where your classes are stored for the application build.
This is not a unique issue to Java, it's a problem faced by any developer of any language wishing to write data locally to the disk. The are many parts to this problem.
If you want to be able to write to the file (and presumably, read the changes), then you need to devise a solution which allows you find the file in a platform independent way.
Some of the issues
The installation location of the program
While most OS's do have some conventions governing this, this doesn't mean they are always used, for what ever reason.
Also, on some OS's, you are actively restricted from writing to the "installation" location. Windows 8+ doesn't allow you to write to the "Program Files" directory, and in Java, this usually (or at least when I was dealing with it) fails silently.
On MacOS, if you're using a "app bundle", the working directory is automatically set to the user's home directory, making it even more difficult to manage
The execution context (or working directory) may be different from the installation location of the program
A program can be installed in one location, but executed from a different location, this will change the working directory location. Many command line tools suffer from this issue and use different conventions to work around it (ever wonder what the JAVA_HOME environment variable is for 🤔)
Restricted disk access
Many OS's are now actively locking down the locations to which programs can write, even with admin privileges.
A reusable solution...
Most OS's have come up with conventions for solving this issue, not just for Java, but for all developers wishing to work on the platform.
Important Like all guide lines, these are not hard and fast rules, but a recommendations made by the platform authors, which are intended to make your life simpler and make the operation of the platform safer
The most common solution is to simply place the file in a "well known location" on the disk, which can be accessed through an absolute path independently of the installation or execution location of the program.
On Windows, this means placing the file in either ~\AppData\Local\{application name} or ~\AppData\Roaming\{application name}
On MacOS, this means placing the file in ~/Library/Application Data/{application name}
On *nix, this typically means placing the file in ~/.{application name}
It could be argued that you could use ~/.{application name} on all three platforms, but as a user who "shows hidden files", I'd prefer you didn't pollute my home directory.
A possible, reusable, solution...
When Windows 8 came out, I hit the "you can't write to the Program Files" issue, which took some time to diagnose, as it didn't generate an exception, it just failed.
I was also working a lot more on Mac OS as well, so I needed a simple, cross platform solution, so my code could automatically adapt without the need for multiple branches per platform.
To this end, I came with a simple utility class...
public enum SystemUtilities {
INSTANCE;
public boolean isMacOS() {
return getOSName().startsWith("Mac");
}
public boolean isMacOSX() {
return getOSName().startsWith("Mac OS X");
}
public boolean isWindowsOS() {
return getOSName().startsWith("Windows");
}
public boolean isLinux() {
return getOSName().startsWith("Linux");
}
public String getOSName() {
return System.getProperty("os.name");
}
public File getRoamingApplicationSupportPath() {
// For *inx, use '~/.{AppName}'
String path = System.getProperty("user.home");
if (isWindowsOS()) {
path += "\\AppData\\Roaming";
} else if (isMacOS()) {
path += "/Library/Application Support";
}
return new File(path);
}
public File getLocalApplicationSupportPath() {
// For *inx, use '~/.{AppName}'
String path = System.getProperty("user.home");
if (isWindowsOS()) {
path += "\\AppData\\Local";
} else if (isMacOS()) {
path += "/Library/Application Support";
}
return new File(path);
}
}
This provides a baseline from which "independent" code can be built, for example, you could use something like...
File appDataDir = new File(SystemUtilities.INSTANCE.getLocalApplicationSupportPath(), "MyAwesomeApp");
if (appDataDir.exists() || appDataDir.mkdirs()) {
File fileToWrite = new File(appDataDir, "Books.txt");
//...
}
to read/write to the file. Although, personally, I might have manager/factory do this work and return the reference to the end File, but that's me.
What about "pre-packaged" files?
Three possible solutions...
Create the file(s) if they don't exist, populating them with default values as required
Copy "template" file(s) out of the Jar file, if they don't exist
Use an installer to install the files - this is the solution we used when we were faced with changing the location of all our "external" configuration files.
Read only files...
For read only files, the simplest solution is to embedded them within the Jar as "embedded resources", this makes it easier to locate and manage...
URL url = getClass().getResource("/path/to/readOnlyResource.txt");
How you do this, will depend on your build system

How to remove ICEpdf temporary files after application has finished working?

I use ICEpdf library for PDF displaying at my desktop java application. Application adds annotations to PDF at runtime, but without changing original files — changes are displayed only during one 'session'. I recently discovered that application creates a lot of temporary files which consume quite a lot of disk space.
Method org.icepdf.core.pobjects.Document.setInputStream has the following code:
// Delete temp file on exit
tempFile.deleteOnExit();
So I suppose it has to remove temporary files after it used them, but it does not:
How can I programmatically remove all files created by application on exit or make standard file removing work?
To get temp folder path:
FileSystems.getDefault().getPath(System.getProperty("java.io.tmpdir"))
To remove files:
try (DirectoryStream<Path> paths = Files.newDirectoryStream(pathToDir, regex)){
paths.forEach(path -> path.toFile().delete());
} catch (IOException e) {
// handle io exception
}
where regex is filename pattern. In your case: "IcePdf*"

How to get a Resource in Apache Brooklyn

I am trying to build my own entity, which is based on VanillaWindowsProcess. The idea is, after the installation of the windows Machine, to execute some powershell commands, which are in a file.
I tried something which I used a lot of times in another Java projects to get a resource:
private void runInstallationScript() {
List<String> lines;
try {
lines = FileUtils.readLines(
new File(TalendWindowsProcessWinRmDriver.class.getResource("/my/path/file.txt").getFile()),
"utf-8");
executePsScript(lines);
} catch (IOException e) {
LOG.error("Error reading the file: ", e);
}
}
But I'm always getting the following:
ava.io.FileNotFoundException: File 'file:/opt/workspace/incubator-brooklyn/usage/dist/target/brooklyn-dist/brooklyn/lib/dropins/myProject-0.0.1-SNAPSHOT.jar!/my/path/file.txt' does not exist
It is strange, because the file is in the jar in that path. I did a test (without Apache Brooklyn infrastructure) and it works, but the other way, it does not.
The project follows the Maven standard structure and the file itself is under, src/main/resources/my/path/file.txt
Is there something that is wrong? Or maybe there is another approach to get that file? Any help would be appreciated.
You cannot access a resource inside a jar as a File object. You need to use an InputStream (or an URL) to access it.
Since you are already using getResource, you should change the method FileUtils.readLines to accept an InputStream (or an URL) as input.
If you don't have access to the source code, you can write your own method or use Files.readAllLines for Java >= 7.

Java: get absolute path of project

I'm trying to run a exe file in path outside of the current package. My code.java file that runs it is in
%Workspace_path%\Project\src\main\java\com\util\code.java
However the directory of where the exe is
%Workspace_path%\Project\src\main\resources\program.exe
If possible, it seems like the best solution here would be to get the absolute path of the Project then append "src\main\resources\" to it. Is there a good way to do this or is there an alternative solution?
I'm using Eclipse, but it would great if it could be used in other IDEs too. Thanks for any help.
The de facto approach to solving this is to bundle the EXE as a classpath resource. It seems you have arranged for this already.
When working with classpath resources, a mature program should not assume that the resource is in the filesystem. The resources could be packaged in a JAR file, or even in a WAR file. The only thing you can trust at that point is the standard methods for accessing resources in Java, as hinted below.
The way to solve your problem, then, is to access the resource contents using the de facto standard of invoking Class.getResourceAsStream (or ClassLoader.getResourceAsStream), save the contents to a temporary file, and execute from that file. This will guarantee your program works correctly regardless of its packaging.
In other words:
Invoke getClass().getResourceAsStream("/program.exe"). From static methods, you can't call getClass, so use the name of your current class instead, as in MyClass.class.getResourceAsStream. This returns an InputStream.
Create a temporary file, preferably using File.createTempFile. This returns a File object identifying the newly created file.
Open an OutputStream to this temp file.
Use the two streams to copy the data from the resource into the temp file. You can use IOUtils.copy if you're into Apache Commons tools. Don't forget to close the two streams when done with this step.
Execute the program thus stored in the temporary file.
Clean up.
In other words (code snippet added later):
private void executeProgramFromClasspath() throws IOException {
// Open resource stream.
InputStream input = getClass().getResourceAsStream("/program.exe");
if (input == null) {
throw new IllegalStateException("Missing classpath resource.");
}
// Transfer.
OutputStream output = null;
try {
// Create temporary file. May throw IOException.
File temporaryFile = File.createTempFile(getClass().getName(), "");
output = new FileOutputStream(temporaryFile);
output = new BufferedOutputStream(output);
IOUtils.copy(input, output);
} finally {
// Close streams.
IOUtils.closeQuietly(input);
IOUtils.closeQuietly(output);
}
// Execute.
try {
String path = temporaryFile.getAbsolutePath();
ProcessBuilder processBuilder = new ProcessBuilder(path);
Process process = processBuilder.start();
process.waitFor();
} catch (InterruptedException e) {
// Optional catch. Keeps the method signature uncluttered.
throw new IOException(e);
} finally {
// Clean up
if (!temporaryFile.delete()) {
// Log this issue, or throw an error.
}
}
}
Well,in your context,the project root is happen to be the current path
.
,that is where the java.exe start to execute,so a easy way is:
String exePath="src\\main\\resources\\program.exe";
File exeFile=new File(".",exePath);
System.out.println(exeFile.getAbusolutePath());
...
I tested this code on Eclipse,It's ok. I think is should work on different ide.
Good Luck!

Categories