I have this code that lists all the files in a dir:
File d = getRootDir(cId);
return d.listFiles(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.endsWith(FILE_EXTENSION);
}
}
It hangs because, the dir got corrupted and has a file name with invalid chars. Is there an alternate way to do this, so that i can get the list of files and not hang?
Thanks.
Personally whenever I do file operations I will use apache Commons-io. I find the API much easier to work with than Java's own while still using standard File object.
That said I do not know if switching will make it list without hanging.
Insert a logging statement in your accept method. I'd recommend a logging library but a simple
System.out.println("Checking " + name + " in " + dir);
Should suffice.
You will now where it is hanging but probably not why. Perhaps it is not hanging for the reason you think it is, the log statement will give you more information on this. You can dig deeper by trace debugging the code though for this you will probably need to download the source code of the JDK.
What is the illegal character you have in your file that is causing problem ?
Also, a second look here at your code. The file name contains the file and extension. By checking with endsWith you risk getting false positive where a file may not have an extention or a different extention. For example :
FILE_EXTENSION = "log";
The following files will trigger though it will not be what you want.
totolog
toto.totolog
You can prepend the . before, if you have full control (not user defined configuration) then it will do but a cleaner solution is to extract the extension and compare that. A simple call to FilenameUtils.getExtension will do the trick.
Have you tried d.list() and interating over that to determine your accept? The problem might be inside FilenameFilter and not the actual interating over the file list.
Are you on Windows? Have you tried chkdsk? (however you get to that these days).
Related
Asked this question, having already tried possible solutions in other questions here on stack but that didn't allow me to fix the problem.
As in the title, I have created a java utility with which I have to perform operations on text files, in particular I have to perform simple operations to move between directories, copy from one directory to another, etc.
To do this I have used the java libraries java.io.File and java.nio.*, And I have implemented two functions for now,copyFile(sourcePath, targetPath) and moveFile(sourcePath, targetPath).
To develop this I am using a mac, and the files are under the source path /Users/myname/Documents/myfolder/F24/archive/, and my target path is /Users/myname/Documents/myfolder/F24/target/.
But when I run my code I get a java.nio.file.NoSuchFileException: /Users/myname/Documents/myfolder/F24/archive
Having tried the other solutions here on stack and java documentation already I haven't been able to fix this yet ... I accept any advice or suggestion
Thank you all
my code:
// copyFile: funzione chiamata per copiare file
public static boolean copyFile(String sourcePath, String targetPath){
boolean fileCopied = true;
try{
Files.copy(Paths.get(sourcePath), Paths.get(targetPath), StandardCopyOption.REPLACE_EXISTING);
}catch(Exception e){
String sp = Paths.get(sourcePath)+"/";
fileCopied = false;
System.out.println("Non posso copiare i file dalla cartella "+sp+" nella cartella "+Paths.get(targetPath)+" ! \n");
e.printStackTrace();
}
return fileCopied;
}
Files.copy cannot copy entire directories. The first 'path' you pass to Files.copy must ALL:
Exist.
Be readable by the process that runs the JVM. This is non-trivial on a mac, which denies pretty much all disk rights to all apps by default until you give it access. This can be tricky for java apps. I'm not quite sure how you fix it (I did something on my mac to get rid of that, but I can't remember what - possibly out of the box java apps just get to read whatever they want and it's only actual mac apps that get pseudo-sandboxed. Point is, there's a chance it's mac's app access control denying it even if the unix file rights on this thing indicate you ought to be able to read it).
Be a plain old file and not a directory or whatnot.
Files.move can (usually - depends on impl and underlying OS) usually be done to directories, but not Files.copy. You're in a programming language, not a shell. If you want to copy entire directories, write code that does this.
Not sure whether my comment is understood though answered.
Ìn java SE target must not be the target directory. In other APIs of file copying
one can say COPY FILE TO DIRECTORY. In java not so; this was intentionally designed to remove one error cause.
That style would be:
Path source = Paths.get(sourcePath);
if (Files.isRegularFile(source)) {
Path target = Paths.get(targetPath);
Files.createDirectories(target);
if (Files.isDirectory(target)) {
target = Paths.get(targetPath, source.getFileName().toString());
// Or: target = target.resolve(source.getFileName().toString());
}
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
}
Better ensure when calling to use the full path.
Checkmarx - v 9.3.0 HF11
I am passing env value as data directory path in docker file which used in dev/uat server
ENV DATA /app/data/
In local, using following Environment variable
DATA=C:\projects\app\data\
getDataDirectory("MyDirectoryName"); // MyDirectoryName is present in data folder
public String getDataDirectory(String dirName)
{
String path = System.getenv("DATA");
if (path != null) {
path = sanitizePathValue(path);
path = encodePath(path);
dirName = sanitizePathValue(dirName);
if (!path.endsWith(File.separator)) {
path = path + File.separator;
} else if (!path.contains("data")) {
throw new MyRuntimeException("Data Directory path is incorrect");
}
} else {
return null;
}
File file = new File(dirName); // NOSONAR
if (!file.isAbsolute()) {
File tmp = new File(SecurityUtil.decodePath(path)); // NOSONAR
if (!tmp.getAbsolutePath().endsWith(Character.toString(File.separatorChar))) {
dirName = tmp.getAbsolutePath() + File.separatorChar + dirName;
} else {
dirName = tmp.getAbsolutePath() + dirName;
}
}
return dirName;
}
public static String encodePath(String path) {
try {
return URLEncoder.encode(path, "UTF-8");
} catch (UnsupportedEncodingException e) {
logger.error("Exception while encoding path", e);
}
return "";
}
public static String validateAndNormalizePath(String path) {
path = path.replaceAll("/../", "/");
path = path.replaceAll("/%46%46/", "/");
path = SecurityUtil.cleanIt(path);
path = FilenameUtils.normalize(path); // normalize path
return path;
}
public static String sanitizePathValue(String filename){
filename = validateAndNormalizePath(filename);
String regEx = "..|\\|/";
// compile the regex to create pattern
// using compile() method
Pattern pattern = Pattern.compile(regEx);
// get a matcher object from pattern
Matcher matcher = pattern.matcher(filename);
// check whether Regex string is
// found in actualString or not
boolean matches = matcher.matches();
if(matches){
throw new MyAppRuntimeException("filename:'"+filename+"' is bad.");
}
return filename;
}
public static String validateAndNormalizePath(String path) {
path = path.replaceAll("/../", "/");
path = path.replaceAll("/%46%46/", "/");
path = SecurityUtil.cleanIt(path);
path = FilenameUtils.normalize(path); // normalize path
return path;
}
[Attempt] - Update code which I tried with the help of few members to prevent path traversal issue.
Tried to sanitize string and normalize string, but no luck and getting same issue.
How to resolve Stored Absolute Path Traversal issue ?
Your first attempt is not going to work because escaping alone isn't going to prevent a path traversal. Replacing single quotes with double quotes won't do it either given you need to make sure someone setting a property/env variable with ../../etc/resolv.conf doesn't succeed in tricking your code into overwriting/reading a sensitive file. I believe Checkmarx won't look for StringUtils as part of recognizing it as sanitized, so the simple working example below is similar without using StringUtils.
Your second attempt won't work because it is a validator that uses control flow to prevent a bad input when it throws an exception. Checkmarx analyzes data flows. When filename is passed as a parameter to sanitizePathValue and returned as-is at the end, the data flow analysis sees this as not making a change to the original value.
There also appears to be some customizations in your system that recognize System.getProperty and System.getenv as untrusted inputs. By default, these are not recognized in this way, so anyone trying to scan your code probably would not have gotten any results for Absolute Path Traversal. It is possible that the risk profile of your application requires that you call properties and environment variables as untrusted inputs, so you can't really just remove these and revert back to the OOTB settings.
As Roman had mentioned, the logic in the query does look for values that are prepended to this untrusted input to remove those data flows as results. The below code shows how this could be done using Roman's method to trick the scanner. (I highly suggest you do not choose the route to trick the scanner.....very bad idea.) There could be other string literal values that would work using this method, but it would require some actions that control how the runtime is executed (like using chroot) to make sure it actually fixed the issue.
If you scan the code below, you should see only one vulnerable data path. The last example is likely something along the lines of what you could use to remediate the issues. It really depends on what you're trying to do with the file being created.
(I tested this on 9.2; it should work for prior versions. If it doesn't work, post your version and I can look into that version's query.)
// Vulnerable
String fn1 = System.getProperty ("test");
File f1 = new File(fn1);
// Path prepend - still vulnerable, tricks the scanner, DO NOT USE
String fn2 = System.getProperty ("test");
File f2 = new File(Paths.get ("", fn2).toString () );
// Path prepend - still vulnerable, tricks the scanner, DO NOT USE
String fn3 = System.getProperty ("test");
File f3 = new File("" + fn3);
// Path prepend - still vulnerable, tricks the scanner, DO NOT USE
String fn4 = System.getProperty ("test");
File f4 = new File("", fn4);
// Sanitized by stripping path separator as defined in the JDK
// This would be the safest method
String fn5 = System.getProperty ("test");
File f5 = new File(fn5.replaceAll (File.separator, ""));
So, in summary (TL;DR), replace the file separator in the untrusted input value:
String fn5 = System.getProperty ("test");
File f5 = new File(fn5.replaceAll (File.separator, ""));
Edit
Updating for other Checkmarx users that may come across this in search of an answer.
After my answer, OP updated the question to reveal that the issue being found was due to a mechanism written for the code to run in different environments. Pre-docker, this would have been the method to use. The vulnerability would have still been detected but most courses of action would have been to say "our deployment environment has security measures around it to prevent a bad actor from injecting an undesired path into the environment variable where we store our base path."
But now, with Docker, this is a thing of the past. Generally the point of Docker is to create applications that run the way same everywhere they are deployed. Using a base path in an environment likely means OP is executing the code outside of a container for development (based on the update showing a Windows path) and inside the container for deployment. Why not just run the code in the container for development as well as deployment as is intended by Docker?
Most of the answers tend to explain that OP should use a static path. This is because they are realizing that there is no way to avoid this issue because taking an untrusted input (from the environment) and prefixing it to a path is the exact problem of Absolute Path Traversal.
OP could follow the good advice of many posters here and put a static base path in the code then use Docker volumes or Docker bind mounts.
Is it difficult? Nope. If I were OP, I'd fix the base path prefix in code to a static value of /app/data and do a simple volume binding during development. (When you think about it, if there is storage of data in the container during a deployment then the deployment environment must be doing this exact thing for /app/data unless the data is not kept after the lifetime of the container.)
With the base path fixed at /app/data, one option for OP to run their development build is:
docker run -it -v"C:\\projects\\app\\data":/app/data {container name goes here}
All data written by the application would appear in C:\projects\app\data the same way it does when using the environment variables. The main difference is that there are no environment-variable-prefixed paths and thus no Absolute Path Traversal results from the static analysis scanner.
It depends on how Checkmarx comes to this point. Most likely because the value that is handed to File is still tainted. So make sure both /../ and /%46%46/ are replaced by /.
checkedInput = userInput.replaceAll("/../", "/");
Secondly, give File a parent directory to start with and later compare the path of the file you want to process. Some common example code is below. If the file doesn't start with the full parent directory, then it means you have a path traversal.
File file = new File(BASE_DIRECTORY, userInput);
if (file.getCanonicalPath().startsWith(BASE_DIRECTORY)) {
// process file
}
Checkmarx can only check if variables contain a tainted value and in some cases if the logic is correct. Please also think about the running process and file system permissions. A lot of applications have the capability of overwriting their own executables.
If there is one thing to remember it is this
use allow lists not deny lists
(traditionally known as whitelists and blacklists).
For instance, consider replacing /../ with / suggested in another answer. My response is to contain the sequence /../../. You could pursue this iteratively, and I might run out of adversarial examples, but that doesn't mean there are any.
Another problem is knowing all the special characters. \0 used to truncate the file name. What happens to non-ASCII characters - I can't remember. Might other code be changed in future so that the path ends up on a command line with other special characters - worse, OS/command line dependent.
Canonicalisation has its problems too. It can be used to some extent probe the file system (and perhaps beyond the machine).
So, choose what you allow. Say
if (filename.matches("[a-zA-Z0-9_]+")) {
return filename;
} else {
throw new MyException(...);
}
(No need to go through the whole Pattern/Matcher palaver in this situation.)
For this issue i would suggest you hard code the absolute path of the directory that you allow your program to work in; like this:
String separator = FileSystems.getDefault().getSeparator();
// should resolve to /app/workdir in linux
String WORKING_DIR = separator + "app"+separator +"workdir"+separator ;
then when you accept the parameter treat it as a relative path like this:
String filename = System.getProperty("test");
sanitize(filename);
filename = WORKING_DIR+filename;
File dictionaryFile = new File(filename);
To sanitize your user's input make sure he does not include .. and does not include also \ nor /
private static void sanitize(filename){
if(Pattern.compile("\\.\\.|\\|/").matcher(filename).find()){
throw new RuntimeException("filename:'"+filename+"' is bad.");
}
}
Edit
In case you are running the process in linux you can change the root of the process using chroot maybe you do some googling to know how you should implement it.
how about using Java's Path to make the check("../test1.txt" is the input from user):
File base=new File("/your/base");
Path basePath=base.toPath();
Path resolve = basePath.resolve("../test1.txt");
Path relativize = basePath.relativize(resolve);
if(relativize.startsWith("..")){
throw new Exception("invalid path");
}
Based on reading the Checkmarx query for absolute path traversal vulnerability (and I believe in general one of the mitigation approach), is to prepend a hard coded path to avoid the attackers traversing through the file system:
File has a constructor that accepts a second parameter that will allow you to perform some prepending
String filename = System.getEnv("test");
File dictionaryFile = new File("/home/", filename);
UPDATE:
The validateAndNormalizePath would have technically sufficed but I believe Checkmarx is unable to recognize this as a sanitizer (being a custom written function). I would advice to work with your App Security team for them to use the CxAudit and overwrite the base Stored Path Traversal Checkmarx query to recognize validateAndNormalizePath as a valid sanitizer.
In a DirectoryWalker class I want to find out if a File instance is actually a symbolic link to a directory (assuming, the walker walks on UNIX systems). Given, I already know the instance is a directory, would the following be a reliable condition to determine the symbolic link?
File file;
// ...
if (file.getAbsolutePath().equals(file.getCanonicalPath())) {
// real directory ---> do normal stuff
}
else {
// possible symbolic link ---> do link stuff
}
The technique used in Apache Commons uses the canonical path to the parent directory, not the file itself. I don't think that you can guarantee that a mismatch is due to a symbolic link, but it's a good indication that the file needs special treatment.
This is Apache code (subject to their license), modified for compactness.
public static boolean isSymlink(File file) throws IOException {
if (file == null)
throw new NullPointerException("File must not be null");
File canon;
if (file.getParent() == null) {
canon = file;
} else {
File canonDir = file.getParentFile().getCanonicalFile();
canon = new File(canonDir, file.getName());
}
return !canon.getCanonicalFile().equals(canon.getAbsoluteFile());
}
Java 1.6 does not provide such low level access to the file system. Looks like NIO 2, which should be included in Java 1.7, will have support for symbolic links. A draft of the new API is available. Symbolic links are mentioned there, creating and following them is possible. I'm not exactly sure that which method should be used to find out whether a file is a symbolic link. There's a mailing list for discussing NIO 2 - maybe they will know.
Also, watch out for file.isFile() and file.isDirectory() both returning results based on the resolved file and therefore both returning false when file refers to a symlink where the target doesn't exist.
(I know this isn't a useful answer in itself but it tripped me up a couple of times so thought I should share)
It looks like getCanonicalPath() can do other things that might make it different from the absolute path.
This method first converts this pathname to absolute form if necessary, as if by invoking the getAbsolutePath() method, and then maps it to its unique form in a system-dependent way. This typically involves removing redundant names such as "." and ".." from the pathname, resolving symbolic links (on UNIX platforms), and converting drive letters to a standard case (on Microsoft Windows platforms).
But it might work for the vast majority of your use cases; your mileage may vary.
If you are already coding something specifically for *nix, then you could do a shell command from Java like this:
Process p = Runtime.getRuntime().exec(new String[]{"test", "-h", yourFileName});
p.waitFor();
if (p.exitValue() == 0)
System.out.println("This file is a symbolic link");
else
System.out.println("This file is not a symbolic link");
That's very specific to *nix, but it does at least work.
Sorry to reply to such an old post, but I was looking for a solution for Windows systems some time back, and some of the previous answers didn't work out for me. If you're not concerned with cross platform compatibility and only need a solution for Windows, the following technique worked well for my purposes.
File f = new File("whatever file or folder");
if (f instanceof ShellFolder) {
ShellFolder sf = (ShellFolder)f;
if (sf.isLink()) {
// Your code when it's a link
}
}
I thought I would share some good fortune I had in dealing with this issue. I am using JDK 1.6.0_23 and so I cannot benefit from NIO2. I am building and running on Windows 7 /x64 ONLY so mileage may vary in other environments. Unfortunately, other solutions here did not work for me in avoiding NullPointerExceptions caused when attempting to traverse a junction (probably because junction != symlink....). While I am not constrained by JDK version, I decided to keep at the problem for a bit longer.
I had this code which would cause a NullPointerException if used on a symbolic link or when encountering the 'System Volume Information' directory. (Note, traverseItem.f() returns an object of type java.io.File)
if (traverseItem.f().isDirectory) {
for (File item : traverseItem.f().listFiles()) {
So, it is supposedly a directory but calling listFiles() on it causes an NPE. What to do? I spied the list() method and wondered if it would exhibit the same behavior. What I discovered was the following:
Calling list() on a File describing an empty folder returns a String[] array of length zero. However, calling list() on a File describing a junction which would otherwise crash from listFiles() returns null
I was able to avoid the NullPointerExceptions by adding the following test before calling listFiles()
String[] contents = traverseItem.f().list();
if (contents != null) { //Non-traversible if null, possibly junction or ???
It remains to exhaustively test all cases of junction, symbolic link, hard link, and dare I mention it, shortcut, but this may help some.
In a DirectoryWalker class I want to find out if a File instance is actually a symbolic link to a directory (assuming, the walker walks on UNIX systems). Given, I already know the instance is a directory, would the following be a reliable condition to determine the symbolic link?
File file;
// ...
if (file.getAbsolutePath().equals(file.getCanonicalPath())) {
// real directory ---> do normal stuff
}
else {
// possible symbolic link ---> do link stuff
}
The technique used in Apache Commons uses the canonical path to the parent directory, not the file itself. I don't think that you can guarantee that a mismatch is due to a symbolic link, but it's a good indication that the file needs special treatment.
This is Apache code (subject to their license), modified for compactness.
public static boolean isSymlink(File file) throws IOException {
if (file == null)
throw new NullPointerException("File must not be null");
File canon;
if (file.getParent() == null) {
canon = file;
} else {
File canonDir = file.getParentFile().getCanonicalFile();
canon = new File(canonDir, file.getName());
}
return !canon.getCanonicalFile().equals(canon.getAbsoluteFile());
}
Java 1.6 does not provide such low level access to the file system. Looks like NIO 2, which should be included in Java 1.7, will have support for symbolic links. A draft of the new API is available. Symbolic links are mentioned there, creating and following them is possible. I'm not exactly sure that which method should be used to find out whether a file is a symbolic link. There's a mailing list for discussing NIO 2 - maybe they will know.
Also, watch out for file.isFile() and file.isDirectory() both returning results based on the resolved file and therefore both returning false when file refers to a symlink where the target doesn't exist.
(I know this isn't a useful answer in itself but it tripped me up a couple of times so thought I should share)
It looks like getCanonicalPath() can do other things that might make it different from the absolute path.
This method first converts this pathname to absolute form if necessary, as if by invoking the getAbsolutePath() method, and then maps it to its unique form in a system-dependent way. This typically involves removing redundant names such as "." and ".." from the pathname, resolving symbolic links (on UNIX platforms), and converting drive letters to a standard case (on Microsoft Windows platforms).
But it might work for the vast majority of your use cases; your mileage may vary.
If you are already coding something specifically for *nix, then you could do a shell command from Java like this:
Process p = Runtime.getRuntime().exec(new String[]{"test", "-h", yourFileName});
p.waitFor();
if (p.exitValue() == 0)
System.out.println("This file is a symbolic link");
else
System.out.println("This file is not a symbolic link");
That's very specific to *nix, but it does at least work.
Sorry to reply to such an old post, but I was looking for a solution for Windows systems some time back, and some of the previous answers didn't work out for me. If you're not concerned with cross platform compatibility and only need a solution for Windows, the following technique worked well for my purposes.
File f = new File("whatever file or folder");
if (f instanceof ShellFolder) {
ShellFolder sf = (ShellFolder)f;
if (sf.isLink()) {
// Your code when it's a link
}
}
I thought I would share some good fortune I had in dealing with this issue. I am using JDK 1.6.0_23 and so I cannot benefit from NIO2. I am building and running on Windows 7 /x64 ONLY so mileage may vary in other environments. Unfortunately, other solutions here did not work for me in avoiding NullPointerExceptions caused when attempting to traverse a junction (probably because junction != symlink....). While I am not constrained by JDK version, I decided to keep at the problem for a bit longer.
I had this code which would cause a NullPointerException if used on a symbolic link or when encountering the 'System Volume Information' directory. (Note, traverseItem.f() returns an object of type java.io.File)
if (traverseItem.f().isDirectory) {
for (File item : traverseItem.f().listFiles()) {
So, it is supposedly a directory but calling listFiles() on it causes an NPE. What to do? I spied the list() method and wondered if it would exhibit the same behavior. What I discovered was the following:
Calling list() on a File describing an empty folder returns a String[] array of length zero. However, calling list() on a File describing a junction which would otherwise crash from listFiles() returns null
I was able to avoid the NullPointerExceptions by adding the following test before calling listFiles()
String[] contents = traverseItem.f().list();
if (contents != null) { //Non-traversible if null, possibly junction or ???
It remains to exhaustively test all cases of junction, symbolic link, hard link, and dare I mention it, shortcut, but this may help some.
I have the following snippet of java code:
File directoryToMoveTo = new File(file.getParent()+"_TEMP");
boolean success = file.renameTo(new File(directoryToMoveTo,file.getName()));
if (!success){
logger.warn("Failed to move [%s] to temp Directory.");
}
file is passed in as an argument to the method and is one of an array of files obtained like this:
File[] files = directory.listFiles(new FilenameFilter() {
#Override
public boolean accept(File dir, String name) {
logger.debug(String.format("Testing file [%s]",name));
boolean passed = name.endsWith(getFileDescription().getFilePattern());
logger.debug(String.format("Passed [%s]",passed));
return passed;
}
});
Why would success by false in the first snippet?
I tried this code in isolation on a different file and it seems to work.
Obvious situations:
the target file already exists
the target directory doesn't exist
the target directory is on a different file system
the target directory is read-only (or at least, the current user doesn't have write access)
I'd expect those to at least potentially fail (the JavaDoc explicitly says that a lot of this behaviour is OS-dependent) - have you tried them?
I found the problem. It was because the directory I was copying to didn't exist.
surrounding with this if statement worked:
if (directoryToMoveTo.exists() || directoryToMoveTo.mkdir()){ }
Original doesn't exist?
Already a file at the destination path?
Destinatination path doesn't exist?
Source file read only?
Just a few ideas
I can think of:
target directory does not exist
not enough access rights (target directory write protected)
not enough free space on target directory's data partition
...
The file may be still open, even though you closed it: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6266377
Not to forget you might not be allowed to write/change/rename a file.
Hardly ever a problem in windows, but common in Unix environments.
To find the exact reason why it is not working you could System.out.println these paths and try to move them from OS level. That would give the good indication why is it not working.