Does anyone know how Java is able to circumvent the windows MAX_PATH limitations. Using the below code I was able to create a really long path in Java and was able to perform I/O, which would have been impossible using windows without prefixing \\?\.
public static void main(String[] args) throws IOException {
BufferedWriter bufWriter = null;
try {
StringBuilder s = new StringBuilder();
for (int i = 0; i < 130; i++) {
s.append("asdf\\");
}
String filePath = "C:\\" + s.toString();;
System.out.println("File Path = " + filePath);
File f = new File(filePath);
f.mkdirs();
f = new File(f, "dummy.txt");
System.out.println("Full path = " + f);
bufWriter = new BufferedWriter(new FileWriter(f));
bufWriter.write("Hello");
}
catch (Exception e) {
e.printStackTrace();
}
finally {
if (bufWriter != null) {
bufWriter.close();
}
}
}
From the JVM's canonicalize_md.c:
/* copy \\?\ or \\?\UNC\ to the front of path*/
WCHAR* getPrefixed(const WCHAR* path, int pathlen) {
[download JVM source code (below) to see implementation]
}
The function getPrefixed is called:
by the function wcanonicalize if ((pathlen = wcslen(path)) > MAX_PATH - 1)
by the function wcanonicalizeWithPrefix.
I didn't trace the call chain farther than that, but I assume the JVM always uses these canonicalization routines before accessing the filesystem, and so always hits this code one way or another. If you want to trace the call chain farther yourself, you too can partake in the joys of browsing the JVM source code! Download at: http://download.java.net/openjdk/jdk6/
Windows bypasses that limitation if the path is prefixed with \\?\.
Most likely Java is in fact using UNC paths (\?) internally.
Related
I've decided to write a recursive program that writes all the files in my C drive into a .txt file, however it is very slow.
I've read online that recursion is slow, but i can't think of any other way. Is there any way i can optimize this ?
EDIT : changed the deepInspect method to use a Stack instead of recursion, which slightly improved performance.
Here is the code
public class FileCount {
static long fCount = 0;
public static void main(String[] args) {
System.out.println("Start....");
long start = System.currentTimeMillis();
File cDir = new File("C:\\");
inspect(cDir);
System.out.println("Operation took : " + (System.currentTimeMillis() - start) + " ms");
}
private static void inspect(File cDir) {
for (File f : cDir.listFiles()) {
deepInspect(f);
}
}
private static void deepInspect(File f) {
Stack<File> stack = new Stack<File>();
stack.push(f);
while (!stack.isEmpty()) {
File current = stack.pop();
if (current.listFiles() != null) {
for (File file : current.listFiles()) {
stack.push(file);
}
}
writeData(current.getAbsolutePath());
}
}
static FileWriter writer = null;
private static void writeData(String absolutePath) {
if (writer == null)
try {
writer = new FileWriter("C:\\Collected\\data.txt");
} catch (IOException e) {}
try {
writer.write(absolutePath);
writer.write("\r\n");//nwline
writer.write("Files : " + fCount);
writer.write("\r\n");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Java 8 provides a stream to process all files.
Files.walk(Paths.get("/"))
.filter(Files::isRegularFile)
.forEach(System.out::println);
You could add "parallel" processing for improved performance
Files.walk(Paths.get("/"))
.parallel()
.filter(Files::isRegularFile)
.forEach(System.out::println);
I tried this under linux, so you would need to replace "/" with "C:" and try it. Besides in my case stops when I try to read I don't have access, so you would need to check that too if you are not running as admin.
Check this out
I don't think the recursion is an issue here. The main issue in your code is the File IO which you are doing at every level. The disk access is extremely costly w.r.t the memory access. If you profile your code you should definitely see huge spike in the disk IO.
So, essentially you want to reduce the disk I/O. To do so you could have a in memory finite size Buffer where you can write the output and when the buffer is full flush the data to the file.
This however considerable more amount of work.
What is the best way of passing a string (arg1 in case of code below) with about 800K characters (yes, that's a huge number) to a java jar. Following is the code I am using to invoke the jar file:
p = Runtime.getRuntime().exec(jrePath+"/bin/java -jar C:/folder/myjar.jar methodName" + arg1);
ALternately, how can I create a jar file to accept one String input and one byte[] input in main{ in void main(String args[])}
Or any other ideas? The requirement is to somehow pass the huge String/byte[] of String to a java jar file that I am creating
As mentioned in this question, there is a maximum argument length set by the operating system. Seeing as this argument is 800K characters, its fairly safe to say that you have exceeded this max value on most computers. To get around this, you can write arg1 to a temp file using the built in API:
final File temp;
try {
temp = File.createTempFile("temp-string", ".tmp");
} catch (final IOException e){
//TODO handle error
return;
}
try (final BufferedWriter writer = new BufferedWriter(new FileWriter(temp))) {
writer.write(arg1);
} catch (final IOException e){
//TODO handle error
return;
}
try {
// run process and wait for completion
final Process process = Runtime.getRuntime().exec(
jrePath + "/bin/java -jar C:/folder/myjar.jar methodName " +
temp.getAbsolutePath());
final int exitCode = process.waitFor();
if (exitCode != 0) {
//TODO handle error
}
} catch (final IOException | InterruptedException e){
//TODO handle error
return;
}
if (!file.delete()) {
//TODO handle error
}
I have the code below, which simply reads all the files from a folder. There are 20,000 files in this folder. The code works good on a local folder (d:/files), but fails on a network path (//robot/files) after reading about 1,000 - 2,000 files.
Update: the folders are copies of each other.
What causes this problem and how to fix it?
package cef_debug;
import java.io.*;
public class Main {
public static void main(String[] args) throws Throwable {
String folder = args[0];
File[] files = (new File(folder)).listFiles();
String line;
for (int i = 0; i < files.length; i++) {
BufferedReader br = new BufferedReader(new FileReader(files[i]));
while ((line = br.readLine()) != null) {
}
br.close();
}
}
}
I get the following error when reading from a network path (//robot/files):
Exception in thread "main" java.io.IOException: Too many open files
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at java.io.FileReader.<init>(FileReader.java:55)
at cef_debug.Main.main(Main.java:12)
Java Result: 1
Line 12 is the line:
BufferedReader br = new BufferedReader(new FileReader(files[i]));
There is a documented bug for some java versions and some file opens to hit a limit of 2035. It is possible that you might've just hit that.
From the comments:
To clarify the issue, on win32 system there are three ways to open a
file:
1: Using Win32 API
2: Using MFC class framework lib.
3: using C-Library API (open() and fopen())
Other than the third option, i.e. option 1 and 2 have practically no
limitation in opening number of files. The third method is restricted
(for the reason not known to me) to open only approx. 2035 files. That
is why MS JVM is able to open unlimited (practically) files, but SUN
JVM fails after 2035 files (my guess is it is using 3rd method to
open file).
Now, this is an old issue fixed quite some time ago, but it is possible that they would be using the same function on network access, where the bug could still exist.
Even without closing the handle or the stream, windows should be able to open >10000 file handles and keep them open, as demonstrated by this test code in the bug comments:
import java.util.*;
import java.io.*;
// if run with "java maxfiles 10000", will create 10k files in the current folder
public class maxfiles
{
static int count = 0;
static List files = new ArrayList();
public static void main(String args[]) throws Exception
{
for (int n = 0; n < Integer.parseInt(args[0]); n++) {
File f = new File("file" + count++);
//save ref, so not gc'ed
files.add(new PrintStream(new FileOutputStream(f)));
}
Iterator it = files.iterator();
while (it.hasNext()) {
PrintStream out = ( PrintStream) it.next();
out.println("foo");
out.flush();
}
System.out.println("current files open: " + files.size());
} //~main
}
You could test running it on the network share, and report a bug if it fails. You could also try with a different JDK. At least with OpenJDK source, I couldn't see any other calls except WinAPI calls, so I'd try if the behaviour is the same.
Try:
package cef_debug;
import java.io.*;
public class Main {
public static void main(String[] args) throws Throwable {
String folder = args[0];
File[] files = (new File(folder)).listFiles();
String line;
for (int i = 0; i < files.length; i++) {
try {
BufferedReader br = new BufferedReader(new FileReader(files[i]));
while ((line = br.readLine()) != null) {
}
} finally {
try {
if (br != null){
br.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
Someone else's process is creating a CSV file by appending a line at a time to it, as events occur. I have no control over the file format or the other process, but I know it will only append.
In a Java program, I would like to monitor this file, and when a line is appended read the new line and react according to the contents. Ignore the CSV parsing issue for now. What is the best way to monitor the file for changes and read a line at a time?
Ideally this will use the standard library classes. The file may well be on a network drive, so I'd like something robust to failure. I'd rather not use polling if possible - I'd prefer some sort of blocking solution instead.
Edit -- given that a blocking solution is not possible with standard classes (thanks for that answer), what is the most robust polling solution? I'd rather not re-read the whole file each time as it could grow quite large.
Since Java 7 there has been the newWatchService() method on the FileSystem class.
However, there are some caveats:
It is only Java 7
It is an optional method
it only watches directories, so you have to do the file handling yourself, and worry about the file moving etc
Before Java 7 it is not possible with standard APIs.
I tried the following (polling on a 1 sec interval) and it works (just prints in processing):
private static void monitorFile(File file) throws IOException {
final int POLL_INTERVAL = 1000;
FileReader reader = new FileReader(file);
BufferedReader buffered = new BufferedReader(reader);
try {
while(true) {
String line = buffered.readLine();
if(line == null) {
// end of file, start polling
Thread.sleep(POLL_INTERVAL);
} else {
System.out.println(line);
}
}
} catch(InterruptedException ex) {
ex.printStackTrace();
}
}
As no-one else has suggested a solution which uses a current production Java I thought I'd add it. If there are flaws please add in comments.
You can register to get notified by the file system if any change happens to the file using WatchService class. This requires Java7, here the link for the documentation http://docs.oracle.com/javase/tutorial/essential/io/notification.html
here the snippet code to do that:
public FileWatcher(Path dir) {
this.watcher = FileSystems.getDefault().newWatchService();
WatchKey key = dir.register(watcher, ENTRY_MODIFY);
}
void processEvents() {
for (;;) {
// wait for key to be signalled
WatchKey key;
try {
key = watcher.take();
} catch (InterruptedException x) {
return;
}
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
if (kind == OVERFLOW) {
continue;
}
// Context for directory entry event is the file name of entry
WatchEvent<Path> ev = cast(event);
Path name = ev.context();
Path child = dir.resolve(name);
// print out event
System.out.format("%s: %s file \n", event.kind().name(), child);
}
// reset key and remove from set if directory no longer accessible
boolean valid = key.reset();
}
}
This is not possible with standard library classes. See this question for details.
For efficient polling it will be better to use Random Access. It will help if you remember the position of the last end of file and start reading from there.
Use Java 7's WatchService, part of NIO.2
The WatchService API is designed for applications that need to be notified about file change events.
Just to expand on Nick Fortescue's last entry, below are two classes that you can run concurrently (e.g. in two different shell windows) which shows that a given File can simultaneously be written to by one process and read by another.
Here, the two processes will be executing these Java classes, but I presume that the writing process could be from any other application. (Assuming that it does not hold an exclusive lock on the file-are there such file system locks on certain operating systems?)
I have successfully tested these two classes on both Windoze and Linux. I would very much like to know if there is some condition (e.g. operating system) on which they fail.
Class #1:
import java.io.File;
import java.io.FileWriter;
import java.io.PrintWriter;
public class FileAppender {
public static void main(String[] args) throws Exception {
if ((args != null) && (args.length != 0)) throw
new IllegalArgumentException("args is not null and is not empty");
File file = new File("./file.txt");
int numLines = 1000;
writeLines(file, numLines);
}
private static void writeLines(File file, int numLines) throws Exception {
PrintWriter pw = null;
try {
pw = new PrintWriter( new FileWriter(file), true );
for (int i = 0; i < numLines; i++) {
System.out.println("writing line number " + i);
pw.println("line number " + i);
Thread.sleep(100);
}
}
finally {
if (pw != null) pw.close();
}
}
}
Class #2:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
public class FileMonitor {
public static void main(String[] args) throws Exception {
if ((args != null) && (args.length != 0)) throw
new IllegalArgumentException("args is not null and is not empty");
File file = new File("./file.txt");
readLines(file);
}
private static void readLines(File file) throws Exception {
BufferedReader br = null;
try {
br = new BufferedReader( new FileReader(file) );
while (true) {
String line = br.readLine();
if (line == null) { // end of file, start polling
System.out.println("no file data available; sleeping..");
Thread.sleep(2 * 1000);
}
else {
System.out.println(line);
}
}
}
finally {
if (br != null) br.close();
}
}
}
Unfortunately, TailInputStream class, which can be used to monitor the end of a file, is not one of standard Java platform classes, but there are few implementations on the web. You can find an implementation of TailInputStream class together with a usage example on http://www.greentelligent.com/java/tailinputstream.
Poll, either on a consistent cycle or on a random cycle; 200-2000ms should be a good random poll interval span.
Check two things...
If you have to watch for file growth, then check the EOF / byte count, and be sure to compare that and the fileAccess or fileWrite times with the lass poll. If ( > ), then the file has been written.
Then, combine that with checking for exclusive lock / read access. If the file can be read-locked and it has grown, then whatever was writing to it has finished.
Checking for either property alone won't necessarily get you a guaranteed state of written++ and actually done and available for use.
I think this will work only on an English language Windows installation:
System.getProperty("user.home") + "/Desktop";
How can I make this work for non English Windows?
I use a french version of Windows and with it the instruction:
System.getProperty("user.home") + "/Desktop";
works fine for me.
I think this is the same question... but I'm not sure!:
In java under Windows, how do I find a redirected Desktop folder?
Reading it I would expect that solution to return the user.home, but apparently not, and the link in the answer comments back that up. Haven't tried it myself.
I guess by using JFileChooser the solution will require a non-headless JVM, but you are probably running one of them.
This is for Windows only. Launch REG.EXE and capture its output :
import java.io.*;
public class WindowsUtils {
private static final String REGQUERY_UTIL = "reg query ";
private static final String REGSTR_TOKEN = "REG_SZ";
private static final String DESKTOP_FOLDER_CMD = REGQUERY_UTIL
+ "\"HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\"
+ "Explorer\\Shell Folders\" /v DESKTOP";
private WindowsUtils() {}
public static String getCurrentUserDesktopPath() {
try {
Process process = Runtime.getRuntime().exec(DESKTOP_FOLDER_CMD);
StreamReader reader = new StreamReader(process.getInputStream());
reader.start();
process.waitFor();
reader.join();
String result = reader.getResult();
int p = result.indexOf(REGSTR_TOKEN);
if (p == -1) return null;
return result.substring(p + REGSTR_TOKEN.length()).trim();
}
catch (Exception e) {
return null;
}
}
/**
* TEST
*/
public static void main(String[] args) {
System.out.println("Desktop directory : "
+ getCurrentUserDesktopPath());
}
static class StreamReader extends Thread {
private InputStream is;
private StringWriter sw;
StreamReader(InputStream is) {
this.is = is;
sw = new StringWriter();
}
public void run() {
try {
int c;
while ((c = is.read()) != -1)
sw.write(c);
}
catch (IOException e) { ; }
}
String getResult() {
return sw.toString();
}
}
}
or you can use JNA (complete example here)
Shell32.INSTANCE.SHGetFolderPath(null,
ShlObj.CSIDL_DESKTOPDIRECTORY, null, ShlObj.SHGFP_TYPE_CURRENT,
pszPath);
javax.swing.filechooser.FileSystemView.getFileSystemView().getHomeDirectory()
Seems not that easy...
But you could try to find an anwser browsing the code of some open-source projects, e.g. on Koders. I guess all the solutions boil down to checking the HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Desktop path in the Windows registry. And probably are Windows-specific.
If you need a more general solution I would try to find an open-source application you know is working properly on different platforms and puts some icons on the user's Desktop.
You're just missing "C:\\Users\\":
String userDefPath = "C:\\Users\\" + System.getProperty("user.name") + "\\Desktop";
public class Sample {
public static void main(String[] args) {
String desktopPath =System.getProperty("user.home") + "\\"+"Desktop";
String s = "\"" + desktopPath.replace("\\","\\\\") + "\\\\" +"satis" + "\"";
System.out.print(s);
File f = new File(s);
boolean mkdir = f.mkdir();
System.out.println(mkdir);
}
}
there are 2 things.
you are using the wrong slash. for windows it's \ not /.
i'm using RandomAccesFile and File to manage fles and folders, and it requires double slash ( \\ ) to separate the folders name.
Simplest solution is to find out machine name, since this name is only variable changing in path to Desktop folder. So if you can find this, you have found path to Desktop. Following code should do the trick - it did for me :)
String machine_name = InetAddress.getLocalHost().getHostName();
String path_to_desktop = "C:/Documents and Settings/"+machine_name+"/Desktop/";