Solved, In short: the problem was that I wrote to an already closed FileOutputStream
I noticed some strange semantics using the FileOutputStream class.
If I create a FileOutputStream using this code:
try {
File astDumpFile = new File(dumpASTPath);
if(!astDumpFile.exists()) {
astDumpFile.createNewFile();
}
astDumpStream = new FileOutputStream(dumpASTPath);
} catch( IOException e ) {
dumpAST = false;
//throw new IOException("Failed to open file for dumping AST: " + dumpASTPath);
System.out.println("Failed to open file for dumping AST: " + dumpASTPath);
}
at the beginning of the program (astDumpStream is a member variable). Then if I later (~3 seconds later) write string data to the file, i get an IOException: stream closed:
try {
String dotGraph = gpvisitor.getDotGraph();
astDumpStream.write(dotGraph.getBytes("UTF8"));
astDumpStream.flush();
astDumpStream.close();
} catch( IOException e ) {
System.out.println("Failed to dump AST to file: " + e.getMessage());
e.printStackTrace();
}
However if I copy the excact code which I use to create the FileOutputStream to directly before writing to it, it works as expected.
Now I wonder why do I get that exception if I create that object earlier, but not if I create it directly before I use it.
EDIT: The exception:
java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:305)
at MyClass.function(MyClass.java:208)
I just noticed, that even though I get an exception, still some data was written to the file. Interrestingly the first line is written completely, then all following lines except the last line are missing.
If I replace the written String dotGraph with something shorter everything is written correctly, however I still get that exception.
EDIT: Environment Information:
[~]> lsb_release -a
Distributor ID: Debian
Description: Debian GNU/Linux testing (wheezy)
Release: testing
Codename: wheezy
[~]> java -version
java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)
The only reason to get an IOException complaining that the stream is closed is because the stream was closed. You'll have to trace through your code to find out where that's happening. Some not-so-obvious places include calls into other methods and finally blocks of try statements. Another thing to look for is reassignment of the variable astDumpStream to a different stream (that was closed before the IOException was raised—possibly even before the first assignment to astDumpStream).
The time doesn't seem relevant unless you have a separate thread that might close the stream after a delay.
The only way this can happen if is the close() function gets called more than once. My guess is that for some reason, the second block of code is being called more than once.
To prevent indentation errors, there are two good pieces of advice I've received:
Always indent consistently. Preferably use a tool that does this for you (like Eclipse).
Always use curly braces, even if you don't think you need them. This helps prevent quite a few minor bugs that take forever to find, so the extra half-second it takes to type each one is more than made up by the hours you don't spend looking for these bugs.
To Second Tedd - in case you happen to use a nested try with resource block and using stream outside tryblock - you could run in this situation as well because once the control comes out of nested try with resource block the stream will be closed.
Related
On Windows, NUL is the null output device similar to /dev/null on Linux.
With Oracle Java 8 Update 331, trying to get a new FileOutputStream("NUL:") throws an exception. Previously (Java 8u321) it worked fine.
The problem seems to be the colon:
new FileOutputStream("NUL") - OK
new FileOutputStream("NUL:") - exception
Can anyone point me to docs or JDK sources regarding this change? I can't change the code itself because it is in a 3rd party lib (xnio-api).
try
{
new FileOutputStream("NUL:");
System.out.println("OK");
}
catch (FileNotFoundException e)
{
System.out.println(e);
}
I suspect this is the offending change.
Apparently it tries to avoid accessing ADS (alternate data streams), but seems to "accidentally" also prevent access to device-files like this.
If that's correct, then you can try setting the system property jdk.io.File.enableADS to true to re-enable the old behaviour.
I made this simple piece of code to test ProcessBuilder:
#SpringBootApplication
public class TerminalDemoApplication {
public static void main(String[] args) {
SpringApplication.run(TerminalDemoApplication.class, args);
try {
System.out.println("hello");
Process process = new ProcessBuilder("python", "--version").start();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
int exitCode = process.waitFor();
System.out.println("\nExited with error code : " + exitCode);
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
}
}
It works in Windows (returns python version of my system) but the same code in my macbook returns end of line, so basically empty. ¿This needs further configuration according to the OS? ¿why is this happening?
What error code are you getting?
There are (at least) two explanations; that error code would indicate which one it is.
You're not running python, or running 'the wrong' python
This would mean you are getting an error code of some sort, or an exception.
The likely reason for this is a path issue.
Running python, just like that - as in, no path information at all, is nominally neccessarily broken: That's just not how your OS works, it has no idea what to do with this path.
It's a bashism (as in, the shell does it, not the OS) to interpret such a command as 'oh, actually, go through each listed entry in the $PATH environment variable, and stick that path in front of this name, see if you find an executable there. If you do, run that and stop).
Java mostly doesn't engage in any bashisms. But, in a few bizarre places, it does - it tries to do basic space splitting when you use the single-string version of new ProcessBuilder), which is a shellism, and it does attempt to do basic PATH lookup, but that's about where it ends. It won't do * unpacking, which on windows is an OS-level thing but on posix systems is a shellism.
I strongly, strongly advise you to avoid java's basic shellisms. It's unreliable and highly OS-specific.
So: Always pass arguments explicitly (good, you're doing that), always use ProcessBuilder (good, you're doing that), never use relative paths (that's where you're going wrong).
It's going to the error stream instead
processes on OSes are generally hooked up to 3 pipes, not 2. There's the 'standard in', the 'standard out' and the 'standard err'. Your own java process exposes these as System.out, in, and err.
In linux in particular, it is common to redirect standard out of some process to a file or another process.
This means that standard err naturally has the property that it tends to emit to the console, even if you are redirecting things. In other words, the terms 'standard out' and 'standard err' are really stupid names on posix. The much better naming would be 'standard process output' and 'standard process messages'.
Asking python to print its version is in a bit of a limbo scenario. The string "Python v3.0.1" or whatnot is certainly not an error, but it's a bit dubious if one should consider this as 'the output of the process'. It's likely that the authors of the python tool consider it more 'some information I should print to you, even if you are redirecting things.
Thus, my guess is that this version is heading out to standard err instead.
You can solve this in two ways: Either read from standard err as well, or, use process builder's features: You can ask it to bundle up standard out and standard err into a single stream (.redirectErrorStream(true)).
I would expect the exit code to be 0 if this explanation is the correct one.
I find myself having to explicitly call System.gc() in my Groovy scripts to prevent errors like the one below. Why doesn't the garbage collector do this for me? Is there something I can do to cause it to garbage collect to prevent these errors (maybe JAVA_OPTS)?
Caught: java.util.concurrent.ExecutionException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Cannot run program "ls": java.io.IOException: error=24, Too many open files
at groovyx.gpars.GParsPool.runForkJoin(GParsPool.groovy:305)
at UsageAnalyzer$_run_closure2_closure6.doCall(UsageAnalyzer.groovy:36)
at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy:170)
at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy)
at groovyx.gpars.GParsPool.withExistingPool(GParsPool.groovy:169)
at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:141)
at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:117)
at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:96)
at UsageAnalyzer$_run_closure2.doCall(<removed>)
at UsageAnalyzer.run(<removed>)
This stack trace is from a parallel program but it happens in sequential programs as well.
As you're using groovy, you can use the convenient methods such as File.withReader(), File.withWriter(), File.withInputStream(), InputStream.withStream() to ensure resources get closed cleanly. This is less cumbersome than using Java's try .. finally idiom, as there's not need to explicitly call close(), or declare a variable outside the try block.
e.g. to read from a file.
File f = new File('/mumble/mumble/')
f.withReader{ r ->
// do stuff with reader here
}
Definitely look for any place you open files or streams and make sure you close them. It's often beneficial to wrap them like this:
final InputStream in = ...;
try
{
// Do whatever.
}
finally
{
// Will always close the stream, regardless of exceptions, return statements, etc.
in.close();
}
This question already has answers here:
Alternative to File.exists() in Java
(6 answers)
Closed 2 years ago.
I am working on a Java program that requires to check the existence of files.
Well, simple enough, the code make use calls to File.exists() for checking file existence. And the problem I have is, it reports false positive. That means the file does not actually exist but exists() method returns true. No exception was captured (at least no exception like "Stale NFS handle"). The program even managed to read the file through InputStream, getting 0 bytes as expected and yet no exception. The target directory is a Linux NFS. And I am 100% sure that the file being looked for never exists.
I know there are known bugs (kind of API limitation) exist for java.io.File.exists(). So I've then added another way round by checking file existence using Linux command ls. Instead of making call to File.exists() the Java code now runs a Linux command to ls the target file. If exit code is 0, file exists. Otherwise, file does not exist.
The number of times the issue is hit seems to be reduced with the introduction of the trick, but still pops. Again, no error was captured anywhere (stdout this time). That means the problem is so serious that even native Linux command won't fix for 100% of the time.
So there are couple of questions around:
I believe Java's well known issue on File.exists() is about reporting false negative. Where file was reported to not exist but in fact does exist. As the API does not throws IOException for File.exists(), it choose to swallow the Exception in the case calls to OS's underlying native functions failed e.g. NFS timeout. But then this does not explain the false positive case I am having, given that the file never exist. Any throw on this one?
My understanding on Linux ls exit code is, 0 means okay, equivalent to file exists. Is this understanding wrong? The man page of ls is not so clear on explaining the meaning of exit code: Exit status is 0 if OK, 1 if minor problems, 2 if serious trouble.
All right, back to subject. Any surefire way to check File existence with Java on Linux? Before we see JDK7 with NIO2 officially released.
Here is a JUnit test that shows the problem and some Java Code that actually tries to read the file.
The problem happens e.g. using Samba on OSX Mavericks. A possible reason
is explaned by the statement in:
http://appleinsider.com/articles/13/06/11/apple-shifts-from-afp-file-sharing-to-smb2-in-os-x-109-mavericks
It aggressively caches file and folder properties and uses opportunistic locking to enable better caching of data.
Please find below a checkFile that will actually attempt to read a few bytes and forcing a true file access to avoid the caching misbehaviour ...
JUnit test:
/**
* test file exists function on Network drive replace the testfile name and ssh computer
* with your actual environment
* #throws Exception
*/
#Test
public void testFileExistsOnNetworkDrive() throws Exception {
String testFileName="/Volumes/bitplan/tmp/testFileExists.txt";
File testFile=new File(testFileName);
testFile.delete();
for (int i=0;i<10;i++) {
Thread.sleep(50);
System.out.println(""+i+":"+OCRJob.checkExists(testFile));
switch (i) {
case 3:
// FileUtils.writeStringToFile(testFile, "here we go");
Runtime.getRuntime().exec("/usr/bin/ssh phobos /usr/bin/touch "+testFileName);
break;
}
}
}
checkExists source code:
/**
* check if the given file exists
* #param f
* #return true if file exists
*/
public static boolean checkExists(File f) {
try {
byte[] buffer = new byte[4];
InputStream is = new FileInputStream(f);
if (is.read(buffer) != buffer.length) {
// do something
}
is.close();
return true;
} catch (java.io.IOException fnfe) {
}
return false;
}
JDK7 was released a few months ago. There are exists and notExists methods in the Files class but they return a boolean rather than throwing an exception. If you really want an exception then use FileSystems.getDefault().provider().checkAccess(path) and it will throw an exception if the file does not exist.
If you need to be robust, try to read the file - and fail gracefully if the file is not there (or there is a permission or other problem). This applies to any other language than Java as well.
The only safe way to tell if the file exist and you can read from it is to actually read a data from the file. Regardless of a file system - local, or remote. The reason is a race condition which can occur right after you get success from checkAccess(path): check, then open file, and you find it suddenly does not exist. Some other thread (or another remote client) may have removed it, or has acquired an exclusive lock. So don't bother checking access, but rather try to read the file. Spending time in running ls just makes race condition window easier to fit.
We have several JUnit tests that rely on creating new files and reading them. However there are issues with the files not being created properly. But this fault comes and goes.
This is the code:
#Test
public void test_3() throws Exception {
// Deletes files in tmp test dir
File tempDir = new File(TEST_ROOT, "tmp.dir");
if (tempDir.exists()) {
for (File f : tempDir.listFiles()) {
f.delete();
}
} else {
tempDir.mkdir();
}
File file_1 = new File(tempDir, "file1");
FileWriter out_1 = new FileWriter(file_1);
out_1.append("# File 1");
out_1.close();
File file_2 = new File(tempDir, "file2");
FileWriter out_2 = new FileWriter(file_2);
out_2.append("# File 2");
out_2.close();
File file_3 = new File(tempDir, "fileXXX");
FileWriter out_3 = new FileWriter(file_3);
out_3.append("# File 3");
out_3.close();
....
The fail is that the second file object, file_2, never gets created. Sometimes. Then when we try to write to it a FileNotFoundException is thrown
If we run only this testcase, everything works fine.
If we run this testfile with some ~40 testcases, it can both fail and work depending on the current lunar cycle.
If we run the entire testsuite, consisting of some 10*40 testcases, it always fails.
We have tried
adding sleeps (5sec) after new File, nothing
adding while loop until file_2.exists() is true but the loop never stopped
catching SecurityException, IOException and even throwable when we do the New File(..), but caught nothing.
At one point we got all files to be created, but file_2 was created before file_1 and a test that checked creation time failed.
We've also tried adding file_1.createNewFile() and it always returns true.
So what is going on? How can we make tests that depend on actual files and always be sure they exist?
This has been tested in both java 1.5 and 1.6, also in Windows 7 and Linux. The only difference that can be observed is that sometimes a similar testcase before fails, and sometimes file_1 isn't created instead
Update
We tried a new variation:
File file_2 = new File(tempDir, "file2");
while (!file_2.canRead()) {
Thread.sleep(500);
try {
file_2.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
This results in alot of Exceptions of the type:
java.io.IOException: Access is denied
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:883)
... but eventually it works, the file is created.
Are there multiple instances of your program running at once?
Check for any extra instances of javaw.exe running. If multiple programs have handles to the same file at once, things can get very wonky very quickly.
Do you have antivirus software or anything else running that could be getting in the way of file creation/deletion, by handle?
Don't hardcode your file names, use random names. It's the only way to abstract yourself from the various external situations that can occur (multiple access to the same file, permissions, file system error, locking problems, etc...).
One thing for sure: using sleep() or retrying is guaranteed to cause weird errors at some point in the future, avoid doing that.
I did some googling and based on this lucene bug and this board question seems to indicate that there could be an issue with file locking and other processes using the file.
Since we are running this on ClearCase it seems plausible that ClearCase does some indexing or something similar when the files are being created. Adding loops that repeat until the file is readable solved the issue, so we are going with that. Very ugly solution though.
Try File#createTempFile, this at least guarantees you that there are no other files by the same name that would still hold a lock.