I have recently scanned our code base with Fortify and I'm confused as to why it's flagging certain issues. One issue that I'm facing the issue is with releasing a resource.
Here is a snippet for context.
String someLocation = getPathToTheLocation(); //gives location
try (InputStream in = someLocation == null ? Thread.currentThread().getContextClassLoader()
.getResourceAsStream("someFile.xml") : new FileInputStream(new File(someLocation))) {
/// Do Something
}
Fortify complains that the method that has this try-with-resources block fails to release a system resource allocated by FileInputStream(). Doesn't the try-with-resources help me close the FileInputStream automatically?
Just assuming that Fortify doesn't recognize the try-with-resources paradigm, I refrained from using it and did it the regular way.
String someLocation = getPathToTheLocation(); //gives location
InputStream in = null;
try {
in = someLocation == null ? Thread.currentThread().getContextClassLoader().getResourceAsStream("someFile.xml")
: new FileInputStream(new File(someLocation));
//Do Something.
} finally {
assert in != null;
try {
in.close();
} catch (IOException ioe) {
throw new IllegalStateException("Could not close input stream.", ioe);
}
}
And yet it still complains about the resource being unreleased. What could be the actual issue here that I fail to recognize?
I think everything okay with the code, the problem in Fortify and maybe you should propose an Issue for it. There was quite the same issue with Idea - https://stackoverflow.com/a/34655863/5790043.
I have a log file, each of whose lines I want to use to create a LogMessage object. I want to stream the lines from the file, and map each one to a new LogMessage. The code below works, but Eclipse issues a warning:
Resource leak: 'lineStream' is never closed
public static Stream<LogMessage> streamSingleLineLogMessages(Path path) {
try {
Stream<String> lineStream = Files.lines(path, StandardCharsets.ISO_8859_1);
Stream<LogMessage> logMessageStream =
lineStream.map(message -> new LogMessage(path, message));
logMessageStream.onClose(lineStream::close);
return logMessageStream;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
If I add a finally block, and close it in there, then the stream is closed when the method returns (I think). In any case, it is closed by the time I come to use it.
So what it is the best way to ensure that the inner stream is closed? Or perhaps the code is correct as it is, but Eclipse doesn't realize it?
You actually shouldn't need any of this and have instead:
public static Stream<LogMessage> streamSingleLineLogMessages(Path path) throws IOException {
return Files.lines(path, StandardCharsets.ISO_8859_1)
.map(message -> new LogMessage(path, message));
}
The method Files.lines(path, cs) returns a Stream<Path> that already has a close handler closing the internal BufferedReader. When this Stream is mapped to a Stream<LogMessage>, the close handlers are kept.
This means that for the new Stream<LogMessage>, there is already a close handler closing the BufferedReader, so you don't need to add it yourself.
You just need to make sure that when you use this method, you wrap it property inside a try-with-resources construct:
try (Stream<LogMessage> messageStream = streamSingleLineLogMessages(path)) {
// do something with the stream
}
I just wanted to see if there was a better way I should be handling this. My understanding of streams is that as long as you close a stream, any streams encapsulated within it will be closed which is why I only close TarArchiveOutputStream in finally. If I get FileNotFound on the rawDir or archiveDir I want to log it, otherwise anything else I want to throw.
public static void createTarGzOfDirectory(File rawDir, File archiveFile) throws IOException {
FileOutputStream fOut = null;
BufferedOutputStream bOut = null;
GzipCompressorOutputStream gzOut = null;
TarArchiveOutputStream tOut = null;
try {
fOut = new FileOutputStream(archiveFile);
bOut = new BufferedOutputStream(fOut);
gzOut = new GzipCompressorOutputStream(bOut);
tOut = new TarArchiveOutputStream(gzOut);
addFileToTarGz(tOut, rawDir, "");
} catch (FileNotFoundException e) {
log.error("File not found: " + e);
} finally {
if(tOut != null) {
tOut.finish();
tOut.close();
}
}
Any other considerations or thoughts on improving things?
My understanding of streams is that as long as you close a stream, any streams encapsulated within it will be closed ...
That is correct.
However, your code is (effectively) assuming that if tOut is null, then none of the other streams in the chain have been created. That's a somewhat dodgy assumption. Consider this sequence:
The FileOutputStream is created and is assigned to fOut.
The BufferedOutputStream is created and is assigned to bOut.
The GzipCompressorOutputStream constructor throws an exception or error. (Maybe the heap is full ...).
The catch is skipped ... wrong exception.
The finally checks tOut, finds it is null, and does nothing.
Net result: we've leaked the file descriptor / channel held by the FileOUtputStream.
The key to getting this example (absolutely) right is to understand which of those stream objects holds the critical resources, and ensuring that THAT stream gets closed. The other streams that don't hold resources don't have to be closed.
} finally {
if (fOut != null) {
fOut.close();
}
}
The other point is that you need to move the tOut.finish() call into the try block after the addFileToTarGz call.
If the addFileToTarGz call throws an exception, or if you don't get that far, the finish call is a waste of time.
The finish call will attempt to write the index to the archive, and THAT could throw an IOException. If this happens in the finally block, then any following code in the finally block to close the stream chain won't get executed ... and a file descriptor will be leaked.
Although it would look ugly and is,maybe, unlikely to be the case, you should close them all in cascade. Yes, if you close the TarArchiveOutputStream, it is supposed to close the underlyning streams. But, depending on the implementation, it may not always be the case. Moreover, and probably mainly, if one of the intermediate constructors throw an exception, tOut will be null, but the other ones may not be. Meaning that your streams are opened but your did not close any.
You could chain all your constructors together like so:
tOut = new TarArchiveOutputStream(new GzipCompressorOutputStream(new BufferedOutputStream(new FileOutputStream(archiveFile))));
And save yourself 6 lines of initialization and 3 local variables for debugging. Not everyone likes chaining things that way - I personally find it more readable but the rest of your team may prefer it your way.
As far as closing the stream, it looks correct to me.
My code makes use of BufferedReader to read from a file [main.txt] and PrintWriter to write to a another temp [main.temp] file. I close both the streams and yet I was not able to call delete() method on the File object associated with [main.txt]. Only after calling System.gc() after closing both the stream was I able to delete the File object.
public static boolean delete (String str1, String str2, File FileLoc)
{
File tempFile = null;
BufferedReader Reader = null;
PrintWriter Writer = null;
try
{
tempFile = new File (FileLoc.getAbsolutePath() + ".tmp");
Reader = new BufferedReader(new FileReader(FileLoc));
Writer = new PrintWriter(new FileWriter(tempFile));
String lsCurrLine = null;
while((lsCurrLine = Reader.readLine()) != null)
{
// ...
// ...
if (true)
{
Writer.println(lsCurrLine);
Writer.flush();
}
}
Reader.close();
Writer.close();
System.gc();
}
catch(FileNotFoundException loFileExp)
{
System.out.println("\n File not found . Exiting");
return false;
}
catch(IOException loFileExp)
{
System.out.println("\n IO Exception while deleting the record. Exiting");
return false;
}
}
Is this reliable? Or is there a better fix?
#user183717 - that code you posted is clearly not all of the relevant code. For instance, those "..."'s and the fact that File.delete() is not actually called in that code.
When a stream object is garbage collected, its finalizer closes the underlying file descriptor. So, the fact that the delete only works when you added the System.gc() call is strong evidence that your code is somehow failing to close some stream for the file. It may well be a different stream object to the one that is opened in the code that you posted.
Properly written stream handling code uses a finally block to make sure that streams get closed no matter what. For example:
Reader reader = new BufferedReader(new FileReader(file));
try {
// do stuff
} finally {
try {
reader.close();
} catch (IOException ex) {
// ...
}
}
If you don't follow that pattern or something similar, there's a good chance that there are scenarios where streams don't always get closed. In your code for example, if one of the read or write calls threw an exception you'd skip past the statements that closed the streams.
Is this [i.e. calling System.gc();] reliable?
No.
The JVM may be configured to ignore your application's gc() call.
There's no guarantee that the lost stream will be unreachable ... yet.
There's no guarantee that calling System.gc() will notice that the stream is unreachable. Hypothetically, the stream object might be tenured, and calling System.gc() might only collect the Eden space.
Even if the stream is found to be unreachable by the GC, there's no guarantee that the GC will run the finalizer immediately. Hypothetically, running the finalizers can be deferred ... indefinitely.
Or is there a better fix ?
Yes. Fix your application to close its streams properly.
try using java.io.File library. here the simple sample:
File f = new File("file path or file name");
f.delete();
When you say you "close both the streams" you mean the BufferedReader and the PrintWriter?
You should only need to close the BufferedReader before the delete will work, but you also need to close the underlying stream; normally calling BufferedReader.close() will do that. It sounds like you think you are closing the stream but you aren't actually succeeding.
One problem with your code: you don't close the streams if exceptions occur. It's usually best to close the streams in a finally block.
Also, the code you posted doesn't use File.delete() anywhere? And what exactly do the ... lines do - are they re-assinging Reader to a new stream by any chance?
try using the apache commons io
http://commons.apache.org/io/description.html
I'm trying to delete a file, after writing something in it, with FileOutputStream. This is the code I use for writing:
private void writeContent(File file, String fileContent) {
FileOutputStream to;
try {
to = new FileOutputStream(file);
to.write(fileContent.getBytes());
to.flush();
to.close();
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
As it is seen, I flush and close the stream, but when I try to delete, file.delete() returns false.
I checked before deletion to see if the file exists, and: file.exists(), file.canRead(), file.canWrite(), file.canExecute() all return true. Just after calling these methods I try file.delete() and returns false.
Is there anything I've done wrong?
Another bug in Java. I seldom find them, only my second in my 10 year career. This is my solution, as others have mentioned. I have nether used System.gc(). But here, in my case, it is absolutely crucial. Weird? YES!
finally
{
try
{
in.close();
in = null;
out.flush();
out.close();
out = null;
System.gc();
}
catch (IOException e)
{
logger.error(e.getMessage());
e.printStackTrace();
}
}
It was pretty odd the trick that worked. The thing is when I have previously read the content of the file, I used BufferedReader. After reading, I closed the buffer.
Meanwhile I switched and now I'm reading the content using FileInputStream. Also after finishing reading I close the stream. And now it's working.
The problem is I don't have the explanation for this.
I don't know BufferedReader and FileOutputStream to be incompatible.
I tried this simple thing and it seems to be working.
file.setWritable(true);
file.delete();
It works for me.
If this does not work try to run your Java application with sudo if on linux and as administrator when on windows. Just to make sure Java has rights to change the file properties.
Before trying to delete/rename any file, you must ensure that all the readers or writers (for ex: BufferedReader/InputStreamReader/BufferedWriter) are properly closed.
When you try to read/write your data from/to a file, the file is held by the process and not released until the program execution completes. If you want to perform the delete/rename operations before the program ends, then you must use the close() method that comes with the java.io.* classes.
As Jon Skeet commented, you should close your file in the finally {...} block, to ensure that it's always closed. And, instead of swallowing the exceptions with the e.printStackTrace, simply don't catch and add the exception to the method signature. If you can't for any reason, at least do this:
catch(IOException ex) {
throw new RuntimeException("Error processing file XYZ", ex);
}
Now, question number #2:
What if you do this:
...
to.close();
System.out.println("Please delete the file and press <enter> afterwards!");
System.in.read();
...
Would you be able to delete the file?
Also, files are flushed when they're closed. I use IOUtils.closeQuietly(...), so I use the flush method to ensure that the contents of the file are there before I try to close it (IOUtils.closeQuietly doesn't throw exceptions). Something like this:
...
try {
...
to.flush();
} catch(IOException ex) {
throw new CannotProcessFileException("whatever", ex);
} finally {
IOUtils.closeQuietly(to);
}
So I know that the contents of the file are in there. As it usually matters to me that the contents of the file are written and not if the file could be closed or not, it really doesn't matter if the file was closed or not. In your case, as it matters, I would recommend closing the file yourself and treating any exceptions according.
There is no reason you should not be able to delete this file. I would look to see who has a hold on this file. In unix/linux, you can use the lsof utility to check which process has a lock on the file. In windows, you can use process explorer.
for lsof, it's as simple as saying:
lsof /path/and/name/of/the/file
for process explorer you can use the find menu and enter the file name to show you the handle which will point you to the process locking the file.
here is some code that does what I think you need to do:
FileOutputStream to;
try {
String file = "/tmp/will_delete.txt";
to = new FileOutputStream(file );
to.write(new String("blah blah").getBytes());
to.flush();
to.close();
File f = new File(file);
System.out.print(f.delete());
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
It works fine on OS X. I haven't tested it on windows but I suspect it should work on Windows too. I will also admit seeing some unexpected behavior on Windows w.r.t. file handling.
If you are working in Eclipse IDE, that could mean that you haven't close the file in the previous launch of the application. When I had the same error message at trying to delete a file, that was the reason. It seems, Eclipse IDE doesn't close all files after termination of an application.
Hopefully this will help. I came across similar problem where i couldn't delete my file after my java code made a copy of the content to the other folder. After extensive googling, i explicitly declared every single file operation related variables and called the close() method of each file operation object, and set them to NULL. Then, there is a function called System.gc(), which will clear up the file i/o mapping (i'm not sure, i just tell what is given on the web sites).
Here is my example code:
public void start() {
File f = new File(this.archivePath + "\\" + this.currentFile.getName());
this.Copy(this.currentFile, f);
if(!this.currentFile.canWrite()){
System.out.println("Write protected file " +
this.currentFile.getAbsolutePath());
return;
}
boolean ok = this.currentFile.delete();
if(ok == false){
System.out.println("Failed to remove " + this.currentFile.getAbsolutePath());
return;
}
}
private void Copy(File source, File dest) throws IOException {
FileInputStream fin;
FileOutputStream fout;
FileChannel cin = null, cout = null;
try {
fin = new FileInputStream(source);
cin = fin.getChannel();
fout = new FileOutputStream(dest);
cout = fout.getChannel();
long size = cin.size();
MappedByteBuffer buf = cin.map(FileChannel.MapMode.READ_ONLY, 0, size);
cout.write(buf);
buf.clear();
buf = null;
cin.close();
cin = null;
fin.close();
fin = null;
cout.close();
cout = null;
fout.close();
fout = null;
System.gc();
} catch (Exception e){
this.message = e.getMessage();
e.printStackTrace();
}
}
the answer is when you load the file, you need apply the "close" method, in any line of code, works to me
There was a problem once in ruby where files in windows needed an "fsync" to actually be able to turn around and re-read the file after writing it and closing it. Maybe this is a similar manifestation (and if so, I think a windows bug, really).
None of the solutions listed here worked in my situation. My solution was to use a while loop, attempting to delete the file, with a 5 second (configurable) limit for safety.
File f = new File("/path/to/file");
int limit = 20; //Only try for 5 seconds, for safety
while(!f.delete() && limit > 0){
synchronized(this){
try {
this.wait(250); //Wait for 250 milliseconds
} catch (InterruptedException e) {
e.printStackTrace();
}
}
limit--;
}
Using the above loop worked without having to do any manual garbage collecting or setting the stream to null, etc.
The problem could be that the file is still seen as opened and locked by a program; or maybe it is a component from your program that it had been opened in, so you have to ensure you use the dispose() method to solve that problem.
i.e. JFrame frame;
....
frame.dispose();
You have to close all of the streams or use try-with-resource block
static public String head(File file) throws FileNotFoundException, UnsupportedEncodingException, IOException
{
final String readLine;
try (FileInputStream fis = new FileInputStream(file);
InputStreamReader isr = new InputStreamReader(fis, "UTF-8");
LineNumberReader lnr = new LineNumberReader(isr))
{
readLine = lnr.readLine();
}
return readLine;
}
if file.delete() is sending false then in most of the cases your Bufferedreader handle will not be closed. Just close and it seems to work for me normally.
I had the same problem on Windows. I used to read the file in scala line by line with
Source.fromFile(path).getLines()
Now I read it as a whole with
import org.apache.commons.io.FileUtils._
// encoding is null for platform default
val content=readFileToString(new File(path),null.asInstanceOf[String])
which closes the file properly after reading and now
new File(path).delete
works.
FOR Eclipse/NetBeans
Restart your IDE and run your code again this is only trick work for me after one hour long struggle.
Here is my code:
File file = new File("file-path");
if(file.exists()){
if(file.delete()){
System.out.println("Delete");
}
else{
System.out.println("not delete");
}
}
Output:
Delete
Another corner case that this could happen: if you read/write a JAR file through a URL and later try to delete the same file within the same JVM session.
File f = new File("/tmp/foo.jar");
URL j = f.toURI().toURL();
URL u = new URL("jar:" + j + "!/META-INF/MANIFEST.MF");
URLConnection c = u.openConnection();
// open a Jar entry in auto-closing manner
try (InputStream i = c.getInputStream()) {
// just read some stuff; for demonstration purposes only
byte[] first16 = new byte[16];
i.read(first16);
System.out.println(new String(first16));
}
// ...
// i is now closed, so we should be good to delete the jar; but...
System.out.println(f.delete()); // says false!
Reason is that the internal JAR file handling logic of Java, tends to cache JarFile entries:
// inner class of `JarURLConnection` that wraps the actual stream returned by `getInputStream()`
class JarURLInputStream extends FilterInputStream {
JarURLInputStream(InputStream var2) {
super(var2);
}
public void close() throws IOException {
try {
super.close();
} finally {
// if `getUseCaches()` is set, `jarFile` won't get closed!
if (!JarURLConnection.this.getUseCaches()) {
JarURLConnection.this.jarFile.close();
}
}
}
}
And each JarFile (rather, the underlying ZipFile structure) would hold a handle to the file, right from the time of construction up until close() is invoked:
public ZipFile(File file, int mode, Charset charset) throws IOException {
// ...
jzfile = open(name, mode, file.lastModified(), usemmap);
// ...
}
// ...
private static native long open(String name, int mode, long lastModified,
boolean usemmap) throws IOException;
There's a good explanation on this NetBeans issue.
Apparently there are two ways to "fix" this:
You can disable the JAR file caching - for the current URLConnection, or for all future URLConnections (globally) in the current JVM session:
URL u = new URL("jar:" + j + "!/META-INF/MANIFEST.MF");
URLConnection c = u.openConnection();
// for only c
c.setUseCaches(false);
// globally; for some reason this method is not static,
// so we still need to access it through a URLConnection instance :(
c.setDefaultUseCaches(false);
[HACK WARNING!] You can manually purge the JarFile from the cache when you are done with it. The cache manager sun.net.www.protocol.jar.JarFileFactory is package-private, but some reflection magic can get the job done for you:
class JarBridge {
static void closeJar(URL url) throws Exception {
// JarFileFactory jarFactory = JarFileFactory.getInstance();
Class<?> jarFactoryClazz = Class.forName("sun.net.www.protocol.jar.JarFileFactory");
Method getInstance = jarFactoryClazz.getMethod("getInstance");
getInstance.setAccessible(true);
Object jarFactory = getInstance.invoke(jarFactoryClazz);
// JarFile jarFile = jarFactory.get(url);
Method get = jarFactoryClazz.getMethod("get", URL.class);
get.setAccessible(true);
Object jarFile = get.invoke(jarFactory, url);
// jarFactory.close(jarFile);
Method close = jarFactoryClazz.getMethod("close", JarFile.class);
close.setAccessible(true);
//noinspection JavaReflectionInvocation
close.invoke(jarFactory, jarFile);
// jarFile.close();
((JarFile) jarFile).close();
}
}
// and in your code:
// i is now closed, so we should be good to delete the jar
JarBridge.closeJar(j);
System.out.println(f.delete()); // says true, phew.
Please note: All this is based on Java 8 codebase (1.8.0_144); they may not work with other / later versions.