We Keep Coding

File cannot be deleted because the JVM holds it - a tricky one

My post got a little too long, sorry. Here is a summary:
File on disk cannot be deleted ("the JVM holds the file" error). both when deleting from the java code and when trying to manually delete the file from windows.
All streams to that file are closed and set to null. All file objects set to null.
The program does nothing at that point; but waiting 30 minutes allows me to deleted the file from windows. Weird. Is the file not used by java anymore? Plus, since nothing happens in the program, it indicates it cannot be some stream I forgot (plus, I triple checked nothing is open).
Invoking System.gc() seemed to work when files were small. Did not help when they got to about 20MB.
[EDIT2] - I tried writing some basic code to explain, but its tricky. I am sorry, I know it's difficult to answer like that. I can however write how I open and close streams, of course:
BufferedWriter bw = new BufferedWriter(new FileWriter(new File("C:\\folder\\myFile.txt")));
for(int i = 0; i < 10; i++)
{
bw.write("line " + i);
bw.newLine();
}
bw.close();
bw = null;
If I've used a file object:
File f = new File("C:\\folder\\myFile.txt");
// use it...
f = null;
Basic code, I know. But this is essentially what I do.
I know for a fact I've closed all streams in this exact way.
I know for a fact that nothing happens in the program in that 30-minutes interval in which I cannot delete the file, until I somehow magically can.
thank you for your input even without the coherent code.
I appreciate that.
Sorry for not providing any specific code here, since I can't pinpoint the problem (not exactly specific-code related). In any case, here is the thing:
I have written a program which reads, writes and modifies files on disk. For several reasons, the handling of the read/write is done in a different thread, which is constantly operating.
At some point, I am terminating the "read/write" thread, keeping only the main thread - it waits for input from a socket, totally unrelated to the file, and does nothing. Then, I try to delete the file (using either File.delete(), even tried nio.Files delete option).
The thing is - and it's very weird - sometimes it works, sometimes it doesn't. Even manually, going to the folder and trying to delete the file via windows, gives me the "The file is open by the JVM" message.
Now, I am well aware that keeping references from all kinds of streams to the file prevents me from deleting it. Well past that by now :)
I have made sure that all streams are closed. I even set their values to null, including any "File" objects I have used (even though it shouldn't make any difference). All set to null, all closed. And the thread which generates all of them - the "read/write" thread - well, it's terminated since it got the the end of its run() method.
Usually, if I wait about 30 minutes, while the JVM still operates, I can delete the file manually from windows. The error magically disappears. When the JVM is closed, I can always delete the file right away.
I am lost here. Tried specifically invoking System.gc() before trying to delete the file, even called it like 10 times (not that it should matter). Sometimes it helped, but on other occasions, for example, when the file got larger (say 20MB), that didn't help.
What am I missing here?
Obviously, this couldn't be my implicit fault (not closing some stream), since the read/write thread is dead, the main thread awaits something unrelated (so the program is at a "standstill"), I have explicitly closed all streams, even nullified the references (inStream = null), invoked the garbage collector.
What am I missing? Why is the file "deletable" after 30 minutes (nothing happens at that time - not something in my code). Am I missing some gentle reference/garbage collection thingy?

What you're doing just calls for problems. You say that "if an IOexception occurred, it is printed immediately" and it may be true, but given that something inexplicable happens, let's better doubt it.
I'd first ensure that everything gets always closed, and then I'd care about related logic (logging, exiting, ...).
Anyway, what you did is not how resources should be managed. The answer above is not exactly correct either. Anyway, try-with-resources is (besides #lombok.Cleanup) about the only way, clearly showing that nothing gets ever left open. Anything else is more complicated and more error-prone. I'd strongly recommend using it everywhere. This may be quite some work, but it also forces you to re-inspect all the critical code pieces.
Things like nullifying references and calling the GC should not help... and if they seem to do, it may be a chance.
Some ideas:
Are you using memory mapped files?
Are you sure System.exit is not disabled by a security manager?
Are you running an antivirus? They love to scan files just after they get written.
Btw., locking files is one reason why the WOW never started for me. Sometimes the locks persisted long after the culprit was gone, at least according to tools I could use.

Are you closing your streams in a try...finally or try(A a = new A()) block? If not the streams may not be closed.
I would strongly recommend using either Automatic Resource Block Management ( try(A a = new A()) ) or a try...finally block for all external resources.
try(BufferedWriter br = new BufferedWriter(new FileWriter(new File("C:\\folder\\myFile.txt")));
for(int i = 0; i < 10; i++)
{
br.write("line " + i);
br.newLine();
})

Is there any harm in failing to close a file when a Java program terminates?

When my program starts, it opens a file and writes to it periodically. (It's not a log file; it's one of the outputs of the program.) I need to have the file available for the length of the program, but I don't need to do anything in particular to end the file; just close it.
I gather that for file I/O in Java I'm supposed to implement AutoCloseable and wrap it in a try-with-resources block. However, because this file is long-lived, and it's one of a few outputs of the program, I'm finding it hard to organize things such that all the files I open are wrapped in try-with-resources blocks. Furthermore, the top-level classes (where my main() function lies) don't know about this file.
Here's my code; note the lack of writer.close():
public class WorkRecorder {
public WorkRecorder(String recorderFile) throws FileNotFoundException {
writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(recorderFile)));
}
private Writer writer;
public void record(Data data) throws Exception {
// format Data object to match expected file format
// ...
writer.write(event.toString());
writer.write(System.lineSeparator());
writer.flush();
}
}
tl;dr do I need to implement AutoCloseable and call writer.close() if the resource is an opened output file, and I never need to close it until the program is done? Can I assume the JVM and the OS (Linux) will clean things up for me automatically?
Bonus (?): I struggled with this in C#'s IDisposeable too. The using block, like Java's try-with-resources construct, is a nice feature when I have something that I'm going to open, do something with quickly, and close right away. But often that's not the case, particularly with files, when the access to that resource hangs around for a while, or when needing to manage multiple such resources. If the answer to my question is "always use try-with-resources blocks" I'm stuck again.

I have similar code that doesn't lend itself to being wrapped in a try-with-resources statement. I think that is fine, as long as you close it when the program is done.
Just make sure you account for any Exceptions that may happen. For example, in my program, there is a cleanup() method that gets called when the program is shut down. This calls writer.close(). This is also called if there is any abnormal behavior that would cause the program to shut down.
If this is just a simple program, and you're expecting the Writer to be open for its duration, I don't think it's really a big deal for it to not be closed when the program terminates...but it is good practice to make sure your resources are closed, so I would go ahead and add that to wherever your program may shut down.

You should always close resources or set them to null so it can be picked up by the garbage collector in Java. Using try-with-resource blocks is a great way to have Java automatically close resources when you're done with them. Even if you use it for the duration of the program, it is good programming practice to close it even at the end. Some might say you don't need to, I personally would say just go ahead and do it and here's why:
"When a stream is no longer needed, always close it using the close() method or automatically close it using a try-with-resource statement. Not closing streams may cause data corruption in the output file, or other programming errors."
-Introduction to Java Programming 10th Edition, Y. Daniel Liang
If possible, just run the .close() method on the resource at the very end of the program.

I (now) think a better answer is "It depends" :-). A detailed treatment is provided by Lukas Eder here. Also check out the Lambda EG group post.
But in general, it's a good idea to return the resource back to the operating system when you are done with it and use try-with-resources all the time (except when you know what you are doing).

Clip sometimes does not play

I have a Sound class that contains a method that, when called, plays a sound using a Clip object (in this case, clip).
public static void play() {
clip.stop(); // The purpose of the first three lines
clip.flush(); // is to restart the Clip object so it
clip.setFramePosition(0); // can be played multiple times.
clip.start();
}
The instantiation of the Clip object occurs in a separate static method which is called prior to this method, which is why the above method can be declared static.
Another class that implements KeyListener contains the following code:
public void keyPressed(KeyEvent e) {
Sound.play(); // Sound is the class that implements the previous method.
}
Therefore, my code should be playing the sound associated with clip everytime a key is pressed. However, if I press a key quickly and repeatedly, the sound will sometimes not play. This is especially noticeable after a while (It seems as though the problem gets worse after each key press).
Why does this happen, and how can I circumvent this problem?

I have had the same kind of problems in the past, and something which worked for me is adding a line listener for whenever a line has finished, and closing it.
The code underneath is a stripped down version of what I use:
music = AudioSystem.getClip();
AudioInputStream ais = AudioSystem.getAudioInputStream(Sound.class.getResource("/sounds" + filename));
music.open(ais);
music.addLineListener(new LineListener(){
public void update(LineEvent e){
if(e.getType() == LineEvent.Type.STOP){
e.getLine().close();
}
}
});
music.start();
When you create the clip, just add in the line listener. When you reset the clip using your play function, it should play properly. I hope this works for you!

I came across this post via a reference in a similar question.
Yes, the number of Clips playing at once can contribute to the lag, but I don't know how to predict how much of an effect it will be, as this depends on how the JVM interacts with the OS.
You might be able to aid the situation by making the buffer size of the Clip smaller. SourceDataLine and Clip both seem to only allow interactions with incoming requests at buffer boundaries (I'm not sure this is a 100% accurate statement). Specifying the Clip's buffer size is a bit circuitous, as it requires obtaining the data as a PCM array. But if you wish to give it a try, the API is here.
A good way to reduce the number of lines is to use a sound library like TinySound. Many of the programmers at Java-gaming.org have used this library with success.
Another alternative that I'm promoting is AudioCue, a class I recently wrote for concurrent Clip playback. The license is BCD, source is provided, so feel free to examine, tinker with, and make use of the code. AudioCue doesn't reduce the number of output lines (yet). But if you are doing the common thing of managing multiple copies of the same cue, it could help in this regard as all concurrent instances are mixed down to a single output. Basic principle: the file is loaded into an array and played back via cursors that iterate over the array and merge their output into a SourceDataLine. This setup also allows implementation of smooth, real-time volume, panning and pitch fading.

Java problems playing sound clips

I have a problem playing a sound correctly in Java using the Clip interface.
Playing the sound works using:
clip = (Clip)mixer.getLine(dataLineInfo);
clip.open(audioFormat, byteData, 0, byteData.length);
clip.start();
However there is a memory leak if clips aren't closed.
I have tried adding a line listener before starting the clip, and use the following code:
public void update(LineEvent e) {
if (e.getType() == LineEvent.Type.STOP) {
e.getLine().close();
However, this causes the sound to degrage.
Adding a 1 second sleep in the method makes things work again on my machine - but I would prefer a more elegant solution - I don't think waiting in a listener method is good practice and other machines may take longer.
It is bizarre that the stop event is sent some time arbitrary time prior to the sound stopping.
Does anyone have any ideas on a better way to solve this?
(Related to this, this and this but none are solutions for me)

This appears to be the symptoms described in bug 4434125.
The suggested solution in that bug comments is to call clip.drain() (in an appropriate thread, because it is a blocking call), and then closing the clip when drain completes (since it will block until the data line's internal buffer is emptied (as per Javadocs for the drain method).

Java keeps lock on files for no apparent reason

Despite closing streams in finally clauses I seem to constantly run into cleaning up problems when using Java. File.delete() fails to delete files, Windows Explorer fails too. Running System.gc() helps sometimes but nothing short of terminating the VM helps consistently and that is not an option.
Does anyone have any other ideas I could try? I use Java 1.6 on Windows XP.
UPDATE: FLAC code sample removed, the code worked if I isolated it.
UPDATE:
More info, this happens in Apache Tomcat, Commons FileUpload is used to upload the file and could be the culprit, also I use Runtime.exec() to execute LAME in a separate process to encode the file, but that seems unlikely to cause this since ProcessExplorer clearly indicates that java.exe has a RW lock on the file and LAME terminates fine.
UPDATE: I am working with the assumption that there is a missing close() or a close() that does not get called somewhere in my code or external library. I just can't find it!

The code you posted looks good - it should not cause the issues you are describing. I understand you posted just a piece of the code you have - can you try extracting just this part to a separate program, run it and see if the issue still happens?
My guess is that there is some other place in the code that does new FileInputStream(path); and does not close the stream properly. You might be just seeing the results here when you try to delete the file.

I assume you're using jFlac. I downloaded jFlac 1.3 and tried your sample code on a flac freshly downloaded from the internet live music archive. For me, it worked. I even monitored it with ProcessExplorer and saw the file handles be opened and then released. Is your test code truly as simple as what you gave us, or is that a simplified version of your code? For me, once close() was called, the handle was released and the file was subsequently successfully deleted.
Try changing your infinite loop to:
File toDelete = new File(path);
if (!toDelete.delete()) {
System.out.println("Could not delete " + path);
System.out.println("Does it exist? " + toDelete.exists());
}
or if you want to keep looping, then put a 1 second sleep between attempts to delete the file. I tried this with JDK6 on WinXP Pro.
Don't forget to put a try/catch around your close() and log errors if the close throws an exception.

Make sure you have your close calls in the finally block not in the try block. If there is no try/finally because the method throws the exception then add a try/finally and put the close in there.
Look at the Windows Task Manager. For the Processes add the "Handles" column (under the View menu). Watch to see if the handles keep going up without ever dropping.
Use a profiler to see if you have Stream/Reader/Writer objects around that you do not think you should have.
EDIT:
Thanks for posting the code... off to see it. One thing - your close methods are not both guaranteed to execute - the first close might throw and then the second won't run.
EDIT 2:
final WavWriter wavWriter = new WavWriter(os);
LACDecoder decoder = new FLACDecoder(is);
The above two lines will cause the strams to be kept in instance variables presumably. As a test see if you can set the stream references to null after the decoder.decode() call (make a decoder.cleanup() method perhaps). See if holding onto the closed streams is causing a problem.
Also, do you do any wrapping of the streams passed into the above constructors? If so you might have to close the streams via the wrappers.

Your code sample should definitely work. In fact I ran your it on Java 1.6/Vista with jflac 1.3 and the source file is deleted, without any looping.
I'm guessing in your case another process is keeping the file open, perhaps a desktop search indexer or an antivirus. You can procexp to find which process is actually holding onto the file.

Isn't that an empty while loop?
you have:
try
{
...code
}
finally
{
}
while (something);
put some whitespace in there, and you actually have:
try
{
...code
}
finally
{
}
while (something)
;
your while loop isn't related to your try/finally. if your original try statement fails and the file isn't created, that while loop will never complete, because the try/finally will never execute a second time.
did you intend to make that a do{ all your code } while (your while statement)?
because that isn't what you have there.
EDIT to clarify:
my suggestion would be to change your while loop to have more info of why it can't delete:
while (!file.delete())
{
if (!file.exists())
break; // the file doesn't even exist, of course delete will fail
if (!file.canRead())
break; // the file isn't readable, delete will fail
if (!file.canWrite())
break; // the file isn't writable, delete will fail
}
because if delete fails once, its just going to fail over and over and over, of course its going to hang there. you aren't changing the state of the file in the loop.
Now that you've added other info, like Tomcat, etc, is this a permissions issue? are you trying to write to a file that the user tomcat is running as (nobody?) vm can't create? or delete a file that the tomcat process can't delete?
If process explorer/etc say java has a lock on the file, then something still has an open stream using it. someone might have not properly called close() on whatever streams are writing to the file?

If you are out of clues and ideas: In cygwin, cd to your javaroot and run something like:
find . -name '*.java' -print0 | xargs -0 grep "new.*new.*putStream"
It might provide a few suspects...

Another thing to try since you're using Tomcat-- in your Context Descriptor (typically Tomcat/conf/Catalina/localhost/your-context.xml), you can set
antiResourceLocking=true, which is designed to "avoid resource locking on Windows". The default for this (if you don't specify) is false. Worth a try.