I have an ImageWrapper class that saves images to temporary files in disk in order to free heap memory, and allows reloading them when needed.
class ImageWrapper {
File tempFile;
public ImageWrapper(BufferedImage img) {
// save image to tempFile and gc()
}
public BufferedImage getImage() {
// read the image from tempFile and return it.
}
public void delete() {
// delete image from disk.
}
}
My concern is, how to make sure that files gets deleted when such ImageWrapper's instance is garbage collected (otherwise I risk filling the disk with unneeded images). This must be done while the application is still running (as opposed to during-termination cleanup suggestions) due to the fact that it is likely to run for long periods.
I'm not fully familiar with java's GC concept, and I was wondering if finalize() is what I'm looking for. My idea was to call delete() (on a separate Thread, for that matters) from an overriden finalize() method. Is that the right way to do it?
UPDATE:
I don't think I can close() the object as suggested by many users, due to the fact that each such image is fetched to a list of listeners which I don't control, and might save a reference to the object. the only time when I'm certain to be able to delete the file is when no references are held, hence I thought finalize() is the right way. Any suggestions?
UPDATE 2:
What are the scenarios where finalize() will not be called? If the only possibles are exiting the program (in an expected/unexpected way), I can take it, because it means I risk only one unneeded temp file left un deleted (the one that was processed during exiting).
Another approach is to use File.deleteOnExit() which marks a file for the JVM to delete upon exit. I realise it's not quite what you're looking for, but may be of interest.
To be clear, if your JVM dies unexpectedly, it won't clear those files. As such, you may want to architect your solution to clear up cache files on startup, such that you don't build up a mass of unused cache files over time.
An good alternative to finalize is the PhantomReference. the best way to use it is:
public class FileReference extends PhantomReference<CachedImage> {
private final File _file;
public FileReference(CachedImage img, ReferenceQueue<CachedImage> q, File f) {
super(img, q);
_file = f;
}
public File getFile() {
_file;
}
}
Then use it like:
public class CachedImage {
private static final ReferenceQueue<CachedImage>
refQue = new ReferenceQueue<CachedImage>();
static {
Thread t = new Thread() {
#Override
public void run() {
try {
while (true) {
FileReference ref = (FileReference)refQue.remove();
File f = ref.getFile();
f.delete();
}
} catch (Throwable t) {
_log.error(t);
}
}
};
t.setDaemon(true);
t.start();
}
private final FileReference _ref;
public CachedImage(BufferedImage bi, File tempFile) {
tempFile.deleteOnExit();
saveAndFree(bi, tempFile);
_ref = new FileReference<CachedImage>(this, refQue, tempFile);
}
...
}
It is not recommended to use finalize().The problem is that you can't count on the garbage collector to ever delete an object. So, any code that you put into your class's overridden finalize() method is not guaranteed to run.
There's no guarantee that your finalize method will ever get called; in particular, any objects hanging around when the program exits are usually just thrown away with no cleanup. Closeable is a much better option.
As an alternative to #Brian Agnew's answer, why not install a ShutdownHook that clears out your cache directory?
public class CleanCacheOnShutdown extends Thread {
#Override
public void run() { ... }
}
System.getRuntime().addShutdownHook(new CleanCacheOnShutdown());
I ended up using a combination of File.deleteOnExit() (thanks #Brian), and a ScheduledExecutorService that goes over a ReferenceQueue of PhantomReferences to my class instances, according to this post.
I add this answer because no one suggested using ReferenceQueue (which I think is the ideal solution for my problem), and I think it will be helpful for future readers.
The (somewhat simplified) outcome is this (changed the class name to CachedImage):
public class CachedImage {
private static Map<PhantomReference<CachedImage>, File>
refMap = new HashMap<PhantomReference<CachedImage >, File>();
private static ReferenceQueue<CachedImage>
refQue = new ReferenceQueue<CachedImage>();
static {
Executors.newScheduledThreadPool(1).scheduleWithFixedDelay(new Thread() {
#Override
public void run() {
try {
Reference<? extends CachedImage> phanRef =
refQue.poll();
while (phanRef != null) {
File f = refMap.get(phanRef);
f.delete();
phanRef = refQue.poll();
}
} catch (Throwable t) {
_log.error(t);
}
}
}, 1, 1, TimeUnit.MINUTES);
}
public CachedImage(BufferedImage bi, File tempFile) {
tempFile.deleteOnExit();
saveAndFree(bi, tempFile);
PhantomReference<CachedImage> pref =
new PhantomReference<CachedImage>(this, refQue);
refMap.put(pref, tempFile);
}
...
}
Related
With my server application I'm using a list of WeakReferences to keep count and handle active sessions to server. I'm running periodic gc to clean the list of inactive sessions, but for some reason one reference always remains. According to overridden finalize method this is the last sessions created.
I'm clueless on why this is happening. I first thought this may have been due to static methods or variables, but for now i have removed such objects from ClientHandlerThread class. There are no other references from the server class but the weak references list. Currently this is not a big issue for me, but to have better understanding on how java selects objects to be garbage collected can be of use in the future. :) Below are most important code snippets:
Server.java:
public class Server {
private List<WeakReference<ClientHandlerThread>> m_connectedClients =
Collections.synchronizedList(
new ArrayList<WeakReference<ClientHandlerThread>>());
/** Counter to identify sessions */
private static AtomicInteger m_NumSession = new AtomicInteger(0);
Server() {
SSLServerSocket sslDataTraffic = null;
// Sockets are initialized here - code removed for clarity
// Run periodic GC
Thread stThread = new Thread() {
public void run() {
do {
try {
Thread.sleep(5000);
}
catch (InterruptedException ignore) {}
System.runFinalization();
System.gc();
cleanUpSessionsList();
} while (true);
}
};
stThread.setPriority(Thread.MIN_PRIORITY);
stThread.start();
// Listen to new connections, create handlers and add to list
while (true) {
try {
SSLSocket sslDataTrafficSocketInstance =
(SSLSocket) sslDataTraffic.accept();
ClientHandlerThread c = new ClientHandlerThread(
sslDataTrafficSocketInstance,
m_NumSession.incrementAndGet());
c.start();
m_connectedClients.add(new WeakReference<>(c));
} catch (Exception e) {
e.printStackTrace();
}
}
}
/** Clean any old references and return the number of active connections
* #return
*/
public int cleanUpSessionList() {
int i = 0;
synchronized(m_connectedClients) {
Iterator<WeakReference<ClientHandlerThread>> it =
m_connectedClients.iterator();
while (it.hasNext()) {
WeakReference<ClientHandlerThread> sessionRef = it.next();
if (sessionRef.get() == null)
it.remove();
else
i++;
}
}
System.out.println("Active sessions: " + i");
return i;
}
}
ClientHandlerThread.java:
public class ClientHandlerThread extends Thread {
private int m_SessionID;
private SSLSocket dataSocket;
public ClientHandlerThread(
SSLSocket dataSocket,
int sessionID) {
this.dataSocket = dataSocket;
m_SessionID = sessionID;
}
public void run() {
// code removed
}
#Override
protected void finalize() throws Throwable {
System.out.println("Session " + m_SessionID + " finalized");
super.finalize();
}
}
That's about all wrong (the code itself isn't bad, but you're doing many things I'd usually avoid).
Use a ReferenceQueue instead of finalize.
Consider using a PhantomReference instead of weak as you AFAICT don't need to access the referee.
If all you want is to count active sessions, the simply count them (surround the handler code by session tracking code).
You should use a thread pool.
Running periodic GC can impact performance (though it may even help the performance, you should not rely on it).
Concerning the question itself... no idea, but there may be something in the code blocking the last thread from freeing. as already suggested, perform a heap snapshot, run it through a memory analyzer.
Found this question as a cross-reference after I posted a related question.
I don't have an answer as to why it's happening, and I think it shouldn't, but I can tell you what I think is happening, and I'm curious if a suggested workaround changes the behavior you're seeing. (If you still have the code lying around; I know it's an older question.)
As far as I can tell, somehow the JRE maintains a reference to the last scoped variable that gets created. By setting the variable to null (or by creating another, new, unrelated scoped variable) this problem goes away.
So something like this:
ClientHandlerThread c = new ClientHandlerThread(
sslDataTrafficSocketInstance,
m_NumSession.incrementAndGet());
c.start();
m_connectedClients.add(new WeakReference<>(c));
c = null;
Again, I'm not saying it should behave this way, but from the testing I've done in my similar situation, it works.
The plan:
I created a little PreferenceActivity (don't hate me, I'm supporting API 10 and up) and need to display the current usage of local storage data by my app. I did this using a specialized class (a pretty big one, as of the moment) that handles all file operations (it's called FileOperations.java for a reason). Inside this class file there is a method getSize(File file) {...} which does just that. It gets the size of a file (or folder) with this little piece of code:
public long getSize(File file) {
long size = 0;
if(file.isDirectory()) {
for(File child : file.listFiles()) {
size += getSize(child);
}
}
size = file.length();
return size;
}
The general idea was to use this in a background Thread so it doesn't clog the UI even the slightest bit. (I am really annoyed by lagging apps and suffer from them daily)
The problem:
This works just fine. However, as soon as I purge the folder the app stores it's data in using this beauty:
private void purgeLocalStorage() {
new Thread(new Runnable() {
#Override
public void run() {
Looper.prepare();
Log.i("ActivityPrefsLocalStorage.purgeLocalStorage.Thread.Runnable.run", "Started to run");
final String directory = context.getResources().getString(R.string.app_name);
final String usedData = context.getResources().getString(R.string.ActivityPrefsLocalStorage_usedData);
final File file = new File(Environment.getExternalStorageDirectory()+"/"+directory);
final FileOperations FO = new FileOperations(context);
Log.i("ActivityPrefsLocalStorage.purgeLocalStorage.Thread.Runnable.run", "deleting folder: "+file);
if(FO.delete(file)) {
Log.i("ActivityPrefsLocalStorage.purgeLocalStorage.Thread.Runnable.run", file+" deleted");
runOnUiThread(new Runnable() {
#Override
public void run() {
Toast.makeText(context, R.string.ActivityPrefsLocalStorage_deleteSucces, Toast.LENGTH_SHORT).show();
setTotalLocalDataTexts(usedData+" "+context.getResources().getString(R.string.pref_totalData_default), "");
getUsedStorage();
}
});
} else {
Log.e("ActivityPrefsLocalStorage.purgeLocalStorage.Thread.Runnable.run", "could not delete "+file);
runOnUiThread(new Runnable() {
#Override
public void run() {
Toast.makeText(context, R.string.ActivityPrefsLocalStorage_deleteError, Toast.LENGTH_SHORT).show();
}
});
}
}
}).start();
}
Things hit the fan...
See, the problem is that my method for reading the size of the folder does not want to function properly when called by the previous method.
Here's a snippet:
private void getUsedStorage() {
new Thread(new Runnable() {
#Override
public void run() {
Log.i("ActivityPrefsLocalStorage.getUsedStorage.Thread.Runnable.run", "Started to run");
final String directory = context.getResources().getString(R.string.app_name);
final String usedData = context.getResources().getString(R.string.ActivityPrefsLocalStorage_usedData);
final File file = new File(Environment.getExternalStorageDirectory()+"/"+directory);
final FileOperations FO = new FileOperations(context);
final DataUsage DU = new DataUsage(context);
Log.i("ActivityPrefsLocalStorage.getUsedStorage.Thread.Runnable.run", "Checking filesize of folder: "+file);
long fileSize = FO.getSize(file);
String usedUnits = DU.getUnit(fileSize, true, false, false);
String usedBytes = DU.getUnit(fileSize, true, true, true);
Log.i("ActivityPrefsLocalStorage.getUsedStorage.Thread.Runnable.run", "filesize of "+file+": "+usedUnits);
setTotalLocalDataTexts(usedData+" "+usedUnits, usedBytes);
}
}).start();
}
However, a quick and easy workaround would be to place it on the UI thread like so:
...blabla code you already read above.
long fileSize = FO.getSize(file);
String usedUnits = DU.getUnit(fileSize, true, false, false);
String usedBytes = DU.getUnit(fileSize, true, true, true);
Log.i("ActivityPrefsLocalStorage.getUsedStorage.Thread.Runnable.run", "filesize of "+file+": "+usedUnits);
runOnUiThread(new Runnable() {
#Override
public void run() {
setTotalLocalDataTexts(usedData+" "+usedUnits, usedBytes);
}
});
}
}).start();
}
And that's where it starts getting interesting. I cannot use non-finals inside the new Runnable(), and I cannot make them final since I want the value to update and not remain stuck at eg. 32MiB (while it has just been purged).
Possible fixes:
I should man up and just use a final. The user will understand they need to refresh the page manually. (oh no...)
Hire... erm. Extend an AsyncTask<Void, Void, Void> to do the work.
My ideal fix:
Someone giving me an awesome snippet of code for free that does all the magic. no, seriously though, I would really appreciate anything apart from my list of possible fixes. There has to be some way to pass the new Runnable() a variable without creating classes and implementing the entire universe? Or is that what I am trying to achieve really a new thing?
TL;DR:
Things go wrong as soon as I call getUsedStorage() from within a new Runnable(). This function is also a background task inside a Runnable, but updates the UI using a private void function that sets it. It only passes variables to this function. and then things fly off the handle(r).
Edit: grammar.
Edit2: Also a pretty interesting thing to note here, I used something similar in another PreferenceActivity, and that one works. (but that one does not update at the press of a button that calls another private something functionName() {new Thread(new Runnable() {public void run() {...});})
There are a couple of ways to use non-finals inside of a Runnable or other enclosed classes.
The first is to change your variables to be members of an enclosing class. This will allow you to use the variables inside the Runnable. An example follows:
public class Foo {
private long time;
public void run() {
time = System.currentTimeMillis();
new Thread(new Runnable() {
#Override
public void run() {
time += 1;
System.out.println("Our current time is: " + time);
}
});
}
}
The other option, and it is quite hacky, is to use a final array with a length of 1. An example of that follows:
public class Foo {
public void run() {
final long[] time = { System.currentTimeMillis() };
new Thread(new Runnable() {
#Override
public void run() {
time[0] += 1;
System.out.println("Our current time is: " + time[0]);
}
});
}
}
As gonemad16 on reddit.com/r/androiddev pointed out, my issue had nothing to do with final vs non final. That was not the reason I was getting an old value. All my variables are given a value, sent to setTotalLocalDataTexts and then go out of scope... nothing is updating their values.. so there is no harm in them being final and no benefit to them being non final...
It was an an issue in getSize()
I thought I had a correct loop there using if file.isDirectory() {...}. It created a directory tree and executed itself using the children it has found. When all items have been scanned the value returns to the function calling it.
This was working just fine for me while I was still running all of my code on the ui thread. Everything was slow. But it worked.
However, I forgot that I removed a very crucial ...} else {...
I believe I removed that one because it caused a stack overflow at some point, so I removed it and I guess forgot to put it back...
And here I was thinking my first SO question wouldn't be a noobish question...
A program that I've developed is crashing the JVM occasionally due to this bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8029516. Unfortunately the bug has not been resolved by Oracle and the bug report says that there are no known workarounds.
I've tried to modify the example code from the bug report by calling .register(sWatchService, eventKinds) in the KeyWatcher thread instead, by adding all pending register request to a list that I loop through in the KeyWatcher thread but it's still crashing. I'm guessing this just had the same effect as synchronizing on sWatchService (like the submitter of the bug report tried).
Can you think of any way to get around this?
From comments:
It appears that we have an issue with I/O cancellation when there is a pending ReadDirectoryChangesW outstanding.
The statement and example code indicate that the bug is triggered when:
There is a pending event that has not been consumed (it may or may not be visible to WatchService.poll() or WatchService.take())
WatchKey.cancel() is called on the key
This is a nasty bug with no universal workaround. The approach depends on the specifics of your application. Consider pooling watches to a single place so you don't need to call WatchKey.cancel(). If at one point the pool becomes too large, close the entire WatchService and start over. Something similar to.
public class FileWatcerService {
static Kind<?>[] allEvents = new Kind<?>[] {
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_DELETE,
StandardWatchEventKinds.ENTRY_MODIFY
};
WatchService ws;
// Keep track of paths and registered listeners
Map<String, List<FileChangeListener>> listeners = new ConcurrentHashMap<String, List<FileChangeListener>>();
Map<WatchKey, String> keys = new ConcurrentHashMap<WatchKey, String>();
boolean toStop = false;
public interface FileChangeListener {
void onChange();
}
public void addFileChangeListener(String path, FileChangeListener l) {
if(!listeners.containsKey(path)) {
listeners.put(path, new ArrayList<FileChangeListener>());
keys.put(Paths.get(path).register(ws, allEvents), path);
}
listeners.get(path).add(l);
}
public void removeFileChangeListener(String path, FileChangeListener l) {
if(listeners.containsKey(path))
listeners.get(path).remove(l);
}
public void start() {
ws = FileSystems.getDefault().newWatchService();
new Thread(new Runnable() {
public void run() {
while(!toStop) {
WatchKey key = ws.take();
for(FileChangeListener l: listeners.get(keys.get(key)))
l.onChange();
}
}
}).start();
}
public void stop() {
toStop = true;
ws.close();
}
}
I've managed to create a workaround though it's somewhat ugly.
The bug is in JDK method WindowsWatchKey.invalidate() that releases native buffer while the subsequent calls may still access it. This one-liner fixes the problem by delaying buffer clean-up until GC.
Here is a compiled patch to JDK. In order to apply it add the following Java command-line flag:
-Xbootclasspath/p:jdk-8029516-patch.jar
If patching JDK is not an option in your case, there is still a workaround on the application level. It relies on the knowledge of Windows WatchService internal implementation.
public class JDK_8029516 {
private static final Field bufferField = getField("sun.nio.fs.WindowsWatchService$WindowsWatchKey", "buffer");
private static final Field cleanerField = getField("sun.nio.fs.NativeBuffer", "cleaner");
private static final Cleaner dummyCleaner = Cleaner.create(Thread.class, new Thread());
private static Field getField(String className, String fieldName) {
try {
Field f = Class.forName(className).getDeclaredField(fieldName);
f.setAccessible(true);
return f;
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
public static void patch(WatchKey key) {
try {
cleanerField.set(bufferField.get(key), dummyCleaner);
} catch (IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
}
Call JDK_8029516.patch(watchKey) right after the key is registred, and it will prevent watchKey.cancel() from releasing the native buffer prematurely.
You might not be able to work around the problem itself but you could deal with the error and handle it. I don't know your specific situation but I could imagine the biggest issue is the crash of the whole JVM. Putting all in a try block does not work because you cannot catch a JVM crash.
Not knowing more about your project makes it difficult to suggest a good/acceptable solution, but maybe this could be an option: Do all the file watching stuff in a separate JVM process. From your main process start a new JVM (e.g. using ProcessBuilder.start()). When the process terminates (i.e. the newly started JVM crashes), restart it. Obviously you need to be able to recover, i.e. you need to keep track of what files to watch and you need to keep this data in your main process too.
Now the biggest remaining part is to implement some communication between the main process and the file watching process. This could be done using standard input/output of the file watching process or using a Socket/ServerSocket or some other mechanism.
It seems like I have a race condition when I call file.getAbsolutePath() in my Java program.
In one thread I am processing a file and when it is finished processing I am changing the filename and moving it to another directory on the UNIX file system.
In a separate thread running in parallel I am attempting to open this file that is being processed and reading its contents. In 99% of use cases this operation is fine however I have noticed sometimes that the operation fails with a FileNotFound exception.
When I catch this exception I am logging the file.getAbsolutePath() value and I see the value is the concatenation of the path of the file in the processed directory it has been moved to and also the path of the file in the directory it was present in before processing completed.
Has anyone experienced a similar problem in the past and how did you get around it?
Thanks
It seems you need to synchronize the file access from separate threads using a class that does this, let's call it FileManager.
First option in implementing the FileManager is to use an exclusive lock. For example:
class FileManager {
private Object lock = new Object();
public void processFile() {
synchronized(lock) {
...
}
}
public void readFile() {
synchronized(lock) {
...
}
}
}
If there are many more readers than writers a Read/Write Lock is more suitable as it allows multiple concurrent readers but only a single writer:
class FileManager {
private final Lock readLock;
private final Lock writeLock;
FileManager() {
ReentrantReadWriteLock readWriteLock = new ReentrantReadWriteLock(false);
readLock = readWriteLock.readLock();
writeLock = readWriteLock.writeLock();
}
public void processFile() {
writeLock.lock();
try {
...
}
finally {
writeLock.unlock();
}
}
public void readFile() {
readLock.lock();
try {
...
}
finally {
readLock.unlock();
}
}
}
this is some kind of long post, so I have to say thanks for reading.
My app is supposed to process a lot of soundfiles, lets say 4000+. My first approach was to load a certain amount (lets say 200mb) of sound data, process it, write it and then "null" the data to let the gc free it. But regarting to the fact that the data is loaded via intranet, this seems not to be the "best" way. (File access is slow) Calculations should start with the first loaded file. To achive this, I changed the concept to a sort of "producer/consumer" (I think). Here are my classes so far:
Reader/Producer
public class ReaderThread extends Thread {
List<Long> files;
ConcurrentLinkedQueue<Long> loaded = new ConcurrentLinkedQueue<Long>();
boolean finished = false;
public ReaderThread( List<Long> soundFiles) {
this.files = soundFiles;
}
#Override
public void run() {
Iterator<Long> it = files.iterator();
while(it.hasNext()) {
Long id = it.next();
if (FileLoader.load(id)) {
loaded.add(id);
}
}
finished = true;
}
public Long getNextId() {
while(loaded.isEmpty()) {
if( finished ) {
return null;
}
}
Long id = loaded.poll();
return id;
}
}
This is the writer/(Not consumer)
public class WriterThread extends Thread {
ConcurrentLinkedQueue<Long> loaded = new ConcurrentLinkedQueue<Long>();
String directory;
boolean abort = false;
public WriterThread(String directory) {
this.directory = directory;
}
#Override
public void run() {
while(!(abort&&loaded.isEmpty())) {
if(!loaded.isEmpty()) {
Long id = loaded.poll();
FileWriter.write(id, directory);
FileManager.unload(id);
}
}
}
public synchronized void submit(Long id) {
loaded.add(id);
}
public synchronized void halt() {
abort = true;
}
}
This is the part where all things get together:
// Forgive me the "t" and "w". ;-)
t = new ReaderThread(soundSystem,soundfilesToProcess);
w = new WriterThread(soundSystem,outputDirectory );
t.start();
w.start();
long start = System.currentTimeMillis();
while(!abort) {
Long id = t.getNextId();
if(id!=null) {
SoundFile soundFile = soundSystem.getSoundfile(id);
ProcessorChain pc = new ProcessorChain(soundFile, getProcessingChain(), w);
Future<List<ProcessorResult>> result = es.submit(pc);
results.add(result);
}else {
break;
}
}
for( Future<List<ProcessorResult>> result : results) {
List<ProcessorResult> tempResults;
try {
tempResults = result.get();
processResults(tempResults);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
w.halt();
"ProcessorChain" is a runnable.
es.submit -> "es" is a CachedThreadPool.
What I need to know in the first place is weather or not this approach is "good", or if it is more like "nonsens". It seems to work quite well, but I have little problems with the writer thread, it seems that in some cases not all files get written. The writer threads submit method is called by the ProcessorChain when it has finished work.
The second thing it thread safety. Did I miss somethin?
I believe it will be (a lot) simpler if each thread reads, process and then writes a whole soundfile (one thread per file).
You use can Java thread pools and let the operating system/Java VM parallelize the read/process/write with multiple files to gain eficiency. I may be wrong, but from what you described a simpler solution would be enough and then you can measure your bottleneck if further improvements are needed.
I think the approach is ok in general (one thread for reading input, one for writing output and one or more for processing).
A couple of suggestions:
1 - you probably want to use semaphores instead of having your threads spinning in a constant loop. For example, with a semaphore your write thread would just block until a file was actually available to write. Currently it will spin, potentially wasting 1/3 of your cpu cycles when there's nothing to write.
2 - you probably want to explicitly create worker threads instead of doing the work on the main thread. That way you can have multiple threads doing processing at the same time.
That may already be what ProcessorChain is doing, but it's not clear from the snippet.