A program that I've developed is crashing the JVM occasionally due to this bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8029516. Unfortunately the bug has not been resolved by Oracle and the bug report says that there are no known workarounds.
I've tried to modify the example code from the bug report by calling .register(sWatchService, eventKinds) in the KeyWatcher thread instead, by adding all pending register request to a list that I loop through in the KeyWatcher thread but it's still crashing. I'm guessing this just had the same effect as synchronizing on sWatchService (like the submitter of the bug report tried).
Can you think of any way to get around this?
From comments:
It appears that we have an issue with I/O cancellation when there is a pending ReadDirectoryChangesW outstanding.
The statement and example code indicate that the bug is triggered when:
There is a pending event that has not been consumed (it may or may not be visible to WatchService.poll() or WatchService.take())
WatchKey.cancel() is called on the key
This is a nasty bug with no universal workaround. The approach depends on the specifics of your application. Consider pooling watches to a single place so you don't need to call WatchKey.cancel(). If at one point the pool becomes too large, close the entire WatchService and start over. Something similar to.
public class FileWatcerService {
static Kind<?>[] allEvents = new Kind<?>[] {
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_DELETE,
StandardWatchEventKinds.ENTRY_MODIFY
};
WatchService ws;
// Keep track of paths and registered listeners
Map<String, List<FileChangeListener>> listeners = new ConcurrentHashMap<String, List<FileChangeListener>>();
Map<WatchKey, String> keys = new ConcurrentHashMap<WatchKey, String>();
boolean toStop = false;
public interface FileChangeListener {
void onChange();
}
public void addFileChangeListener(String path, FileChangeListener l) {
if(!listeners.containsKey(path)) {
listeners.put(path, new ArrayList<FileChangeListener>());
keys.put(Paths.get(path).register(ws, allEvents), path);
}
listeners.get(path).add(l);
}
public void removeFileChangeListener(String path, FileChangeListener l) {
if(listeners.containsKey(path))
listeners.get(path).remove(l);
}
public void start() {
ws = FileSystems.getDefault().newWatchService();
new Thread(new Runnable() {
public void run() {
while(!toStop) {
WatchKey key = ws.take();
for(FileChangeListener l: listeners.get(keys.get(key)))
l.onChange();
}
}
}).start();
}
public void stop() {
toStop = true;
ws.close();
}
}
I've managed to create a workaround though it's somewhat ugly.
The bug is in JDK method WindowsWatchKey.invalidate() that releases native buffer while the subsequent calls may still access it. This one-liner fixes the problem by delaying buffer clean-up until GC.
Here is a compiled patch to JDK. In order to apply it add the following Java command-line flag:
-Xbootclasspath/p:jdk-8029516-patch.jar
If patching JDK is not an option in your case, there is still a workaround on the application level. It relies on the knowledge of Windows WatchService internal implementation.
public class JDK_8029516 {
private static final Field bufferField = getField("sun.nio.fs.WindowsWatchService$WindowsWatchKey", "buffer");
private static final Field cleanerField = getField("sun.nio.fs.NativeBuffer", "cleaner");
private static final Cleaner dummyCleaner = Cleaner.create(Thread.class, new Thread());
private static Field getField(String className, String fieldName) {
try {
Field f = Class.forName(className).getDeclaredField(fieldName);
f.setAccessible(true);
return f;
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
public static void patch(WatchKey key) {
try {
cleanerField.set(bufferField.get(key), dummyCleaner);
} catch (IllegalAccessException e) {
throw new IllegalStateException(e);
}
}
}
Call JDK_8029516.patch(watchKey) right after the key is registred, and it will prevent watchKey.cancel() from releasing the native buffer prematurely.
You might not be able to work around the problem itself but you could deal with the error and handle it. I don't know your specific situation but I could imagine the biggest issue is the crash of the whole JVM. Putting all in a try block does not work because you cannot catch a JVM crash.
Not knowing more about your project makes it difficult to suggest a good/acceptable solution, but maybe this could be an option: Do all the file watching stuff in a separate JVM process. From your main process start a new JVM (e.g. using ProcessBuilder.start()). When the process terminates (i.e. the newly started JVM crashes), restart it. Obviously you need to be able to recover, i.e. you need to keep track of what files to watch and you need to keep this data in your main process too.
Now the biggest remaining part is to implement some communication between the main process and the file watching process. This could be done using standard input/output of the file watching process or using a Socket/ServerSocket or some other mechanism.
Related
With my server application I'm using a list of WeakReferences to keep count and handle active sessions to server. I'm running periodic gc to clean the list of inactive sessions, but for some reason one reference always remains. According to overridden finalize method this is the last sessions created.
I'm clueless on why this is happening. I first thought this may have been due to static methods or variables, but for now i have removed such objects from ClientHandlerThread class. There are no other references from the server class but the weak references list. Currently this is not a big issue for me, but to have better understanding on how java selects objects to be garbage collected can be of use in the future. :) Below are most important code snippets:
Server.java:
public class Server {
private List<WeakReference<ClientHandlerThread>> m_connectedClients =
Collections.synchronizedList(
new ArrayList<WeakReference<ClientHandlerThread>>());
/** Counter to identify sessions */
private static AtomicInteger m_NumSession = new AtomicInteger(0);
Server() {
SSLServerSocket sslDataTraffic = null;
// Sockets are initialized here - code removed for clarity
// Run periodic GC
Thread stThread = new Thread() {
public void run() {
do {
try {
Thread.sleep(5000);
}
catch (InterruptedException ignore) {}
System.runFinalization();
System.gc();
cleanUpSessionsList();
} while (true);
}
};
stThread.setPriority(Thread.MIN_PRIORITY);
stThread.start();
// Listen to new connections, create handlers and add to list
while (true) {
try {
SSLSocket sslDataTrafficSocketInstance =
(SSLSocket) sslDataTraffic.accept();
ClientHandlerThread c = new ClientHandlerThread(
sslDataTrafficSocketInstance,
m_NumSession.incrementAndGet());
c.start();
m_connectedClients.add(new WeakReference<>(c));
} catch (Exception e) {
e.printStackTrace();
}
}
}
/** Clean any old references and return the number of active connections
* #return
*/
public int cleanUpSessionList() {
int i = 0;
synchronized(m_connectedClients) {
Iterator<WeakReference<ClientHandlerThread>> it =
m_connectedClients.iterator();
while (it.hasNext()) {
WeakReference<ClientHandlerThread> sessionRef = it.next();
if (sessionRef.get() == null)
it.remove();
else
i++;
}
}
System.out.println("Active sessions: " + i");
return i;
}
}
ClientHandlerThread.java:
public class ClientHandlerThread extends Thread {
private int m_SessionID;
private SSLSocket dataSocket;
public ClientHandlerThread(
SSLSocket dataSocket,
int sessionID) {
this.dataSocket = dataSocket;
m_SessionID = sessionID;
}
public void run() {
// code removed
}
#Override
protected void finalize() throws Throwable {
System.out.println("Session " + m_SessionID + " finalized");
super.finalize();
}
}
That's about all wrong (the code itself isn't bad, but you're doing many things I'd usually avoid).
Use a ReferenceQueue instead of finalize.
Consider using a PhantomReference instead of weak as you AFAICT don't need to access the referee.
If all you want is to count active sessions, the simply count them (surround the handler code by session tracking code).
You should use a thread pool.
Running periodic GC can impact performance (though it may even help the performance, you should not rely on it).
Concerning the question itself... no idea, but there may be something in the code blocking the last thread from freeing. as already suggested, perform a heap snapshot, run it through a memory analyzer.
Found this question as a cross-reference after I posted a related question.
I don't have an answer as to why it's happening, and I think it shouldn't, but I can tell you what I think is happening, and I'm curious if a suggested workaround changes the behavior you're seeing. (If you still have the code lying around; I know it's an older question.)
As far as I can tell, somehow the JRE maintains a reference to the last scoped variable that gets created. By setting the variable to null (or by creating another, new, unrelated scoped variable) this problem goes away.
So something like this:
ClientHandlerThread c = new ClientHandlerThread(
sslDataTrafficSocketInstance,
m_NumSession.incrementAndGet());
c.start();
m_connectedClients.add(new WeakReference<>(c));
c = null;
Again, I'm not saying it should behave this way, but from the testing I've done in my similar situation, it works.
I'm looking for a recommendation on how to make this code thread-safe with locks in Java. I know there are a lot of gotchas with locks; obscure problems, race-conditions, etc that can pop up. Here is the basic idea of what I'm trying to achieve, implemented rather naïvely:
public class MultipleThreadWriter {
boolean isUpgrading=false;
boolean isWriting=false;
public void writeData(String uniqueId) {
if (isUpgrading)
//block until isUpgrading is false
isWriting = true;
{
//do write stuff
}
isWriting = false;
}
public void upgradeSystem() {
if (isWriting)
//block until isWriting is false
isUpgrading = true;
{
//do updates
}
isUpgrading = false;
}
}
The basic idea is that multiple threads are allowed to write data simultaneously. It doesn't matter, since no two threads will ever be writing to data pertaining to the same uniqueId. However, the "system upgrade" manipulates data for all uniqueIds, so it must block (wait in line) until no data is being written before it can start, at which point it blocks all writes until it is finished. (It is definitely not a consumer/producer pattern going on here- upgrading occurs arbitrarily, i.e. has no relation to the data being written.)
This sounds like a good application for a readers-writer lock.
However, in this case your "readers" are the small update tasks that can all run concurrently, and your "writer" is the system upgrade task.
There's an implementation of this in the Java standard library:
java.util.concurrent.locks.ReentrantReadWriteLock
The lock has fair and non-fair modes. If you want the system upgrade to run as soon as possible after it's scheduled, then use the fair mode of the lock. If you want the upgrade to be applied during idle time (i.e., wait until there are no small updates going on), then you can use the non-fair mode instead.
Since this is a bit of an unorthodox application of the readers-writer lock (your readers are actually writing too!), make sure to comment this well in your code. You might even consider writing a wrapper around the ReentrantReadWriteLock class that provides localUpdateLock vs globalUpdateLock methods, which delegate to the readLock and writeLock, respectively.
Based on answer from #DaoWen , this is my untested solution.
public class MultipleThreadWriter {
private final ReentrantReadWriteLock rwl = new ReentrantReadWriteLock();
private final Lock r = rwl.readLock();
private final Lock w = rwl.writeLock();
public void writeData() {
r.lock();
try {
//do write stuff
} finally {
r.unlock();
}
}
public void upgradeSystem() {
w.lock();
try {
//do updates
} finally {
w.unlock();
}
}
}
I need to make a program, which can be executed in single instance. I tried to create a temporary file and delete it before exit program.
public static boolean isLocked() {
File f = new File("lock.txt");
return f.exists();
}
public static void lock() {
String fname = "lock.txt";
File f = new File(fname);
try {
f.createNewFile();
} catch (IOException e) {
e.printStackTrace();
}
}
public static void unlock() {
File f = new File("lock.txt");
f.delete();
}
In frame
private void initialize() {
lock();
}
private void setFrameHandler() {
frame.addWindowListener(new java.awt.event.WindowAdapter() {
#Override
public void windowClosing(java.awt.event.WindowEvent windowEvent) {
unlock();
}
});
}
Problem occurs if program is finished with emergency (e.g. electricity cuts). File does not remove, and running a new instance is impossible.
How to make a reliable single-instance verification?
You could check for another instance of the program at startup using the GetProcesses method as described here
But that only works depending on the scenario you have (might not see all processes of other users)
Another thing you could do is simply checking, if a specific file is locked via File.Open
File.Open ("path.lock", FileMode.OpenOrCreate, FileAccess.ReadWrite);
As long as you keep the resulting FileStream open in your program no other program can open the file in that mode either. This is basically how Unix lock files work too. Of course you have to catch an IOException (hinting you to a locked file).
Disclaimer: I did not try that code out so please check if I gave you the right parameters.
Edit: You could also check out this Code-Project article on how to do it with the win32 API
Another attempt using windows messaging has been done here
A simple approach to this on a single machine is to write a 'PID file', which is literally a file containing the operating system's ID of the process currently running. You create this when you start your "critical" work, and remove it on successful completion.
Since it is unlikely that the process would be started again with the same PID, you can simply check to see if the PID file already exists, and if so, if that process is still running.
I have an ImageWrapper class that saves images to temporary files in disk in order to free heap memory, and allows reloading them when needed.
class ImageWrapper {
File tempFile;
public ImageWrapper(BufferedImage img) {
// save image to tempFile and gc()
}
public BufferedImage getImage() {
// read the image from tempFile and return it.
}
public void delete() {
// delete image from disk.
}
}
My concern is, how to make sure that files gets deleted when such ImageWrapper's instance is garbage collected (otherwise I risk filling the disk with unneeded images). This must be done while the application is still running (as opposed to during-termination cleanup suggestions) due to the fact that it is likely to run for long periods.
I'm not fully familiar with java's GC concept, and I was wondering if finalize() is what I'm looking for. My idea was to call delete() (on a separate Thread, for that matters) from an overriden finalize() method. Is that the right way to do it?
UPDATE:
I don't think I can close() the object as suggested by many users, due to the fact that each such image is fetched to a list of listeners which I don't control, and might save a reference to the object. the only time when I'm certain to be able to delete the file is when no references are held, hence I thought finalize() is the right way. Any suggestions?
UPDATE 2:
What are the scenarios where finalize() will not be called? If the only possibles are exiting the program (in an expected/unexpected way), I can take it, because it means I risk only one unneeded temp file left un deleted (the one that was processed during exiting).
Another approach is to use File.deleteOnExit() which marks a file for the JVM to delete upon exit. I realise it's not quite what you're looking for, but may be of interest.
To be clear, if your JVM dies unexpectedly, it won't clear those files. As such, you may want to architect your solution to clear up cache files on startup, such that you don't build up a mass of unused cache files over time.
An good alternative to finalize is the PhantomReference. the best way to use it is:
public class FileReference extends PhantomReference<CachedImage> {
private final File _file;
public FileReference(CachedImage img, ReferenceQueue<CachedImage> q, File f) {
super(img, q);
_file = f;
}
public File getFile() {
_file;
}
}
Then use it like:
public class CachedImage {
private static final ReferenceQueue<CachedImage>
refQue = new ReferenceQueue<CachedImage>();
static {
Thread t = new Thread() {
#Override
public void run() {
try {
while (true) {
FileReference ref = (FileReference)refQue.remove();
File f = ref.getFile();
f.delete();
}
} catch (Throwable t) {
_log.error(t);
}
}
};
t.setDaemon(true);
t.start();
}
private final FileReference _ref;
public CachedImage(BufferedImage bi, File tempFile) {
tempFile.deleteOnExit();
saveAndFree(bi, tempFile);
_ref = new FileReference<CachedImage>(this, refQue, tempFile);
}
...
}
It is not recommended to use finalize().The problem is that you can't count on the garbage collector to ever delete an object. So, any code that you put into your class's overridden finalize() method is not guaranteed to run.
There's no guarantee that your finalize method will ever get called; in particular, any objects hanging around when the program exits are usually just thrown away with no cleanup. Closeable is a much better option.
As an alternative to #Brian Agnew's answer, why not install a ShutdownHook that clears out your cache directory?
public class CleanCacheOnShutdown extends Thread {
#Override
public void run() { ... }
}
System.getRuntime().addShutdownHook(new CleanCacheOnShutdown());
I ended up using a combination of File.deleteOnExit() (thanks #Brian), and a ScheduledExecutorService that goes over a ReferenceQueue of PhantomReferences to my class instances, according to this post.
I add this answer because no one suggested using ReferenceQueue (which I think is the ideal solution for my problem), and I think it will be helpful for future readers.
The (somewhat simplified) outcome is this (changed the class name to CachedImage):
public class CachedImage {
private static Map<PhantomReference<CachedImage>, File>
refMap = new HashMap<PhantomReference<CachedImage >, File>();
private static ReferenceQueue<CachedImage>
refQue = new ReferenceQueue<CachedImage>();
static {
Executors.newScheduledThreadPool(1).scheduleWithFixedDelay(new Thread() {
#Override
public void run() {
try {
Reference<? extends CachedImage> phanRef =
refQue.poll();
while (phanRef != null) {
File f = refMap.get(phanRef);
f.delete();
phanRef = refQue.poll();
}
} catch (Throwable t) {
_log.error(t);
}
}
}, 1, 1, TimeUnit.MINUTES);
}
public CachedImage(BufferedImage bi, File tempFile) {
tempFile.deleteOnExit();
saveAndFree(bi, tempFile);
PhantomReference<CachedImage> pref =
new PhantomReference<CachedImage>(this, refQue);
refMap.put(pref, tempFile);
}
...
}
I have a dedicated server running CentOS 5.9, Apache-Tomcat 5.5.36. I have written a JAVA web applications which runs every minute to collect the data from multiple sensors. I am using ScheduledExecutorService to execute the threads. (one thread for each sensor every minute and there can be more than hundred sensors) The flow of the thread is
Collect sensor information from the database.
Sends the command to the instrument to collect data.
Update the database with the data values.
There is another application that checks the database every minute and send the alerts to the users (if necessary). I have monitored the application using jvisualVM, I cant find any memory leak. for every thread. The applications work fine but after some time(24 Hour - 48 Hours) the applications stop working. I cant find out what the problem could be, is it server configuration problem, too many threads or what?
Does anyone have any idea what might be going wrong or is there anyone who has done think kind of work? Please help, Thanks
UPDATE : including code
public class Scheduler {
private final ScheduledExecutorService scheduler =
Executors.newScheduledThreadPool(1);
public void startProcess(int start) {
final Runnable uploader = new Runnable() {
#SuppressWarnings("rawtypes")
public void run()
{
//Select data from the database
ArrayList dataList = getData();
for(int i=0;i<dataList.size();i++)
{
String args = dataList.get(i).toString();
ExecutorThread comThread = new ExecutorThread(args...);
comThread.start();
}
}
};
scheduler.scheduleAtFixedRate(uploader, 0, 60 , TimeUnit.SECONDS);
}
}
public class ExecutorThread extends Thread {
private variables...
public CommunicationThread(args..)
{
//Initialise private variable
}
public void run()
{
//Collect data from sensor
//Update Database
}
}
Can't say much without a code, but you need to be sure that your thread always exits properly - doesn't hang in memory on any exception, closes connection to database, etc.
Also, for monitoring your application, you can take a thread dump every some period of time to see how many threads the application generates.
Another suggestion is configure Tomcat to take a heap dump on OutOfMemoryError. If that's an issue, you'll be able to analyze what is filling up the memory
Take heed of this innocuous line from the ScheduledExecutorService.schedule... Javadoc
If any execution of the task encounters an exception, subsequent executions are suppressed.
This means that if you are running into an Exception at some point and not handling it, the Exception will propagate into the ScheduledExecutorService and it will kill your task.
To avoid this problem you need to make sure the entire Runnable is wrapped in a try...catch and Exceptions are guaranteed to never be unhandled.
You can also extend the ScheduledExecutorService (also mentioned in the javadoc) to handle uncaught exceptions :-
final ScheduledExecutorService ses = new ScheduledThreadPoolExecutor(10){
#Override
protected void afterExecute(Runnable r, Throwable t) {
super.afterExecute(r, t);
if (t == null && r instanceof Future<?>) {
try {
Object result = ((Future<?>) r).get();
} catch (CancellationException ce) {
t = ce;
} catch (ExecutionException ee) {
t = ee.getCause();
} catch (InterruptedException ie) {
Thread.currentThread().interrupt(); // ignore/reset
}
}
if (t != null) {
System.out.println(t);
}
}
};
Here the afterExecute method simply System.out.printlns the Throwable but it could do other things. Alert users, restart tasks etc...