We Keep Coding

Critical guava memory leak - Workaround needed

Is there any way to workaround the Google Guava r15 memory leak (link to the bug report) in the cache component?
(Without relying that the application server might clean things up and/or considering that the web application will never be restarted/redeployed)

I guess you don't need to care about it. The Tomcat message says
Threads are going to be renewed over time to try and avoid a probable memory leak.
IIUIC it means that once all old threads are gone, so will all the pointers to the old version of your class.
Details: The reason for the thread pooling is the big cost of thread creation. The pooling itself is hacky as you get a thread which was doing something else and thread are not stateless. Thread creation is expensive assuming you'd need a lot of them and never recycle them. There's nothing wrong with renewing all threads every few minutes, so I hoped, Tomcat's workaround solves it perfectly. But it's not the case.
EDIT
I'm afraid, I misunderstood something. The linked bug says
It seems that web applications which are using guava cache might face a memory leak.
After several redeployments, the application container crashes or stalls with an OutOfMemoryError.
I thought Tomcat could solve it easily, but for whatever reason it doesn't. So I'm afraid, you have to clean the ThreadLocals yourself. This is easily possible via reflection, the concerned fields are Thread.threadLocals and possibly inheritableThreadLocals. It's a bad hack and the harder part is to make this happen when nothing can go wrong, i.e., when no application is loaded.
EDIT 2 and 3
I guess it's safe to do something like
Stripped64.threadHashCode = new ThreadHashCode();
as the contained things are only needed for performance under heavy contention and they get recreated upon use. But according to MRalwasser's comment, it won't help at all as alive threads will still refer the old value. So there seem to be no way.
As ThreadLocal works by storing data with the threads (rather then using a real Map<Thread, Something>), you'd have to through all threads and remove references there. Fooling around with other threads' private fields is a terrible idea as they are not thread-safe and also due to visibility issues.
Another thing that might or mighn't not work is my proposal on the issue page. It's just a 20 line patch. Or simply wait, the issue has been assigned yesterday.
EDIT 4
Thread locals which don't get used can't cause any problems. AFAIK the only use of this TL is in cache stats. So avoid both CacheBuilder.recordStats and Cache.stats and Stripped64 won't get loaded.
EDIT 5
It looks like it's gonna get fixed finally. From the issue:
Doug fixed this upstream for us and we patched it back into Guava:
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/Striped64.java?revision=1.9
At the first glance his change seems to be identical to mine.
EDIT 6
Finally, this has been marked as fixed and Guava 18.0-rc1 has been announced. It's just sad it took that long given that the change is the same as mine (9 month ago).

You can use the ServletListener ClassLoaderLeakPreventor https://github.com/mjiderhamn/classloader-leak-prevention/ which also clears ThreadLocals on undeploy/stop. It also has fixes/workarounds for other common leaks.

Seems to be a drawback of ThreadLocals. You'll get the same every time you put an application level class in ThreadLocal.
The only workaround is to restart server on deploy I guess. I think it's a known issue of Java applications. Are you sure that it's the only place that stops unloading classloader?

Tomcat behaves as if it's not threading requests, causes slow response time

I'm seeing strange behavior and I don't know how to gain any further insight into and am hoping someone can help.
Background: I have a query that takes a long time to return results so instead of making the user wait for the data directly upon request I execute this query via a Timer object at regular intervals and store the results in a static variable. Therefore, when the user requests the data I always just pull from the static variable, therefore making the response virtually instant. So far so good.
Issue: The behavior I'm seeing, however, is that if I make a request for the data just as the background (Timer) request has begun to query the data, my user's request waits for the data to come back before responding -- forcing the user to wait. It's as if tomcat is behaving synchronously with the threads (I know it's not -- it just looks that way).
This is in a Production environment and, for the most part, everything works great but for users there are times when the site just hangs for them and they feel it's unreliable (well, in a sense it is).
What I've done: Being that the requests for the data were in a static method I thought "A ha! The threads are syncronized which is causing the delay!" so i pulled all of my static methods out, removed the syncronization and forced each call to instantiate it's own object to retrieve the data (to keep it thread safe). There isn't any syncronization on a semaphore to the static variable either.
I've also installed javamelody to try and gain some insight into what's going on but nothing new thus far. I have noticed a lot (majority) of threads are in "WAITING" state but they also have 0ms for User and CPU time so don't think that is pointing to anything(?).
Running Tomcat 5.5 (no apache layer), struts 2, Java 1.5
If anyone has any idea why a simple request to a static variable hangs for longer background processes I would really appreciate it! Or if you know how I can gain insight that would be great too.
Thanks!

One possible explanation is that the threads are actually blocking at the database level due to database locking (or something) caused by the long-running query.
The way to figure out what is going on is to find out exactly where the blocked threads are blocking. A thread dump can be produced by sending a SIGQUIT (or equivalent) to the JVM, and included stack traces for all Java thread stacks. Alternatively, you can get the same information (and more) by attaching a debugger, etcetera. Either way, the class name and line number of the top frame of each stack should allow you to look at the source code and figure out (at least) what kind of locking or blocking is going on.

For those who would like to know I eventually found VisualVM (http://visualvm.java.net/download.html). It's perfect. I run Tomcat from eclipse like I normally do and it appears within the VisualVM client. Right-mouse click the tomcat icon, choose Thread Dump and, boom, I've got it all.
Thanks, all, for the help and pointers towards the right direction!

Java .jar uses too much memory

I'm making an application in Java using Eclipse Indigo. When I run it using Eclipse the Task Manager shows javaw.exe is using 50mb of memory. When I export the application as a runnable .jar and execute the .jar the Task Manager shows javaw.exe is using 500mb.
Why is this? How could I fix this?
Edit: I'm using a Windows 7 64 bit, and my system says I have Java 1.7 installed. Apparently the memory problem is caused by a while loop. I'll study what's inside the while loop causing the problem.
Edit: Problem found. At one point in the while loop new BufferedImage instances where created, instead of replacing the same BufferedImage.

Without any additional details about your code, I would suggest using a profiler to analyze the problem. I know YourKit and the one that is available for NetBeans are very good.
Once you run you app from the profiler, you should initially look at the objects and listeners created by your application's packages. If the issue is not there, you can expand your search to other packages until you identify things that are growing out-of-control, and then look at the code that handles those entities.
When you run certain parts of the code multiple times and still see memory utilization after that code stopped running, then you might have a leak and may consider nulling or emptying variables/listeners on exit.
It should be a good starting point, but please report your results back, so we know how it goes. By the way, what operating system are you using and what version of java?
--Luiz

You need to profile your code to get the exact answer, but from my experience when I see similar things I often equate it to garbage collecting. For example, I ran the same job and gave one job 10 gigs and the other 2 gigs..Both ran and completed but the 10gigs one used more memory(and finished faster) while the second(2gig) one, I believe, garbage collected so it still completed but took a bit more time with less memory. I'm a bit new to java so I maybe assuming the garbage collecting but I have seen what you are talking about.
You need to profile your code(check out jconsole, which is included with java, or visualVM)..

That sounds most peculiar.
I can think of two possible explanations:
You looked at the wrong javaw.exe instance. Perhaps you looked at the instance that is running Eclipse ... which is likely to be that big, or bigger.
You have (somehow) managed to configure Java to run with a large heap by default. On Linux you could do this with a wrapper script, a shell function or a shell alias. You can do at least the first of those on Windows.
I don't think it is the JAR file itself. AFAIK, you can't set JVM parameters in a JAR file. It is possible that you've somehow included a different version of something in the JAR file, but that's a bit of a stretch ...
If none of these ideas help, try profiling.
Problem found. At one point in the while loop new BufferedImage instances where created, instead of replacing the same BufferedImage.
Ah yes. BufferedImage uses large amounts of out-of-heap memory and that needs to be managed carefully.
But this doesn't explain why your application used more memory when run from the JAR than when launched from Eclipse ... unless you were telling the application to do different things.

Dumping a Java program into a file and restarting it

I was just wondering if it's possible to dump a running Java program into a file, and later on restart it (same machine)
It's sounds a bit weird, but who knows
--- update -------
Yes, this is the hibernate feature for a process instead of a full system. But google 'hibernate jvm process' and you'll understand my pain.
There is a question for linux on this subject (here). Quickly, it's possible to hibernate a process (far from 100% reliable) with CryoPID.
A similar question was raised in stackoverflow some years ago.
With a JVM my educated guess is that hibernating should be a lot easier, not always possible and not reliable at 100% (e.g. UI and files).
Serializing a persistent state of the application is an option but it is not an answer to the question.

This may me a bit overkill but one thing you can do is run something like VirtualBox and halt/save the machine.
There is also:
- JavaFlow from Apache that should do just that even though I haven't personally tried
it.
- Brakes that may be exactly what you're looking for

There are a lot restrictions any solution to your problem will have: all external connections might or might not survive your attempt to freeze and awake them. Think of timeouts on the other side, or even stopped communication partners - anything from a web server to a database or even local files.
You are asking for a generic solution, without any internal knowledge of your program, that you would like to hibernate. What you can always do, is serialize that part of the state of your program, that you need to restart your program. It is, or at least was common wisdom to implement restart point in long running computations (think of days or weeks). So, when you hit a bug in your program after it run for a week, you could fix the bug and save some computation days.
The state of a program could be surprisingly small, compared to the complete memory size used.
You asked "if it's possible to dump a running Java program into a file, and later on restart it." - Yes it is, but I would not suggest a generic and automatic solution that has to handle your program as a black box, but I suggest that you externalize the important part of your programs state and program restart points.
Hope that helps - even if it's more complicated than what you might have hoped for.

I believe what the OP is asking is what the Smalltalk guys have been doing for decades - store the whole programming/execution environment in an image file, and work on it.
AFAIK there is no way to do the same thing in Java.

There has been some research in "persisting" the execution state of the JVM and then move it to another JVM and start it again. Saw something demonstrated once but don't remember which one. Don't think it has been standardized in the JVM specs though...
Found the presentation/demo I was thinking about, it was at OOPSLA 2005 that they were talking about squawk
Good luck!
Other links of interest:
Merpati
Aglets
M-JavaMPI

How about using SpringBatch framework?
As far as I understood from your question you need some reliable and resumable java task, if so, I believe that Spring Batch will do the magic, because you can split your task (job) to several steps while each step (and also the entire job) has its own execution context persisted to a storage you choose to work with.
In case of crash you can recover by analyzing previous run of specific job and resume it from exact point where the failure occurred.
You can also pause and restart your job programmatically if the job was configured as restartable and the ExecutionContext for this job already exists.
Good luck!

I believe :
1- the only generic way is to implement serialization.
2- a good way to restore a running system is OS virtualization
3- now you are asking something like single process serialization.
The problem are IOs.
Says your process uses a temporary file which gets deleted by the system after
'hybernation', but your program does not know it. You will have an IOException
somewhere.
So word is , if the program is not designed to be interrupted at random , it won't work.
Thats a risky and unmaintable solution so i believe only 1,2 make sense.

I guess IDE supports debugging in such a way. It is not impossible, though i don't know how. May be you will get details if you contact some eclipse or netbeans contributer.

First off you need to design your app to use the Memento pattern or any other pattern that allows you to save state of your application. Observer pattern may also be a possibility. Once your code is structured in a way that saving state is possible, you can use Java serialization to actually write out all the objects etc to a file rather than putting it in a DB.
Just by 2 cents.

What you want is impossible from the very nature of computer architecture.
Every Java program gets compiled into Java intermediate code and this code is then interpreted into into native platform code (when run). The native code is quite different from what you see in Java files, because it depends on underlining platform and JVM version. Every platform has different instruction set, memory management, driver system, etc... So imagine that you hibernated your program on Windows and then run it on Linux, Mac or any other device with JRE, such as mobile phone, car, card reader, etc... All hell would break loose.
You solution is to serialize every important object into files and then close the program gracefully. When "unhibernating", you deserialize these instances from these files and your program can continue. The number of "important" instances can be quite small, you only need to save the "business data", everything else can be reconstructed from these data. You can use Hibernate or any other ORM framework to automatize this serialization on top of a SQL database.

Probably Terracotta can this: http://www.terracotta.org
I am not sure but they are supporting server failures. If all servers stop, the process should saved to disk and wait I think.
Otherwise you should refactor your application to hold state explicitly. For example, if you implement something like runnable and make it Serializable, you will be able to save it.

Is it likely appdomains will help in this scenario?

I have a dotnet process that through calls to an unmanaged dll is communicating with a Java process.
Under some circumstances, the Java process appears to be crashing and taking my dotnet process down with it. No exceptions are raised, the process just dies. Upon crashing, java is creating a log file with names like "hs_err_pid3228" etc.
Not having received any satisfaction from the vendor that is providing the unmanaged dll and the java process, I am reduced to trying to mitigate the problem which would necessitate ensuring the calls into the java process, if they crash, don't take down my process.
Having read various articles, appdomains seem a likely candidate to use - my theory being I can with a bit of work separate my functionality that calls the java process and run it in a separate appdomain, which will hopefully allow me to if not catch the appdomain going down, at least detect that it has happened and restart that functionality.
Has anyone had a similar sort of issue? Does this approach seem reasonable to those of you with more experience of appdomain?
To make it even more fun, the Java crash is not really reproducible - it seems very random and I'm still battling with how I'm going to TEST that separating into the appdomain

This is a reasonable use of AppDomains, and what you propose will work.
In a similar vein, I once used AppDomains to create a single application that watched for itself crashing for exception reporting purposes. The application started itself up, created a new AppDomain, then re-executed itself in the new AppDomain, which then detected it was running in an AppDomain and executed normally. When an exception happened in that AppDomain, the original process is notified, it tears down the child domain reports to the user that an error occured, asks whether they want to report it or not, then picked itself up and tried it all over again.
EDIT: To give you a headstart, if you want to look at the Program.cs for that project, I've uploaded a stripped down version here. (It's pretty long, so I didn't think I should post it here.)

Yep, leveraging AppDomains make a lot of sense here.
I've recently reworked my Windows service to load its various WCF services as plug-ins that operate within their own AppDomain. I've got a few cases in the bootstrapping process where I'm using MarshalByRefObject objects to get things up and running, but once the plug-ins are loaded, communication between the AppDomains is extremely easy using WCF.