How come every object appears to be marked new, instead of just objects that are in the second snapshot but not in my baseline snapshot? Looking around online, I see some suggestions that I need to use hprof instead of jmap to make my memory dumps, but it appears that hprof generates dumps in exactly the same format.
This is JDK 1.6.0_14; I have tried on both Windows and UNIX.
jhat -baseline indeed won't work with dumps produced by jmap. I'm not certain, but I believe this is because hprof attaches to the JVM right from the start and keeps its own track of objects, allowing it to produce consistent IDs across multiple dumps. Don't quote me on that. Either way, the important point as far as you're concerned is that jmap dumps don't work.
However, all is not lost. Go and get the Eclipse Memory Analyzer. (If you don't use Eclipse, fear not, you can get it as a standalone executable.) It's faster than jhat, uses less memory than jhat and it can do what you want:
Open dump2 (with File|Open Heap Dump). Don't bother having it create a report for you.
Open dump1 (same way). Again, no report.
In the tab for dump2, click "Histogram"
On the right of the toolbar in the Histogram subtab is "Compare to another Heap Dump". Click it.
Select dump1 from your dialog as the dump to use as the baseline.
Presto, you have the differences between dump2 and the baseline dump1.
All of this works fine with a jmap dump.
It seems that you need to use hprof. But are you sure you use the same VM instance ?
The -baseline option allows two dumps to be compared if they were produced by HPROF and from the same VM instance. If the same object appears in both dumps it will be excluded from the list of new objects reported. One dump is specified as a baseline and the analysis can focus on the objects that are created in the second dump since the baseline was obtained.
If everything is considered as new, I will make sure that's the same instance of the VM.
Here
Related
I have a HotSpot JVM heap dump that I would like to analyze. The VM ran with -Xmx31g, and the heap dump file is 48 GB large.
I won't even try jhat, as it requires about five times the heap memory (that would be 240 GB in my case) and is awfully slow.
Eclipse MAT crashes with an ArrayIndexOutOfBoundsException after analyzing the heap dump for several hours.
What other tools are available for that task? A suite of command line tools would be best, consisting of one program that transforms the heap dump into efficient data structures for analysis, combined with several other tools that work on the pre-structured data.
Normally, what I use is ParseHeapDump.sh included within Eclipse Memory Analyzer and described here, and I do that onto one our more beefed up servers (download and copy over the linux .zip distro, unzip there). The shell script needs less resources than parsing the heap from the GUI, plus you can run it on your beefy server with more resources (you can allocate more resources by adding something like -vmargs -Xmx40g -XX:-UseGCOverheadLimit to the end of the last line of the script.
For instance, the last line of that file might look like this after modification
./MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$#" -vmargs -Xmx40g -XX:-UseGCOverheadLimit
Run it like ./path/to/ParseHeapDump.sh ../today_heap_dump/jvm.hprof
After that succeeds, it creates a number of "index" files next to the .hprof file.
After creating the indices, I try to generate reports from that and scp those reports to my local machines and try to see if I can find the culprit just by that (not just the reports, not the indices). Here's a tutorial on creating the reports.
Example report:
./ParseHeapDump.sh ../today_heap_dump/jvm.hprof org.eclipse.mat.api:suspects
Other report options:
org.eclipse.mat.api:overview and org.eclipse.mat.api:top_components
If those reports are not enough and if I need some more digging (i.e. let's say via oql), I scp the indices as well as hprof file to my local machine, and then open the heap dump (with the indices in the same directory as the heap dump) with my Eclipse MAT GUI. From there, it does not need too much memory to run.
EDIT:
I just liked to add two notes :
As far as I know, only the generation of the indices is the memory intensive part of Eclipse MAT. After you have the indices, most of your processing from Eclipse MAT would not need that much memory.
Doing this on a shell script means I can do it on a headless server (and I normally do it on a headless server as well, because they're normally the most powerful ones). And if you have a server that can generate a heap dump of that size, chances are, you have another server out there that can process that much of a heap dump as well.
First step: increase the amount of RAM you are allocating to MAT. By default it's not very much and it can't open large files.
In case of using MAT on MAC (OSX) you'll have file MemoryAnalyzer.ini file in MemoryAnalyzer.app/Contents/MacOS. It wasn't working for me to make adjustments to that file and have them "take". You can instead create a modified startup command/shell script based on content of this file and run it from that directory. In my case I wanted 20 GB heap:
./MemoryAnalyzer -vmargs -Xmx20g --XX:-UseGCOverheadLimit ... other params desired
Just run this command/script from Contents/MacOS directory via terminal, to start the GUI with more RAM available.
I suggest trying YourKit. It usually needs a little less memory than the heap dump size (it indexes it and uses that information to retrieve what you want)
The accepted answer to this related question should provide a good start for you (if you have access to the running process, generates live jmap histograms instead of heap dumps, it's very fast):
Method for finding memory leak in large Java heap dumps
Most other heap analysers (I use IBM http://www.alphaworks.ibm.com/tech/heapanalyzer) require at least a percentage of RAM more than the heap if you're expecting a nice GUI tool.
Other than that, many developers use alternative approaches, like live stack analysis to get an idea of what's going on.
Although I must question why your heaps are so large? The effect on allocation and garbage collection must be massive. I'd bet a large percentage of what's in your heap should actually be stored in a database / a persistent cache etc etc.
This person http://blog.ragozin.info/2015/02/programatic-heapdump-analysis.html
wrote a custom "heap analyzer" that just exposes a "query style" interface through the heap dump file, instead of actually loading the file into memory.
https://github.com/aragozin/heaplib
Though I don't know if "query language" is better than the eclipse OQL mentioned in the accepted answer here.
The latest snapshot build of Eclipse Memory Analyzer has a facility to randomly discard a certain percentage of objects to reduce memory consumption and allow the remaining objects to be analyzed. See Bug 563960 and the nightly snapshot build to test this facility before it is included in the next release of MAT. Update: it is now included in released version 1.11.0.
A not so well known tool - http://dr-brenschede.de/bheapsampler/ works well for large heaps. It works by sampling so it doesn't have to read the entire thing, though a bit finicky.
This is not a command line solution, however I like the tools:
Copy the heap dump to a server large enough to host it. It is very well possible that the original server can be used.
Enter the server via ssh -X to run the graphical tool remotely and use jvisualvm from the Java binary directory to load the .hprof file of the heap dump.
The tool does not load the complete heap dump into memory at once, but loads parts when they are required. Of course, if you look around enough in the file the required memory will finally reach the size of the heap dump.
I came across an interesting tool called JXray. It provides limited evaluation trial license. Found it very useful to find memory leaks. You may give it a shot.
Try using jprofiler , its works good in analyzing large .hprof, I have tried with file sized around 22 GB.
https://www.ej-technologies.com/products/jprofiler/overview.html
$499/dev license but has a free 10 day evaluation
When the problem can be "easily" reproduced, one unmentioned alternative is to take heap dumps before memory grows that big (e.g., jmap -dump:format=b,file=heap.bin <pid>).
In many cases you will already get an idea of what's going on without waiting for an OOM.
In addition, MAT provides a feature to compare different snapshots, which can come handy (see https://stackoverflow.com/a/55926302/898154 for instructions and a description).
I have encountered some strange text formatting in our application and I need to find the cause. The problem is I have no idea where to start looking in our Legacy code.Is it possible to make the IntelliJ IDEA debugger search for a value throughout the entire instance of an application in the JVM? As opposed to the usual way of setting a breakpoint with a condition.
Judging from the comment on this question, there is no such feature in IntelliJ.
You could however try performing a heap dump and then searching it using OQL. It can be done in the VisualVM tool which is bundled with the JDK:
Run the jvisualvm command (assuming you have the JDK's bin folder in your PATH).
Start your application, find it in the left column under Local, and select Heap Dump from the right-click menu.
Head to the OQL Console view. In Query Editor, type your query, for example:
select s from java.lang.String s where s.toString().contains("hello")
and execute it.
If your object is on the heap, you should see it. Click it and check the referencing objects in References.
This should get you closer to the classes/objects your object is used by.
There are some tricky parts:
you have to perform the heap dump before the object in question is garbage-collected - that is, as soon as possible after you are sure the object was created (other ways to acquire a heap dump are described here),
the object can be unreachable. In such cases, try to look through the object graph it is part of and find other objects which are still referenced,
the object may not live in the heap at all (escape analysis).
Another tool for heap dump analysis is Eclipse Memory Analyzer (MAT). (For your use case, enable the Keep unreachable objects option before analyzing the heap dump.)
You mentioned that you need to search for the value at runtime. However, if you suspect what you are looking for is a string literal, try just searching the JARs of your application as described in this answer.
The application i am working on suddenly crashed with
java.io.IOException: ... Too many open files
As i understand the issue it means that files are opened but not closed.
Stacktrace of course happens after the fact and can only help understand before what event error occurred.
What would be an intelligent way to search your code base to find this issue which only seems to occur when app is under high stress load.
use lsof -p pid to check what cause leak of file references;
use ulimit -n to see the limit of opened file references of a single process;
check any IO resources in your project,are they released in time?,Note that,File,Process,Socket(and Http connections) are all IO resources.
sometimes, too many threads will cause this problem too.
I think the best way to use a tool specifically designed for the purpose, such as this one:
This little Java agent is a tool that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
In addition, upon "too many open files" exception, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.
I seem to remember YourKit also having some facilities around this, but can't find any specific information at the moment.
What OS? If it's linux/mac, there is information under /proc that should help. On Windows, use the Process Explorer.
As far as searching the code base, perhaps look for code that catches or raises IOException - I think I/O methods that already catch/raise this have a high likelihood of needing a close() call.
Have you tried attaching to the running process using jvisualvm (Java 5.0 and later in the JDK bin directory). You can open the running process and do a heap dump (which if you have an older JDK you will need to analyze using eclipse or intellij or netbeans et. al.).
In JDK 7 the heap dump button is under the "Monitor" tab. It will create a heap dump tab, "Classes" sub-tab that you can check and see if any classes that open files exist in high quantity. Another very useful feature is heap dump compare, so you can take a reference heap dump, let your app run a bit and then take another heap dump and compare the two (the link to compare is on the "[heapdump]" tab you get when you take one. There is also a flag in java for taking a heapdump on crash or OOM exception, you can go down that route if comparing heap dumps does not give you an obvious class that is causing the problem. Also, "Instances" subtab in the heap dump diff will show you what has been allocated in the time between the two heap dumps which may also help.
jvisualvm is an awesome tool that does not get enough mentions.
My application currently consumes quite a lot of memory because it is running physics simulations. The issue is that consistently, at the 51st simulation, Java will throw an error usually because of a heap space out of memory (my program eventually runs thousands of simulations).
Is there anyway I can not just increase the heap space but modify my program so that the heap space is cleared after every run so that I can run an arbitrary number of simulations?
Edit: Thanks guys. Turns out the simulator software wasn't clearing the information after every run and I had those runs all stored in an ArrayList.
There is no way to dynamically increase the heap programatically since the heap is allocated when the Java Virtual Machine is started.
However, you can use this command
java -Xmx1024M YourClass
to set the memory to 1024
or, you can set a min max
java -Xms256m -Xmx1024m YourClassNameHere
If you are using a lot of memory and facing memory leaks, then you might want to check if you are using a large number of ArrayLists or HashMaps with many elements each.
An ArrayList is implemented as a dynamic array. The source code from Sun/Oracle shows that when a new element is inserted into a full ArrayList, a new array of 1.5 times the size of the original array is created, and the elements copied over. What this means is that you could be wasting up to 50% of the space in each ArrayList you use, unless you call its trimToSize method. Or better still, if you know the number of elements you are going to insert before hand, then call the constructor with the initial capacity as its argument.
I did not examine the source code for HashMap very carefully, but at a first glance it appears that the array length in each HashMap must be a power of two, making it another implementation of a dynamic array. Note that HashSet is essentially a wrapper around HashMap.
There are a variety of tools that you can use to help diagnose this problem. The JDK includes JVisualVM that will allow you to attach to your running process and show what objects might be growing out of control. Netbeans has a wrapper around it that works fairly well. Eclipse has the Eclipse Memory Analyzer which is the one I use most often, just seems to handle large dump files a bit better. There's also a command line option, -XX:+HeapDumpOnOutOfMemoryError that will give you a file that is basically a snapshot of your process memory when your program crashed. You can use any of the above mentioned tools to look at it, it can really help a lot when diagnosing these sort of problems.
Depending on how hard the program is working, it may be a simple case of the JVM not knowing when a good time to garbage collect may be, you might also look into the parallel garbage collection options as well.
I also faced the same problem.I resolved by doing the build by following steps as.
-->Right click on the project select RunAs ->Run configurations
Select your project as BaseDirectory. In place of goals give eclipse:eclipse install
-->In the second tab give -Xmx1024m as VM arguments.
I would like to add that this problem is similar to common Java memory leaks.
When the JVM garbage collector is unable to clear the "waste" memory of your Java / Java EE application over time, OutOfMemoryError: Java heap space will be the outcome.
It is important to perform a proper diagnostic first:
Enable verbose:gc. This will allow you to understand the memory growing pattern over time.
Generate and analyze a JVM Heap Dump. This will allow you to understand your application memory footprint and pinpoint the source of the memory leak(s).
You can also use Java profilers and runtime memory leak analyzer such as Plumbr as well to help you with this task.
Try adding -Xmx for more memory ( java -Xmx1024M YourClass ), and don't forget to stop referencing variables you don't need any more (memory leaks).
Are you keeping references to variables that you no longer need (e.g. data from the previous simulations)? If so, you have a memory leak. You just need to find where that is happening and make sure that you remove the references to the variables when they are no longer needed (this would automatically happen if they go out of scope).
If you actually need all that data from previous simulations in memory, you need to increase the heap size or change your algorithm.
Java is supposed to clear the heap space for you when all of the objects are no longer referenced. It won't generally release it back to the OS though, it will keep that memory for it's own internal reuse. Maybe check to see if you have some arrays which are not being cleared or something.
No. The heap is cleared by the garbage collector whenever it feels like it. You can ask it to run (with System.gc()) but it is not guaranteed to run.
First try increasing the memory by setting -Xmx256m
How do I compare memory heap dumps in Netbeans?
What I have done is to configure my project to use profiling, then added several profiling points at chosen lines of code (similar to setting breakpoints). These profiling points trigger a "snapshot", which creates a memory dump.
As my application is running, the profiling tab lists each of the profiling points, and marks the number of hits it has encounteed, providing a link to open a report for that profiling point. In addition, the profiler control panel adds the new snapshots to a list.
If I open these reports and follow the links, or open the snapshots from the control panel, in both cases, Netbeans opens a snapshot tab with Summary, Classes, Instances and OQL subscreens.
If I select one of them from the Profiler control panel, and press save, the file gets saved in HPROF format.
If I select the menu Profile --> Compare Memory Snapshots, it only allows me to select NPS format files, of which I cannot obtain any.
I believe Compare Memory Snapshots sounds like it should be able to compare heap dumps, but I cannot figure out how to do it.
Questions
Where is Netbeans putting the NPS files (if it is indeed creating them)? If not how do I get Netbeans to create NPS files triggered from profiling points?
Is there a way to get Netbeans to compare HPROF files, since that is the memory heap dump after all?
Or am I simply completely overlooking something altogether?
Thanks!
Background
Using Netbeans 6.7.1.
I am doing memory profiling for a really large & complex application that has a memory leak problem. I have managed to solve large chunks of the leaks using a combination of Netbeans' profiler & JHAT (command line util that comes with JDK). It's the remaining stragglers that I need a more powerful heap analysis capabilities for - the
hypothesis-first approach that worked previously is becoming decreasingly effective.
This question's really specific, for more background see a question I have asked previously.
Unfortunately, I was not asking for recommendations on other tools which can do this.
I have since found that Netbeans does indeed support comparison of memory heap dumps, but only indirectly. Edit the profiling points such that the "Snapshot" is selected over "heap". This means that Netbeans will produce a .nps file instead of a .hprof file.
The .nps file is Netbean's own proprietary file format, whereas .hprof is the "universal" heap dump format, understood by almost all profiling tools, including JHAT, MAT & Netbeans.
However, the limitation with Netbeans is that it can look at .hprof files, and analyse them, but it cannot compare them to another .hprof file. It can, though, compare one .nps file to another .nps file. However the level of analysis available is limited with .nps files, so that's the trade-off.
So, to answer my own questions:
Where is Netbeans putting the NPS files (if it is indeed creating them)? If not how do I get Netbeans to create NPS files triggered from profiling points?
It is one or the other, you can only choose one per profiling point.
Is there a way to get Netbeans to compare HPROF files, since that is the memory heap dump after all?
No. You cannot compare to HPROF's. The only way to compare the memory at two different points in time appears to be to compare two NPS'es.
I would suggest using eclipse's memory analyzer: http://www.eclipse.org/mat/
I know that you are a netbeans user but mat is actually a standalone application (rcp based) and is capable of loading and comparing hprof files.
I found that it was the best heap analyzer around, particularly with larger heap dumps.