Having some performance issues in a Weblogic 11g production system.
As part of the debugging effort, I’m interested in finding a way to map threads to their memory usage, then seeing the stack to determine what part of the application is consuming so much.
Anyone know of a tool or method to do what I want to do?
I'm not interested in JProbe memory profiling as it requires too much overhead (taking snapshots of everything). Also, I've read about Heapwalker in NetBeans that seems promising.
Eclipse has a memory analyzer (or heap walker, if you will) called mat - http://eclipse.org/mat/.
I've used it in the past and it was pretty helpful. I don't remember off-hand all the features, but I do remember being able to identify "heavy" threads, querying for largest objects, and such.
The home page links to several tutorials and a blog that are useful as well.
Related
Quite recently i went thru the interview process of Adobe Systems. There is one question they asked to me is :-
"there is a J2EE application and there is memory leakage in the that application,and we don't have the source code of application, hereby how could you find the memory leakage"
i was clueless at that time so i said :-
"there are many third party tools i.e. there is one which is integrated with eclipse and many more. i don't know the mechanics of those tools."
Still i am searching for answer.
Thank You
You are right there are many tools like visualvm, jmeter. what they simply do is to hook to running jvm and collect data just like you simply get the Threaddumps with jstat or a heapdump, the tools are just fancy data analyser and provides visualisation, under the hood everything resides on heapdump and threaddump which can tell you the memory leak.
In your JDK folder, look in /bin and find "jvisualvm.exe" (Java VisualVM) and double-click. It will scan the machine processes for anything Java that is currently running and allow you to start monitoring its memory usage (including itself!). The interface is pretty intuitive so you ought to figure out how to use it fast enough.
Nice to see a free utility ship with a free app dev't kit that isn't all but useless... in fact, this one helped me a lot in tracking down places in one large, data-intensive project codebase where I really needed to execute System.gc() at particular times to keep me needing >1 GB of memory. (Contrary to religious belief, running System.gc() is in fact a perfectly reasonable thing to do when you need to free up memory that is needed sooner rather than later.) Previously, I was blowing the heap space at all the wrong times (and there's no right time to do that), but this utility helped me locate the places in my code most guilty of memory-holding so I could stop that from happening.
EDIT: Over 2 years later I want to add as follows. I have not used the cited JVM switches myself for the purpose of tracking down memory leaks but I want to share this info b/c it may help someone else.
At: http://javarevisited.blogspot.com/2011/11/hotspot-jvm-options-java-examples.html
Quote: '8) JVM parameters to trace classloading and unloading
-XX:+TraceClassLoading and -XX:+TraceClassUnloading are two JVM options which we use to print logging information whenever classes loads into JVM or unloads from JVM. These JVM flags are extremely useful if you have any memory leak related to classloader and or suspecting that classes are not unloading or garbage collected.'
Is there a way to determine/calculate the values of all of the memory usages that a Java program is using during runtime, and to continue to calculate them every so often? I am trying to create a visual representation of the total memory being used in a program and display it on some sort of chart or line graph, and continue to update the graph periodically for a project I am currently working on.
I've seen other questions that ask to view the memory and people have suggested VisualVM, but I do not just want to see the memory being used, but actually use the values of the memory used. (Does VisualVM do that?)
As a side question, is it possible to see how much memory a particular thread in the code is using, or can I only view the overall usage?
You can try
VisualVM . It
is a visual tool integrating several commandline JDK tools and lightweight profiling capabilities. Designed for both production and development time use, it further enhances the capability of monitoring and performance analysis for the Java SE platform.
I am using EHCache as second level cache for my application's Hibernate DAO layer. To implement distributed cache I am planning to include Terracotta Server.
Recently I came to know about Terracotta's another product Big Memory.
Few questions regarding that:
How is Big Memory will help on top of Terracotta/EhCache?
Will it compliment Terracotta/EhCache implementation?
Will it be worth giving a try?
I work on a Java EE application which has a flex UI, Hibernate ORM layer, SQL Server 2008 and Tomcat application server.
How will Big Memory help on top of Terracotta/EhCache?
The way I've understood the point of BigMemory is that it stores large amounts of data on the memory outside JVM. This will help if you have lots of stuff you want to be cached, so much so that the GC times are impacting your performance like explained here.
If your stuff fits with your cache just fine and you don't experience such slowups, I would imagine Big Memory can even slow you down in contrast to terracotta, as heap within JVM would be faster than outside JVM. At least, it would not improve much.
Will it compliment Terracotta/EhCache implementation?
Based on the documentation, integration to ehcache/terracotta should be quite seamless. So, yes.
Will it be worth giving a try?
I'd go first with Terracotta, measuring memory usage, GC times and the impact, and if it would seem like Big Memory could help some more, then sure. If it looks ok, no reason to add extra stuff.
BigMemoryGO offers upto 32 GB free usage. I would suggest to give a try to BigMemory.
BigMemory Go lets you keep all of your application's data instantly available in your server's memory so I don't think it will slow down your app in contrast to Terracotta.
I had an old application, a JAR file, that went through some enhancements. Basically some parts of the code had to be modified along with modifying some of the logic.
Comparing the OLD version against the NEW version, the NEW version is about 2X slower than the old one.
I'm trying to narrow down whats causing the slow down, but I'm finding myself measuring the time for certain for-loops using System.println with System.currentTimeMillis(). This is really getting very tedious.
Is there a Java performance tool that will help me in figuring out why the NEW JAR is about 2X slower than the old one?
Thanks in advance.
JProfiler has the capability to compare CPU snapshots. Record the execution for the old and the new JAR file and save snapshots (if the JVM exits at the end, configure a "JVM exit" trigger that saves a snapshot).
Then open the snapshot comparison window with "Session->Compare Snapshots in New Window" and add the two snapshot. A hot spots comparison will look like this (a view filter is set in this case):
It will immediately show you which methods are responsible for the increase in execution time.
Another way to analyze the differences in execution time is the call tree comparison which will look like this:
Disclaimer: My company develops JProfiler.
You should use a profiler. This will show you which methods are taking the most time (and what is calling them), without you having to guess which ones to measure.
Java comes with a built-in profiler called hprof, but see also:
https://stackoverflow.com/questions/14762/please-recommend-a-java-profiler
5 things you didn't know about ... Java performance monitoring
The JConsole and VisualVM tools
Depending on how long-running the process is, I'd think about Visual VM 1.3.3. If you download all the plugins, you'll be able to see heap, threads, objects, etc. That ought to help, and it won't cost a dime.
I believe it assumes the Oracle/Sun JVM.
A profiler tool like YourKit or something to measure performance reliably like Hyperic's Sigar is a good canditate for your case. Have a look at those tools.
The former will find bottlenecks in your code and/or memory leaks (not all of them) while as the latter is an API that you can measure performance reliably since Oracle's JVM & OpenJDK have no way of getting perfomance metrics reliably/consistently/accurately (like CPU wall clock time or CPU time spent from the application, memory usage, application threads, etc).
By default, Java provides packages for these things.
For example:
java.lang.management.ManagementFactory
java.lang.management.ThreadMXBean
but depending on your case they may or may not be adequate (keep in mind they are OK for most cases unless we are talking about something critical).
I am calling a vendor's Java API, and on some servers it appears that the JVM goes into a low priority polling loop after logging into the API (CPU at 100% usage). The same app on other servers does not exhibit this behavior. This happens on WebSphere and Tomcat. The environment is tricky to set up so it is difficult to try to do something like profiling within Eclipse.
Is there a way to profile (or some other method of inspecting) an existing Java app running in Tomcat to find out what methods are being executed while it's in this spinwait kind of state? The app is only executing one method when it gets in this state (vendor's method). Vendor can't replicate the behavior (of course).
Update:
Using JConsole I was able to determine who was running and what they were doing. It took me a few hours to then figure out why it was doing it. The problem ended up being that the vendor's API jar that was being used did not match exactly to the the database configuration that it was using. It was defaulting to having tracing and performance monitoring enabled on the servers that had the slight mis-match in configuration. I used a different jar and all is well.
So thanks, Joshua, for your answer. JConsole was extremely easy to setup and use to monitor an existing application.
#Cringe - I did some experimenting with some of the options you suggested. I had some problems with getting JProfiler set up, it looks good (but pricey). Going forward I went ahead and added the Eclipse Profiler plugin and I'll be looking over the different open source profilers to compare functionality.
If you are using Java 5 or later, you can connect to your application using jconsole to view all running threads. jstack also will do a stack dump. I think this should still work even inside a container like Tomcat.
Both of these tools are included with JDK5 and later (I assume the process needs to be at least Java 5, though I could be wrong)
Update:
It's also worth noting that starting with JDK 1.6 update 7 there is now a bundled profiler called VisualVM which can be launched with 'jvisualvm'. It looks like it is a java.net project, so additional info may be available at that page. I haven't used this yet but it looks useful for more serious analysis.
Hope that helps
Facing the same problem I used YourKit profiler. It's loader doesn't activate unless you actually connect to it (though it does open a port to listen for connections). The profiler itself has a nice "get amount of time spent in each method" while working in it's less obtrusive mode.
Another way is to detect CPU load (via JNI, so you'd need an external library for this) in a "watchdog" thread with highest priority and start logging all threads when the CPU is high enough for a long enough time. You might find this article enlightining.
If it's for professional purpose and you have some money to spend, try to get your hands on JProfiler. If you just want to get some insights, try out the Eclipse Profiler Plugin. I used it several times, but I don't know the current state.
A new(?) project from the eclipse project itself is available too: http://www.eclipse.org/tptp/ (See this article). Never used it, so I can't tell if it is worth the effort.
There's also a very good list of open source profilers available at http://www.manageability.org/blog/stuff/open-source-profilers-for-java
If JConsole can't be used you can
press CTRL+BREAK under Windows
send kill -3 <process id> under Linux
to get a full Thread Dump. This doesn't affect performance and can always be run in production.
JRockit Mission Control Latency Analyzer.
The Latency Analyzer that comes with JRockit shows you what the JVM is "doing" when it's not doing anything. In the latest version you can see latencies for:
Java wait/blocked/sleep/parked.
File I/O
Network I/O
Memory allocation
GC pauses
JVM latencies, e.g code generation and class loading
Thread suspension
The tool will give you the stack trace when the latency occurred. You can view the latency data in many different ways (aggregated traces, as a histogram, in a thread graph etc.). The tool also allows you to see transitions between threads, for instance when one thread notifies another.
latency analyzer http://blogs.oracle.com/hirt/WindowsLiveWriter/The.0LatencyAnalyserMigratedfromtheoldBE_7246/latency_graph_2.png
The overhead is negligible and unlike many other tools it can be used in a production environment.
This blog post gives you a brief introduction and the program can be downloaded here.
It's free to use for development!
Use a profiler. Yes they cost money, and using them can occasionally be a bit awkward, but they do provide you with a great deal more real evidence rather than guesswork.
Human beings are universally bad at guessing where performance bottlenecks are. It just seems to be something our brains aren't build to do very well. It may seem obvious, you may have great ideas about what the problem is, but the real world often turns out to be doing something different. And optimising the wrong part of code means, at best, lots of work for minimal benefit. More often it makes things slower, and sometimes it breaks things entirely. So before you make any changes for the sake of optimisation, you should always have real evidence from a profiler or other accurate tool.
As mentioned, both JProfiler and YourKit are both fairly good and not prohibitively expensive. Last time I looked, they both had free demos too.
For completeness sake: even though my company more or less standardizes on Eclipse we use Netbeans (6 and up) with its included, free profiler on a daily basis. It works better than the Eclipse TPTP plugin (last checked 3 months ago) and for us it removes any need for a commercial profiler such as JProfiler, which is excellent, but fast becoming unnecessary.
VisualVM should be the profiler from netbeans as standalone. I tried the TPTP for eclipse but visualVm seems as a much nicer option!