I have an app server process that's constantly at 100% CPU. By constantly I mean hours, or even days.
I know how to generate a heap/thread dump, but I'm looking for more dynamic information. I would like to know what is using so much CPU in there. There are tens (or probably 100+) threads. I know what those threads are, but I need to know which of them are using my CPU so much.
How can I obtain this information?
Use a profiler. There is one included in VisualVM which comes with the Oracle JDK.
An advanced commercial one (trial licenses available) is YourKit.
By creating a thread dump. You can use the jstack to connect to a running java process to get the thread dump. If you take two or more thread dumps over a period of time you can by analyzing them figure out which ones are actively using CPU. Typically the threads in the RUNNING state are the ones you need to focus on.
I personally use YourKit for this.
VisualVM also has some profiling capabilities, but I haven't used them.
in linux try kill -3 processid it will generate thread dump. You can analyze this to see what is happening in the java process.
Related
How do I find top 10 or top few CPU intensive thread stack traces created by Java process in Linux ? I would like to know how much time spent as well if possible
This is simple and easy. And it worked! We need more tools like this in Java.
https://github.com/patric-r/jvmtop
You get the below information by using one command jvmptop.sh <pid>
Standard linux tool like top will just give top processes, which are consuming the most cpu. but will not be able to tell that what all the threads inside a single java process is taking most of the cpu.
You need a profiling tool like YourKit to determine what threads in java process is using most of the cpu and you can enable trace based sampling in yourkit to even get the invocation count of a method as well.
please refer https://www.yourkit.com/docs/java/help/cpu_intro.jsp doc , on how to get started with CPU profiling using yourkit.
I need to monitor the performance of a Java process and take reports automatically. The reports should contain data on memory utilization thread usage, process usage etc. But I'm unsure how to accomplish this. Any suggestions?
I need to monitor the performance of a Java process and take reports automatically.
You need to determine what measures are important to the users of the application like latency and throughput. These are often impacted even if everything looks fine system wise. For example an 8 cpu system which is only 6% busy over 5 minutes might sound fine, except it could be that there is one request every 5 minutes which is taking more than 2 minutes.
The reports should contain data on memory utilization thread usage,
A key feature of threads share objects by default. This means the thread local memory usage is almost always trivial and not worth measuring in general.
process usage etc.
This can be useful for capacity planning of a long period of time, but not useful for find application specific problems (see above).
But I'm unsure how to accomplish this. Any suggestions?
Work out what metrics will help you find problems which impact the users of the application.
You may use JMX API for this purpose if want to get the data via program. Here is oracle tutorial on this topic.
If you just want to monitor the process, there are tools like VisualVM.
VisualVM is a nice tool to monitor memory utilization and other things VisualVM
i am trying to analyze the CPU usage for a Java UI application running on Windows. I connected it to VisualVM, but it looks like the highest percentage for CPU usage is being used by
sum.rmi.transport.tcp.TCPTransport$ConnectionHandler.run();
I believe this is being used to supply information to VisualVM and hence VisualVM is skewing the results that i'm trying to investigate. Does any one have a way to get a better indication of what is occurring or a better method to determine what in a running java application is taking up so much CPU.
Try to use sampler first.
For detailed information use the profiler and set root methods. See Profiling With VisualVM, Part 1 and Profiling With VisualVM, Part 2 for more information about CPU and Memory profiling.
That sounds awfully suspicious. Try cross referencing the data with results from hprof. You won't need any external applications running, and the data will simply be dumped to a text file from your own process. Are you connecting to your process remotely?
I am running a Java process with Xmx2000m, the host OS is linux centos, jdk 1.6 update 22. Lately I have been experiencing a weird behavior in the process, it becomes totally unresponsive with no apparent reason, no logs, no errors, nothing.. I am using jconsole to monitor the processor, heap and Perm memory are not full, threads and loaded classes are not leaking..
Explanation anyone?
I doubt anyone can give you an explanation since there are lots of possible reasons and not nearly enough information. However, I suggest that you jstack the process once it's hung to figure out what the threads are doing, and take it from there. It sounds like a deadlock or thrashing of some sort.
Do a thread dump. If you have access to the foreground process on Linux, use ctrl-\. Or use jstack to dump stack remotely. Or you can actually poke it through JMX via jconsole at MBeans/java.lang/Threading/Operations/dumpAllThreads.
Without knowing more about your app, it's hard to speculate about the cause. Presumably your threads are either a) blocked or b) exited. If they are blocked, they could be waiting for I/O on a database or other operation OR they could be waiting on a lock or monitor (deadlocked). If a deadlock exists, the thread dump will tell you which threads are deadlocked, which lock, and (in Java 6) annotate the stack with where locks have been taken. You can also search for deadlocks with the JMX method, available through jconsole at MBeans/java.lang/Threading/Operations/find[Monitor]DeadlockedThreads().
Or your threads may have received unhandled exceptions and exited. Check out Thread's uncaughtExceptionHandlers or (better) use Executors in java.util.concurrent.
And finally, the other classic source of pauses in Java is GC. Run with -verbose:gc and other GC flags to see if it's doing a full GC collection. You can also turn this on dynamically in jconsole by flipping the flag at MBeans/java.lang/Memory/Attributes/Verbose.
Agree with aix, but would like to add a couple of recommendataions.
1. check your system. Run top to see whether the system itself is healthy, CPU is not 100% and memory is available. If not, fix this.
2. application may freeze as a result of dead lock. Check this.
Ok here are some updates I wanted to share:
There is an incompatability between NTPL (Linux’s new thread library) and the Java 1.6+ JVM. A random bug causes the JVM to hang and eat up 100% CPU.
To work around it set LD_ASSUME_KERNEL=2.4.1 before running the JVM, export LD_ASSUME_KERMEL=2.4.1 . This disables NTPL: problem solved!
But for compatibility reasons, I'm still looking for a solution that uses NTPL.
Threads can be traced using jvisualvm and jconsole, and deadlocks can be avoided too. Note that there are several network services each with separate thread pools, and they all become unreachable.
Check the jvisualvm of the process right before the crash.
http://www.jadyounan.com/wp-content/uploads/2010/12/process.png
Could you elaborate more on what you are doing ? 2000 for memory is rather a lot.
Say your application is unresponsive and you cannot attach a debugger to it, as it rejects everything. All you have is a Linux Bash and process id. How would you investigate the issue? What tools would you use? My goal is to better my troubleshooting skills using Java.
This particular issue we had in production, on customer site.
You could take a thread dump from the application by issuing:
kill -3
That would give you some information as to the current state of the threads and hopefully help diagnose the issue. However, the trick is not in taking the thread dump, but reading the thread dump produced - since they can be a little overwhelming to look at. See this link for more info on reading a thread dump.
http://manikandakumar.blogspot.com/2006/12/reading-thread-dumps.html
You could also take a look at jstack which is part of the JDK - I've not used it specifically, see:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html
I agree with Jon that you should use kill -3 to get a thread dump. I have found Thread Dump Analyzer useful for viewing thread dumps.
You should also take a look at the memory usage of the process using top. Does it look like the app has run out of heap space? If so, you could try and use the jmap tool to obtain a heap dump and/or histogram count of the objects on the heap. You may need to use the -F option if the app has really hung up and I have experienced cases where jmap simply would not work against a hung Java process. Once you have a heap dump you could use Eclipse Memory Analyzer to investigate it.
You don't mention whether your application has any logging. If not you should look into adding logging that could help debug production issues.
jstack <pid>
Sounds like an interview question.
You could also try attach jconsole to see what it is doing.
If you have Java 6 you can try to connect with Visualvm (https://visualvm.dev.java.net/) which ships with current JDKs to connect to the VM. With this Tool you are able to create a complete MemoryDump (not only thread dump) of your VM Process. You can load this Memory Dump into VisualVM or Eclipse with the MAT Plugin (Memory Analyzer Tools http://www.eclipse.org/mat/).
After some time of loading an computation you can browse the complete Heap of your Application, search form Memory Leaks etc.
Analysing Heap Dumps is a great way to improve your TroubleShooting Skills.
I agree with others that Thread dumps are the way to go.
I would like to add that you should get lot's of thread dumps.
You can do very simple profiling with just a few unix commands.
Check my post here
I know this is an old question but I would like to share information with our other friends who are facing this issue and come across this post.
You can capture the thread dump and use some tools like fastThread, Samurai to analyze your thread dumps.
You can check out the following blog to see 8 different options to take thread dump: How to capture thread dump?