In Java profiling, it seems like all (free) roads nowadays lead to the VisualVM profiler included with JDK6. It looks like a fine program, and everyone touts how you can "attach it to a running process" as a major feature. The problem is, that seems to be the only way to use it on a local process. I want to be able to start my program in the profiler, and track its entire execution.
I have tried using the -Xrunjdwp option described in how to profile application startup with visualvm, but between the two transport methods (shared memory and server), neither is useful for me. VisualVM doesn't seem to have any integration with the former, and VisualVM refuses to connect to localhost or 127.0.0.1, so the latter is no good either. I also tried inserting a simple read of System.in into my program to insert a pause in execution, but in that case VisualVM blocks until the read completes, and doesn't allow you to start profiling until after execution is under way. I have also tried looking into the Eclipse plugin but the website is full of dead links and the launcher just crashes with a NullPointerException when I try to use it (this may no longer be accurate).
Coming from C, this doesn't seem like a particularly difficult task to me. Am I just missing something or is this really an impossible request? I'm open to any kinds of suggestions, including using a different (also free) profiler, and I'm not averse to the command line.
Consider using HPROF and opening the data file with a tool like HPjmeter - or just reading the resulting text file in your favorite editor.
Command used: javac -J-agentlib:hprof=heap=sites Hello.java
SITES BEGIN (ordered by live bytes) Fri Oct 22 11:52:24 2004
percent live alloc'ed stack class rank self accum bytes objs bytes objs trace name
1 44.73% 44.73% 1161280 14516 1161280 14516 302032 java.util.zip.ZipEntry
2 8.95% 53.67% 232256 14516 232256 14516 302033 com.sun.tools.javac.util.List
3 5.06% 58.74% 131504 2 131504 2 301029 com.sun.tools.javac.util.Name[]
4 5.05% 63.79% 131088 1 131088 1 301030 byte[]
5 5.05% 68.84% 131072 1 131072 1 301710 byte[]
HPROF is capable of presenting CPU usage, heap allocation statistics,
and monitor contention profiles. In addition, it can also report
complete heap dumps and states of all the monitors and threads in the
Java virtual machine.
The best way to solve this problem without modifying your application, is to not use VisualVM at all. As far as other free options are concerned, you could use either Eclipse TPTP or the Netbeans profiler, or whatever comes with your IDE.
If you can modify your application, to suspend it's state while you setup the profiler in VisualVM, it is quite possible to do so, using the VisualVM Eclipse plugin. I'm not sure why you are getting the NullPointerException, since it appears to work on my workstation. You'll need to configure the plugin by providing the path to the jvisualvm binary and the path of the JDK; this is done by visiting the VisualVM configuration dialog at Windows -> Preferences -> Run/Debug - > Launching -> VisualVM Configuration (as shown in the below screenshot).
You'll also need to configure your application to start with the VisualVM launcher, instead of the default JDT launcher.
All application launches from Eclipse, will now result in VisualVM tracking the new local JVM automatically, provided that VisualVM is already running. If you do not have VisualVM running, then the plugin will launch VisualVM, but it will also continue running the application.
Inferring from the previous sentence, it is evident that having the application halt in the main() method before performing any processing is quite useful. But, that is not the main reason for suspending the application. Apparently, VisualVM or its Eclipse plugin does not allow for automatically starting the CPU or memory profilers. This would mean that these profilers would have to be started manually, thereby necessitating the need to suspend the application.
Additionally, it is worth noting that adding the flags: -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y to the JVM startup will not help you in the case of VisualVM, to suspend the application and setup up the profilers. The flags are meant to help you in the case of profilers that can actually connect to the open port of the JVM, using the JDWP protocol. VisualVM does not use this protocol and therefore you would have to connect to the application using JDB or a remote debugger; but that would not resolve the problem associated with profiler configuration, as VisualVM (at least as of Java 6 update 26) does not allow you to configure the profilers on a suspended process as it simply does not display the Profiler tab.
This is now possible with the startup profiler plugin to VisualVM.
The advice with -Xrunjdwp is incorrect. It just enables debugger and with suspend=y it waits for debugger to attach. Since VisualVM is not debugger, it does not help you. However inserting System.in or Thread.sleep() will pause the startup and allows VisualVM to attach to your application. Be sure to read Profiling with VisualVM 1 and Profiling with VisualVM 2 to better understand profiler settings. Note also that instead of profiling, you can use 'Sampler' tab in VisualVM, which is more suitable for profiling entire java program execution. As other mentioned you can also use NetBeans Profiler, which directly support profiling of the application startup.
Related
I assume the question is strongly related to this one but it is closed and the solution does not seem clear to me.
I am trying to use VisualVM to profile a console Scala application of mine. I do this by starting VisualVM, starting my Scala application, opening its node in VisualVM, going to the "Profiler" tab and clicking "CPU". The button goes grey for some time (while status never changes to anything from "profiling inactive"), then back active but no profiling data appears shown.
The application does some file reading, data processing (which takes about a minute), stdout data output and exits. I've also tried adding Thread.sleep(60000) to the beginning and to the end of the program to ensure enough time for VisualVM to capture it and do the job, nothing changed though.
Unlike to the related question I've linked to I don't launch my app from Eclipse or anything else - I use the following command line to launch it:
java -classpath myapp.jar:lib/* MyApp.Main
all the libraries (including scala-library.jar) are put in lib/. The app works as expected.
Updates:
I have tried YourKit Java Profiler 11.0.2 and it fails with AttachNotSupportedException.
I've managed with YourKit by attaching the agent manually.
It seems worth stressing that I neither run the application from an IDE (or a build tool) nor have I modified any JVM options but classpath. As far as I understand this problem appears to depend on a JVM version and user/permission issues and. The objective is to figure out the actual mechanism of the problem and the way to configure all the things to work.
Versions of the software used:
YourKit Java Profiler 11.0.2
VisualVM 1.3.4
SBT 0.12.3 (to build but not to run (I've tried both actually))
Scala 2.9.2
Oracle Java 1.7.0_04-b20
XUbuntu Linux 12.04 32-bit i386
Linux kernel 3.2.0-24-generic-pae
I have used VisualVM to profile a Scala application two days ago, so I believe it should work. Here are my suggestions:
Take out YourKit out of the equation and run things without it for a starter
Take out scala out of the equation, try profiling a Java hello world that does while (true) { Thread.sleep(10000); }
Get that to work then put back Scala, then put back YourKit.
Edit: based on your input, I suggest to explicitly set the remote JMX connection. You can start java with the following options:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=20000
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then in VisualVM, add a JMX connection to localhost:20000.
I have a java application that its memory starting to jump and fall after few days.
Is there a tool that can show me the variables/members sizes during run/debug in real time?
Debugging it with eclipce is impossible.
Check out jhat and jmap.
In the longer run, consider adding a monitor to your app (with JMX, or write your own). It may help you in many situations in future.
Install this: http://visualvm.java.net/eclipse-launcher.html and then launch using Eclipse. This does not use the debugger, but launches the VisualVM application which lets you monitor the app. You'll need to go into the Run Configurations.. settings to set this up, and select VisualVM launcher at the bottom (Select Other... -> VisualVM launcher). You'll need to go into Preferences to set up the location of VisualVM too.
(On later Eclipse versions, drop it into the drops folder, unpacked).
If you're using >JDK6 and can get access to the machine then jvisualvm.exe may work, part of the JDK (in the bin dir)
yourkit (http://www.yourkit.com/) can do that, but expect a HUGE degradation on the performance of the app, as it has to track the size of each object.
I think it's better if you run yourkit with a few less invasive options, and then you can take snapshots of the memory usage.
I forgot to add, that with yourkit you can run the application without almost any instrumentation (which will slightly degrade the performance of the app) and when the problems start ocurring, you can enable the more heavy weight profiling while the app is running.
I have a Tomcat running as a Windows Service, and those are known not to work well with jstack. jconsole is working well, on the other hand, and I can see stacks of individual threads (I'm connecting to "localhost:port" to access it).
How can I use jconsole or a similar tool to dump all the thread stacks into a file? (similar to jstack)
You can use the ThreadMXBean management interface.
This FullThreadDump class demonstrates the capability to get a full thread dump and also detect deadlock remotely using JMX.
Nowadays you can use jvisualvm tool to connect to your remote JVM through JMX and create a thread dump. Don't know if this was available
Here's another code sample that will write a stack dump to a file:
http://pastebin.com/zwcKC0hz
We use this over JMX to give us an approximation of the stack dump you get when you make a JMX request or if the process detects high, unexpected load.
It would be helpful if you take a flight recording to get a deeper view on the JVM behavior, specially focusing on the Hot Methods.
Usually, a recording of half an hour is enough. To trigger a recording, you must be logged in to the machines, and issue the following command:
If using Java HotSpot 1.8.x:
$JAVA_HOME/bin/jcmd VM.unlock_commercial_features
$JAVA_HOME/bin/jcmd JFR.start duration=1800s settings=profile filename=/tmp/recording.jfr
IF using java HotSpot 1.7.x:
Edit your $HOME/conf/wrapper.conf file by adding the following parameters on JVM startup:
wrapper.java.additiona.=-XX:+UnlockCommercialFeatures
wrapper.java.additional.=-XX:+FlightRecorder
(replace with the corresponding positional number )
Then, have your instances restarted. Once done, issue the following command :
$JAVA_HOME/bin/jcmd JFR.start duration=1800s settings=profile filename=/tmp/recording.jfr
The flight recording wil produce a file on /tmp/recording.jfr upon termination.
I have a java process running on a Linux box, but it cannot be started/re-started in debug mode. I read about the jsadebugd command and started the daemon, but I want to connect to it from an IDE(possibly Eclipse) debugger or maybe YourKit or jconsole. How can I do this? The remote JVM is 1.6+.
I assume since you mentioned Yourkit and other tool that what you really want to do is look at object state inside your applications. There are a couple of options, though I don't think it gets you the ability to set break-points like Eclipse or another debugger would (though I'd be intersted in hearing what restricts you from starting the process in debug mode - technical? process?)
have you tried connecting with: VisualVM? I don't believe you need to start in debug mode, and it will give you the ability to navigate the object graph, and inspect objects. You can even use it to take heapdumps and do some ad-hoc analysis through them using OQL queries.
If you're running on JDK6+ - have you tried instrumenting with btrace? Some notes from the sailfin team sounded promising, and like DTrace in Solaris, this seem like it would be useful in a variety of situations.
I am not sure if I understand your restrictions correctly but you can start JVM with debugging server (serving JDWP) enabled. See for example "Sun VM Invocation Options" in http://java.sun.com/j2se/1.4.2/docs/guide/jpda/conninv.html
Then you can connect your Eclipse debugger to running JVM. See "Remote debugging" section here http://www.ibm.com/developerworks/library/os-ecbug/
I'm not sure that this is exposed by Eclipse, but here's how to do it with jdb:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/tooldescr.html#gbmog
I've been running Tomcat 5.5 with Java 1.4 for a while now with a huge webapp. Most of the time it runs fine, but sometimes it will just hang, with no exception generated, and no apparant way of getting it to run again other than re-starting Tomcat. The tomcat instance is allowed a gigabyte of memory on the heap, but rarely exceeds 300 MB. Has anyone else run into this issue, and is there a solution for it?
For clarification: I determined how much memory it is using via Task Manager and via Eclipse (I've also tried running it outside of Eclipse, but get the same problem eventually, though it takes a little longer). With Eclipse, I look at the memory allocated via its little (optional) memory pane and the amount allocated to javaw.exe via the task manager. I use the sysdeo? tomcat plugin for Eclipse.
For any jvm process, force a thread dump. In windows, this can be done with CTRL-BREAK, I believe, in the console window.
In *nix, it is almost always "kill -3 jvm-pid".
This may show if you have threads waiting on db connection pool/thread pool, etc.
Another thing to check out is how many connections you have currently to the JVM -- either use NETSTAT or SysInternals utility such as tcpconn/tcpview (google it).
Also, try to run with the verbose:gc JVM flag. For Sun's JVM, run like "java -verbose:gc". This will show your garbage collections. If it is collecting a lot (FULL COLLECTIONS, expecially) then you probably have a memory leak. The full collections are costly, especially on large heaps like that.
How are you determining that only 300mb are being used?
It sounds like you're hitting a deadlock.
If you can reproduce it in a dev environment then try attaching a debugger once it's happened. Take a look at your threads and see if you have any deadlocks.
If you can't get a debugger to attach you should be able to generate a thread dump, as Dustin pointed out.
Try increasing the logging sensitivity for the Tomcat application server.
http://tomcat.apache.org/tomcat-5.5-doc/logging.html
You can increase the sensitivity to FINEST or ALL for most of them for a few days and see if that helps you catch anything.
I agree with creating multiple thread dumps and viewing them though this: Thread Dump Analyzer