I just installed Java 1.6_07 so I could try profiling with VisualVM. It tells me that my app is spending 60% of its time in sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
How do I find out what it was doing during that time? How much of the time was it waiting for something to call it, or doing something else? What was calling it and what was it calling? I just can't seem to find any way to drill down to deeper levels like there is in Quantify or the Perl profiler.
I don't have experience with VisualVM -- but JRockit's profiler does provide this information; you may consider using it instead.
Update: a question with a list of java profilers can be found here, for users with sufficient rep to view deleted questions.
Does your App use RMI over TCP? If not, is it possible that this is a heisenbug, caused by instrumenting the VM? I assume VisualVM must use RMI calls to figure out what's going on in the JVM....
I have started using the new VisualVM 1.2. It allows profiling CPU and drilling down using a call graph. Try it out.
Using 1.3.2 also seeing this being the reported hangup I am hitting. In 1.3.2 if you do a thread dump and look for this call you can see where it lands in the call chain for that thread. Not sure if Yuval F was referring to this or something else. Look up the call chain to see what it's calling and so on, look down to see what it's being called by and so on.
Related
I am using eclipse to write java code. If I'm debugging some code I can set a breakpoint and follow along as the code goes through each of the functions or I can backtrack. I can also look at the call hierarchy or the references to get an idea. But that's not enough.
I would like to have a some sort of time-based visualization of what each thread is doing along the process from ... let's say "point A" (pressing a button on the interface) to "point B" (getting the result). I want to see which classes/methods were called in what order. I want a good way to visualize what kind of output is coming from one method and going into another method which fires off a new process ...etc.
Is a profiler the only thing available for this type of visualization? Basically I want an action diagram or flow diagram created. Is there some plugin or app which can generate something like this?
Edit: Here is an example of what I'm thinking ... at least visually:
essmodel.sourceforge.net/index.html
It has some flow of where the code is leading. But I think this is just a static map of what classes lead to other classes and what inputs/output options are available. I would want to map the flow based on a specific case.
JProfiler offers such a view, it's called the "Call tracer":
It's important to restrict your filters very carefully in order not to record to much data.
Disclaimer: My company develops JProfiler.
I believe using a profiler is going to be your best option. Are you familiar with VisualVM? It comes with the JDK (look for "jvisualvm.exe" inside your JDK's bin directory) and is capable of profiling local virtual machines automatically as well as remote machines when configured properly. And it does give a pretty slick overview of what threads are running and the code they are spending time in, so I think you could easily do what you need from it. And best of all, it's free :)
As I said, local profiling is a breeze. You just run JVisualVM.exe standalone, and it will find any and all java processes running on the local machine automatically (you can just pick them out of a menu that VisualVM gives you upfront). If you want to profile remotely, set the following VM arguments for whatever it is that you're running:
-Dcom.sun.management.jmxremote.port=[0-65535]
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then within VisualVM, use the hostname of the machine your remote JVM is running on and the port you configured in the first VM argument above.
I am looking for any tool that allows me to see how objects are created on heap in run time.
I was using VisualVM - Profiles but was not able to find when a variable of specific type (the one I am looking for) is being created. Maybe I do something wrong...
I will be also thankful getting any hint how to get such information using any API.
Regards,
Marcin
Typically, profilers (such as JProfiler) will allow you to see this - see for example the Allocation recording explained screencast.
However, they achieve this by attaching an agent to the JVM that allows them to intercept the low-level operations - this information is not usually available to either users or Java programs. As such, you won't be able to see the heap via JMX apps such as JConsole or JVisualVM.
Inside VisualVM Profiler, select the Settings and specify the class you want to profile. May be you also need to look on the option which record allocation stacks.
It sounds like you are trying to debug a program and that using the debugger would be the best option. You should be able to add a conditional breakpoint to stop the program when a variable is assigned the value you are looking for. This will allow you to see all the values at that time and the call stack to see what was called to create it.
We have an Java ERP type of application. Communication between server an client is via RMI. In peak hours there can be up to 250 users logged in and about 20 of them are working at the same time. This means that about 20 threads are live at any given time in peak hours.
The server can run for hours without any problems, but all of a sudden response times get higher and higher. Response times can be in minutes.
We are running on Windows 2008 R2 with Sun's JDK 1.6.0_16. We have been using perfmon and Process Explorer to see what is going on. The only thing that we find odd is that when server starts to work slow, the number of handles java.exe process has opened is around 3500. I'm not saying that this is the acual problem.
I'm just curious if there are some guidelines I should follow to be able to pinpoint the problem. What tools should I use? ....
Can you access to the log configuration of this application.
If you can, you should change the log level to "DEBUG". Tracing the DEBUG logs of a request could give you a usefull information about the contention point.
If you can't, profiler tools are can help you :
VisualVM (Free, and good product)
Eclipse TPTP (Free, but more complicated than VisualVM)
JProbe (not Free but very powerful. It is my favorite Java profiler, but it is expensive)
If the application has been developped with JMX control points, you can plug a JMX viewer to get informations...
If you want to stress the application to trigger the problem (if you want to verify whether it is a charge problem), you can use stress tools like JMeter
Sounds like the garbage collection cannot keep up and starts "halt-the-world" collecting for some reason.
Attach with jvisualvm in the JDK when starting and have a look at the collected data when the performance drops.
The problem you'r describing is quite typical but general as well. Causes can range from memory leaks, resource contention etcetera to bad GC policies and heap/PermGen-space allocation. To point out exact problems with your application, you need to profile it (I am aware of tools like Yourkit and JProfiler). If you profile your application wisely, only some application cycles would reveal the problems otherwise profiling isn't very easy itself.
In a similar situation, I have coded a simple profiling code myself. Basically I used a ThreadLocal that has a "StopWatch" (based on a LinkedHashMap) in it, and I then insert code like this into various points of the application: watch.time("OperationX");
then after the thread finishes a task, I'd call watch.logTime(); and the class would write a log that looks like this: [DEBUG] StopWatch time:Stuff=0, AnotherEvent=102, OperationX=150
After this I wrote a simple parser that generates CSV out from this log (per code path). The best thing you can do is to create a histogram (can be easily done using excel). Averages, medium and even mode can fool you.. I highly recommend to create a histogram.
Together with this histogram, you can create line graphs using average/medium/mode (which ever represents data best, you can determine this from the histogram).
This way, you can be 100% sure exactly what operation is taking time. If you can't determine the culprit, binary search is your friend (fine grain the events).
Might sound really primitive, but works. Also, if you make a library out of it, you can use it in any project. It's also cool because you can easily turn it on in production as well..
Aside from the GC that others have mentioned, Try taking thread dumps every 5-10 seconds for about 30 seconds during your slow down. There could be a case where DB calls, Web Service, or some other dependency becomes slow. If you take a look at the tread dumps you will be able to see threads which don't appear to move, and you could narrow your culprit that way.
From the GC stand point, do you monitor your CPU usage during these times? If the GC is running frequently you will see a jump in your overall CPU usage.
If only this was a Solaris box, prstat would be your friend.
For acute issues like this a quick jstack <pid> should quickly point out the problem area. Probably no need to get all fancy on it.
If I had to guess, I'd say Hotspot jumped in and tightly optimised some badly written code. Netbeans grinds to a halt where it uses a WeakHashMap with newly created objects to cache file data. When optimised, the entries can be removed from the map straight after being added. Obviously, if the cache is being relied upon, much file activity follows. You probably wont see the drive light up, because it'll all be cached by the OS.
I'm not experienced with java applications but I found out that finding static pointers etc. to these applications' memory addresses is often (nearly) impossible, apparently because of the java engine that handles the code (correct me if this way of naming it is wrong please).
Now, I've used VisualVM (https://visualvm.dev.java.net/) and it's great. I can select my java process and create a heap dump. It then shows me all classes and their values.
Can I use this method to continousely poll the heap dump and receive object values, for example the X Y and Z of a game? How would I programmatically interact with such application, and if this should not be done with VisualVM, what would be an alternative?
Edit: this is what I need to do:
I need to be able to find all classes with properties that have a certain value. For example: I'd search for the X coordinate (a float) and it should return the class "PlayerCoordsHandler" (just an example) and the corresponding float with it's value... or alternatively just a way to find this same float again (after restarting for example). This process does not have to be programmatic, aslong as requesting the value of the now known property (x float) can be retrieved programmatically (for example with a command line utility or reading from a file).
Edit2:
The target application is a windows executable (but made with java) and launches it's own java VM. It's not possible to add java parameters for debugging. This does not seem to be required though, as VirtualVM is able to debug the process just fine. Anyone knows how?
Thanks in advance.
It looks like you want to debug running Java applications.
The "official" Java debugger is JDB. I believe it's part of the JDK. It has the ability to set breakpoints, examine heaps, list and display and even change variables, show running threads and so on. The usual debugger stuff. But it's command line, which makes it a pain in the neck to work with.
Instead, it makes a lot of sense to use an IDE with integrated debugger. I use Eclipse. You can do all the usual debuggery things, including displaying windows with variables. You can set conditional breakpoints and there's much more. Specifically in answer to your question, you can set up watch expressions, which will be evaluated during the program's execution and their displays refreshed with new values when they change.
You may not want to run your Java app inside the IDE; or it may be running in a Web application server. That's no problem for JDB or Eclipse (or other IDEs, like NetBeans or IntelliJ Idea): They can connect to a running JVM and debug remotely with the same level of convenience.
A program being debugged like this, remotely or otherwise, run somewhat more slowly than if it were not. Your game, while being debugged, will run at rather bad-looking FPS; but it should still respond more or less normally to gameplay interaction.
Remote debugging:
To be able to attach your EclipseNetBeans debugger to a running Java process you need to start that process with the following Java options…
-Xdebug -Xrunjdwp:transport=dt_socket,address=3704,server=y,suspend=n
Have a look at YourKit. You can monitor CPU, memory and threads live, and generate dumps whenever you want. It can even compare different memory dumps to show you which objects were added/removed.
It's not free though, it has a 15 day (or 30 day?) fully functional eval period. If free is not a real concern it's definitely a great tool.
I good starting point is the jps and jstat tools added in Java 6 (i think). jps gives you the pid and main class for each application. jstat give you more details about process
Triggering a heapdump is usefull for post-mortem analysis of say memory leaks, but as the Java garbage collector moves objects around, you cannot use the memory values of a heapdump to reliably access those objects.
If you need a way to query internal values from outside of the application you could look into setting up an RMI service API via which you can retrieve the values you need.
Another method (if you just need to test something) could be to connect to the process via de Java debugging API.
If you know the JRE location that is used, you could rename java.exe and write a (C/C++) wrapper that adds the debug options listed by Carl and calls the renamed_java.exe in turn.
Another posibility might be to add or update classes in the .jar file of the application. You do not need the source to do this.
Tom, are you trying to reverse engineer an application that specifically tries to obfuscate its working? If so you might get further if you contact the manufacturer and ask them what possibilities they see for what you try to achieve?
You can easily generate a heap dump by creating your own JMX connection to the JVM, just like VisualVM does it. Analyzing the heapdump is very possible (the data is there and totally disconnected from the JVM so there is no interference from the gc).
However, unless it is a very specific scenario you are looking for you are probably much better off giving the heapdump to MAT and find a good workflow in there to use.
Edit: In this particular case it is probably better to create some kind of specific API to access the values from the outside (and maybe publish the values as MBeans using JMX). Taking a heap dump is way to much work if all you want to do is monitoring a few values.
Edit2: Based on your edits, it seems to me like you could really benefit from publishing your own MBean over JMX. I have to run for a meeting but, unless someone else does it while I am away, I will try to remember to give you some pointers later. Either in an edit of this one or in a new post.
If you want to poll the values of specific objects while your Java application is running you would probably find that using JMX is a better and more efficient approach rather than using a heap dump. With JMX you can define what values should be exposed and use tools such as VisualVM or JConsole to view them at runtime.
With VisualVM and heapdump you can find all classes with certain property by OQL:
var out = "";
var cls = filter(heap.classes(), "/java./(it.name)")
while (cls.hasNext()) {
var cl = cls.next();
var fls = cl.fields;
while (fls.hasMoreElements()) {
var fl = fls.nextElement();
if (/size/(fl.name)) {
out = toHtml(cl) + "." + fl.name + "()\n";
}
}
}
out.toString()
and write custom logging for BTrace
It is alternative for debugging.
FusionReactor could be a good alternative. For example;
VisualVM doesn’t give you a lot of insides on application memory
except for the total Heap allocation. Heap is a good metric to start
with, but I feel this is not enough to troubleshoot the actual cause
of a memory-related issue.
FusionReactor will display all of the memory spaces it detects, which
depends on the version of Java you’re running:
Heap allocation Non-Heap allocation CodeHeap (profiled and
non-profiled methods) Compressed Class Space FusionReactor also shows
the amount of memory that each generation takes Eden Space Old Space
Survivor Space
https://www.fusion-reactor.com/blog/java-visualvm-alternatives/
I like to generate a thread dump programmatically. I've learned that there a basically two ways to do it:
Use the "Java Virtual Machine Tool Interface" JVM-TI
Use the higher abstracted "Java Debugger Interface" JDI
For the JVM-TI I was able to find some useful information, but I would have to write a JNI-DLL which, at least for the moment, I would like to avoid. With the JDI I can use Java and it seems I'm able to use it from within the application. But I wasn't able to find some kind of tutorial or HOWTO for it. The only documentation I could find, were the Java-Docs http://java.sun.com/j2se/1.5.0/docs/guide/jpda/jdi/ which isn't very helpful, because it doesn't show me how to use this classes.
So, does anybody know of a good tutorial/book I could read?
Thx for any help!
There is a third way: Thread.getAllStackTraces()
http://java.sun.com/javase/6/docs/api/java/lang/Thread.html#getAllStackTraces()
This is much easier than the debugger interface...
You can get just about all the Thread info you need including deadlocks from http://java.sun.com/javase/6/docs/api/java/lang/management/ThreadMXBean.html
Thread.getAllStackTraces() dumps only the execution trace of all the threads, but doesn't give the information of object locks that have been obtained by a particular thread or the lock on which a particular thread has been waiting. Basically, we'll not be able to nail down deadlocks with this.
Did you consider the remote alternative ? I.e. VisualVM
jps and jstack are also useful tools included in JDK 5, providing a quick command line method for obtaining stack traces of all current threads.
This article suggest JDI is also used as a remote tool.
So I am not sure you can triggers a thread dump within your own program, instead you find a way to send to yourself a SIGQUIT signal (kill -3) on Unix platforms, or press the Ctrl-\ key on Unix or Ctrl-Break on Windows platforms.
Plus, JDI wasn't intended to be used to debug the same process in which the JDI client is running. Still this thread I just linked to is the closest I have found to actually use JDI within the same program.