I need to peek into the stack of 2 deadlocked threads to analyze the situation. The JVM is live right now and the data is there, but I need some kind of tool to extract it from the process. I only care about 6 variables in the stack of type String. Any ideas are greatly appreciated. JVM versions 6_35, it's a linux, JMX is enabled, but I dont have a profiler/debugger connection configured on it. It's very difficult to reproduce.
I found a little trick using a heap dump viewer (YourKit in this instance, but may be others work as well). Basically you enumerate all instances of the Thread class, then you find the thread you want by name and open it. The stack variables are marked as < local variable > like this:
Not all variables are here, but all that are passed as arguments to method are displayed. I wonder if the profilers can address this issue even better?
You can't do this easily. Normal jstack tool will only dump stack. Technically you can try dumping whole heap (using jmap) but looking for this particular variables can be a pain if possible.
Note that this is not easily doable for security reasons. Stack traces can contain credentials or other sensitive data.
You can send the process a SIGQUIT which will give you a dump and keep the VM running, on a Unix-like OS with the Sun/Oracle JVM, as will IBM's JVM -- not sure if the output will be suitable for your purposes, tough. Probably similar to jstack/jmap in the other answer.
Related
I have set up and using https://github.com/jvm-profiling-tools/async-profiler which is extremely useful but I have a strange thing I cannot explain.
My setup is exactly where multiple presentation showed it can help:
AKS kubernetes cluster with a nodepool
A pod deployed on one node
Within the container I have set up openjdk-11 with the debuginfo
The profiling setup is a simple ./profiler start -e malloc PID
Since I'm in a virtualised environment profiling works, the only warning I get is
[WARN] Kernel symbols are unavailable due to restrictions. Try
sysctl kernel.kptr_restrict=0
sysctl kernel.perf_event_paranoid=1
I think regarding malloc call capturing that is probably not needed.
And the question is that after some profiling time I have captured portions for allocation where the flame graph says: "unknown" for the stacktrace (see attached pic). May it be that I still do not have a full setup in place in the container or I would really need those sysctls in place?
Problem is that it is not trivial with virtualization to put them in place since as I understood that is practically affecting the underlying node we are running on.
Flame graphs for allocations
UPDATE
Now that I restarted the profiling after all main functionality fired at least once for my microservice seems that there are no unknown allocation. Stupid question, but can it happen that I started profiling immediately before all classloading happened (since beans are instantiated lazy) and this is why it was classified like that?
UPDATE 2
Actually my hypothesis is wrong I did one good dump
Short after that again the same phenomenon happened that reportedly huge amount of captured malloc event are unkonwn, top shows no dramatic increas. Can this be due to virtualization and I'm actually capturing event from other containers on the same node? In my container there are no more java processes and I'm also specifying directly the PID
UPDATE 3
So after Andrei provided me the "dwarf stackwalker" this looks much better. I only have one question which is still not clear but it is only me. We are profiling here malloc event with my:
./profile.sh start --cstack dwarf -e malloc PID
So what I see on these flame graphs: Is it only the captured event number which could be freed in the meantime or it currently held native memory by all those mallocs?
My current situation is that I see payara-micro healthcheck and autodeploy holds significant amount of memory which is weird and my first guess for the leak source.
I also made a jeprof output anybody has a guess what "updatewindow/inflate" can point to?
Container environment is not related here.
It seems like libc (where malloc implementation resides) on your system is compiled without frame pointers. So the standard stack walking mechanism in the kernel is unable to find a parent of malloc frame.
I've recently implemented an alternative stack walking algorithm that relies on DWARF unwinding information. New version has not been yet released, but you may try to build it from sources. Or, for your convenience, I prepared the new build here: async-profiler-2.6-dwarf-linux-x64.tar.gz
Then add --cstack dwarf option, and all malloc stack traces should be in place.
I have to evaluate how difficult it would be to extract some object (e.g. java.security.PrivateKey) from memory of a running java program.
I'm not very into this low level memory stuff, so I started out with small C programs and familiarized myself with gdb, /proc/<pid>/maps, /proc/<pid>/mem and a script that dumps all the memory areas.
However, things change when switching to java. Memory is allocated and managed very differently with java thanks to garbage collection. In C programs I'd look at a stack address and know for certain that it contained the variable I wanted to extract.
So my questions are:
Do Java objects have some kind of type ID so I can locate objects of that type in a memory dump?
If so, how do I find out the ID of a type (e.g. what's the ID of a String)?
If there is no such type ID, what other possibilities would attackers have to extract, let's say, a java.security.PrivateKey from a java process?
Suppose that JMX is turned off.
Thanks for your help
This is even easier than you might think :)
HotSpot Serviceability Agent does the magic. It can open a core dump or attach to a live Java process using ptrace and then extract the layout of JVM structures and all Java objects. No cooperation from target JVM is needed. This works even when JMX and Attach Mechanism are disabled.
Here is an example how to inspect the instances of a given class in the remote JVM.
sa-jdi.jar must be in the classpath to work with Serviceability Agent.
Finally the easiest solution ever. Run
jmap -F -dump:format=b,file=heap.bin PID
Note -F argument - it forces jmap to use Serviceability Agent to make the heap dump.
P.S. Here are the sources of SA if you'd like to know how it works under the hood.
I was wondering. I have a simple Java web project with a servlet. When no users are using it (I host it local on GlassFish) I still see a sawtooth pattern in the memory usage of the process. I can't seem to understand what the JVM is doing? I understand the normal flow of the memory used by JVM. The heap is getting filled with objects the application is creating. At a certain point the heap reaches a point where the garbage collector comes in, which will remove all the 'cached' objects which no longer are used, so that new objects can be created and can be filled in the heap size.
What I wonder is, what is the JVM doing all the time when it's idle?
EDIT:
Like I said in the comments. I created a very simple Java application in Eclipse which showed a Swing window saying 'hello world'. When I watch the memory usage of the JVM in Java VisualVM, I see the same sawtooth pattern.
It turns out that VisualVM is the guilty one. This is because VisualVM constantly asks the JVM what it's statistics are, so the JVM creates objects to give the information to VisualVM.
Thanks to
Peter Lawrey
My guess is that the app server is doing some sort of behind-the-scenes book-keeping. Keep in mind that, even though no one is using it, there are still threads running in the background. Also, does your app use any type of chronological trigger framework, like Quartz? Or, what about logging. Even though it may not be actively sending log messages to a file, the Loggers are still getting called, which is causing all sorts of things to be created/destroyed.
I am looking for any tool that allows me to see how objects are created on heap in run time.
I was using VisualVM - Profiles but was not able to find when a variable of specific type (the one I am looking for) is being created. Maybe I do something wrong...
I will be also thankful getting any hint how to get such information using any API.
Regards,
Marcin
Typically, profilers (such as JProfiler) will allow you to see this - see for example the Allocation recording explained screencast.
However, they achieve this by attaching an agent to the JVM that allows them to intercept the low-level operations - this information is not usually available to either users or Java programs. As such, you won't be able to see the heap via JMX apps such as JConsole or JVisualVM.
Inside VisualVM Profiler, select the Settings and specify the class you want to profile. May be you also need to look on the option which record allocation stacks.
It sounds like you are trying to debug a program and that using the debugger would be the best option. You should be able to add a conditional breakpoint to stop the program when a variable is assigned the value you are looking for. This will allow you to see all the values at that time and the call stack to see what was called to create it.
I'm not experienced with java applications but I found out that finding static pointers etc. to these applications' memory addresses is often (nearly) impossible, apparently because of the java engine that handles the code (correct me if this way of naming it is wrong please).
Now, I've used VisualVM (https://visualvm.dev.java.net/) and it's great. I can select my java process and create a heap dump. It then shows me all classes and their values.
Can I use this method to continousely poll the heap dump and receive object values, for example the X Y and Z of a game? How would I programmatically interact with such application, and if this should not be done with VisualVM, what would be an alternative?
Edit: this is what I need to do:
I need to be able to find all classes with properties that have a certain value. For example: I'd search for the X coordinate (a float) and it should return the class "PlayerCoordsHandler" (just an example) and the corresponding float with it's value... or alternatively just a way to find this same float again (after restarting for example). This process does not have to be programmatic, aslong as requesting the value of the now known property (x float) can be retrieved programmatically (for example with a command line utility or reading from a file).
Edit2:
The target application is a windows executable (but made with java) and launches it's own java VM. It's not possible to add java parameters for debugging. This does not seem to be required though, as VirtualVM is able to debug the process just fine. Anyone knows how?
Thanks in advance.
It looks like you want to debug running Java applications.
The "official" Java debugger is JDB. I believe it's part of the JDK. It has the ability to set breakpoints, examine heaps, list and display and even change variables, show running threads and so on. The usual debugger stuff. But it's command line, which makes it a pain in the neck to work with.
Instead, it makes a lot of sense to use an IDE with integrated debugger. I use Eclipse. You can do all the usual debuggery things, including displaying windows with variables. You can set conditional breakpoints and there's much more. Specifically in answer to your question, you can set up watch expressions, which will be evaluated during the program's execution and their displays refreshed with new values when they change.
You may not want to run your Java app inside the IDE; or it may be running in a Web application server. That's no problem for JDB or Eclipse (or other IDEs, like NetBeans or IntelliJ Idea): They can connect to a running JVM and debug remotely with the same level of convenience.
A program being debugged like this, remotely or otherwise, run somewhat more slowly than if it were not. Your game, while being debugged, will run at rather bad-looking FPS; but it should still respond more or less normally to gameplay interaction.
Remote debugging:
To be able to attach your EclipseNetBeans debugger to a running Java process you need to start that process with the following Java options…
-Xdebug -Xrunjdwp:transport=dt_socket,address=3704,server=y,suspend=n
Have a look at YourKit. You can monitor CPU, memory and threads live, and generate dumps whenever you want. It can even compare different memory dumps to show you which objects were added/removed.
It's not free though, it has a 15 day (or 30 day?) fully functional eval period. If free is not a real concern it's definitely a great tool.
I good starting point is the jps and jstat tools added in Java 6 (i think). jps gives you the pid and main class for each application. jstat give you more details about process
Triggering a heapdump is usefull for post-mortem analysis of say memory leaks, but as the Java garbage collector moves objects around, you cannot use the memory values of a heapdump to reliably access those objects.
If you need a way to query internal values from outside of the application you could look into setting up an RMI service API via which you can retrieve the values you need.
Another method (if you just need to test something) could be to connect to the process via de Java debugging API.
If you know the JRE location that is used, you could rename java.exe and write a (C/C++) wrapper that adds the debug options listed by Carl and calls the renamed_java.exe in turn.
Another posibility might be to add or update classes in the .jar file of the application. You do not need the source to do this.
Tom, are you trying to reverse engineer an application that specifically tries to obfuscate its working? If so you might get further if you contact the manufacturer and ask them what possibilities they see for what you try to achieve?
You can easily generate a heap dump by creating your own JMX connection to the JVM, just like VisualVM does it. Analyzing the heapdump is very possible (the data is there and totally disconnected from the JVM so there is no interference from the gc).
However, unless it is a very specific scenario you are looking for you are probably much better off giving the heapdump to MAT and find a good workflow in there to use.
Edit: In this particular case it is probably better to create some kind of specific API to access the values from the outside (and maybe publish the values as MBeans using JMX). Taking a heap dump is way to much work if all you want to do is monitoring a few values.
Edit2: Based on your edits, it seems to me like you could really benefit from publishing your own MBean over JMX. I have to run for a meeting but, unless someone else does it while I am away, I will try to remember to give you some pointers later. Either in an edit of this one or in a new post.
If you want to poll the values of specific objects while your Java application is running you would probably find that using JMX is a better and more efficient approach rather than using a heap dump. With JMX you can define what values should be exposed and use tools such as VisualVM or JConsole to view them at runtime.
With VisualVM and heapdump you can find all classes with certain property by OQL:
var out = "";
var cls = filter(heap.classes(), "/java./(it.name)")
while (cls.hasNext()) {
var cl = cls.next();
var fls = cl.fields;
while (fls.hasMoreElements()) {
var fl = fls.nextElement();
if (/size/(fl.name)) {
out = toHtml(cl) + "." + fl.name + "()\n";
}
}
}
out.toString()
and write custom logging for BTrace
It is alternative for debugging.
FusionReactor could be a good alternative. For example;
VisualVM doesn’t give you a lot of insides on application memory
except for the total Heap allocation. Heap is a good metric to start
with, but I feel this is not enough to troubleshoot the actual cause
of a memory-related issue.
FusionReactor will display all of the memory spaces it detects, which
depends on the version of Java you’re running:
Heap allocation Non-Heap allocation CodeHeap (profiled and
non-profiled methods) Compressed Class Space FusionReactor also shows
the amount of memory that each generation takes Eden Space Old Space
Survivor Space
https://www.fusion-reactor.com/blog/java-visualvm-alternatives/