Its not problem specific but is it possible to get a copy of current memory state, as in just get whatever is there in the main memory. I mean is there any way we can get an image of RAM in java.
I am editing my question. So here is a screenshot of my Windows 7 Task Manager.
#peter I see that current memory usage is 3.27GB. So, can I get that whole thing in some read only memory and when I restart my OS, it resumes where I left off, as in whatever my last memory snapshot was.
Yes, it's called a heap dump.
jmap -heap {pid}
dumps the heap to a file.
You can use jvisualvm to analyse the heap dump.
It depends on what you mean by "main memory".
The JRE is designed
to insulate your Java program from the OS around it, and vice versa.
to insulate Java objects from the implementation details of other objects
With "ordinary" Java, you only get to see what classes and objects expose through public methods.
However, Java has its Reflection API, and with that, if your JRE is configured to allow it, you can break through these boundaries, and look deeper into the classes and objects within the JRE.
In Oracle HotSpot, you can start the JRE with the Java Servicability Agent - http://openjdk.java.net/groups/hotspot/docs/Serviceability.html - this gives you access through an API to much more detail of the Java heap. But you're still restricted to memory claimed by the JRE process, and allocated to its heap.
One further possibility is to write a native library using JNI. There are C API calls that allow you to browse the OS address space. You need to be root (or the equivalent on your OS) to see other people's address space). You could write C code, and JNI to call it from Java.
Related
I have to evaluate how difficult it would be to extract some object (e.g. java.security.PrivateKey) from memory of a running java program.
I'm not very into this low level memory stuff, so I started out with small C programs and familiarized myself with gdb, /proc/<pid>/maps, /proc/<pid>/mem and a script that dumps all the memory areas.
However, things change when switching to java. Memory is allocated and managed very differently with java thanks to garbage collection. In C programs I'd look at a stack address and know for certain that it contained the variable I wanted to extract.
So my questions are:
Do Java objects have some kind of type ID so I can locate objects of that type in a memory dump?
If so, how do I find out the ID of a type (e.g. what's the ID of a String)?
If there is no such type ID, what other possibilities would attackers have to extract, let's say, a java.security.PrivateKey from a java process?
Suppose that JMX is turned off.
Thanks for your help
This is even easier than you might think :)
HotSpot Serviceability Agent does the magic. It can open a core dump or attach to a live Java process using ptrace and then extract the layout of JVM structures and all Java objects. No cooperation from target JVM is needed. This works even when JMX and Attach Mechanism are disabled.
Here is an example how to inspect the instances of a given class in the remote JVM.
sa-jdi.jar must be in the classpath to work with Serviceability Agent.
Finally the easiest solution ever. Run
jmap -F -dump:format=b,file=heap.bin PID
Note -F argument - it forces jmap to use Serviceability Agent to make the heap dump.
P.S. Here are the sources of SA if you'd like to know how it works under the hood.
I want to see how much memory is allocated to my Java JVM on my Windows XP installation. I'm running an executable jar that has a main class that makes some JNI calls to a C library via a dll that is loaded using the System.loadLibrary("SampleJni"). Some calls are working and some are not. Whenever there are more than one String parameters passed I get a system dump. If I just have one String, one int, two ints..etc, no crashes. The machine only has .99 GB of ram, so I'm thinking the JVM can't allocate the need memory.
Use jconsole to check the memory used by your program. Jconsole comes with the JDK so you already have it. This memory won't include memory used by your JNI C code, but it will tell you what memory Java is using. Your more likely culprit is JNI mapping isn't correct when using multiple parameters.
I've run JVMs (Java 6) on machines with less memory than that. IIRC the default for the JVM on windows was 64Mb, but that may have changed. Even if it did, it should be enough to start up. You'd also see OutOfMemoryErrors if this were the case rather than hard crashes.
There are various methods in java.lang.Runtime that will let you inspect how much memory you have.
The likely cause is the JNI interface. Its very easy to crash the JVM if the JNI code isn't 100% correct.
I've inherited a legacy application that uses ProcessBuilder.start() to execute a script on a Solaris 10 server.
Unfortunately, this script call fails due to a memory issue, as documented here
Oracle's recommendation is to use posix_spawn() since, under the covers, ProcessBuilder.start() is using fork/exec.
I have been unable to find any examples (e.g., how to call "myScript.sh")
using posix_spawn() in Java, or even what are the packages that are required.
Could you please, point me to a simple example on how to use posix_spawn() in Java?
Recent version of Java 7 and 8 support posix_spawn internally.
command line option
-Djdk.lang.Process.launchMechanism=POSIX_SPAWN
or enable at runtime
System.setProperty("jdk.lang.Process.launchMechanism", "POSIX_SPAWN");
I'm a little confused as to which Java version/OS combinations have this enabled by default, but I'm sure you could test and find out pretty quickly whether setting this option makes a difference.
For reference, to go back to the old fork method simply use
-Djdk.lang.Process.launchMechanism=fork
To prove whether this option is respected in your JVM version use
-Djdk.lang.Process.launchMechanism=dummy
and you will get an error next time you exec. This way you know the JVM is receiving this option.
An alternative, which does not require JNI, is to create a separate "process spawner" application. I would probably have this application expose an RMI interface, and create a wrapper object that is a drop-in replacement for ProcessBuilder.
You might also want to consider having this "spawner" application be the thing that starts your legacy application.
You will need to familiarize yourself with JNI first. Learn how to call out into a native routine from Java code. Once you do - you can look at this example and see if it helps with your issue. Of particular interest to you is:
if( (RC=posix_spawn(&pid, spawnedArgs[0], NULL, NULL, spawnedArgs, NULL)) !=0 ){
printf("Error while executing posix_spawn(), Return code from posix_spawn()=%d",RC);
}
A much simpler solution would be to keep your code unchanged and simply add more virtual memory to your server.
i.e.:
mkfile 2g /somewhere/swap-1
swap -a /somewhere/swap-1
Edit: To clarify as the link present in the question is now broken:
the question is about a system out of virtual memory due to the JVM being forked. Eg, assuming the JVM uses 2 GB of VM, an extra 2 GB of VM is required for the fork to succeed on Solaris. There is no pagination involved here, just memory reservation. Unlike the Linux kernel which by default overcommits memory, Solaris makes sure allocated memory is backed by either RAM or swap. As there is not enough swap available, fork is failing. Enlarging the swap allows the fork to succeed without any performance impact. Just after the fork, the exec "unreserves" this 2GB of RAM and revert to a situation identical to the posix_spawn one.
See also this page for an explanation about memory allocation under Solaris and other OSes.
How can I determine the address in memory of the Java heap for a JVM running in the current process? That is, get a void* pointer or equivalent to the contiguous area of memory that the JVM has allocated for the heap, using Java, C, or other calls?
Matlab has a JVM embedded in its process. The memory the JVM allocates is unavailable for Matlab arrays, and of this, the heap is important, because it takes a big contiguous chunk of memory and never shrinks, and Matlab also needs contiguous memory for its arrays. If the heap is reallocated during expansion, that could cause fragmentation.
I'd like to instrument my process to examine the interaction between the Java heap and Matlab's view of memory, and to find out when it moves due to resizing, preferably all from within the process. This needs the address of the heap. It's easy to find the heap size from java.lang.Runtime, but not its address in memory. How can this be done?
I'm running Sun's JRE 1.6.0_04 in a Matlab R2008b process on Windows XP and Server 2003. I realize this probably needs to be a vendor-specific technique. The process runs code we've written, so we can use custom Java, Matlab, JNI, and C/C++ code. Java method calls or supported hooks in the JVM would be preferred to low-level hackery.
EDIT: The goal of this is to examine the interaction between the JVM's GC and Matlab's GC. I have no need to see into the Java heap and won't be reading anything from that memory; I just want to see where it is in the context of the overall virtual memory space that Matlab's GC is also trying to fit data into.
A quick 'n dirty way to get the actual heap address of the JVM is to jump into WinDbg, attaching to the JVM and issue a single !address command. Somewhere around 0x2??????? (It differes between jvm versions but remains static for that version) will be a large VAD marked PAGE_EXECUTE_READWRITE, this is your JVM's heap in the process's memory.
To confirm, you can set a breakpoint on kernel32!VirtualAlloc and upon JVM initilization in the module JVM.DLL you will hit on the call to VirtualAlloc showing you the jvm allocation its heap. If you check out the code around this call you can see how the address is calculated.
Stepping back a bit... Could you go with a fixed-size Java heap? At that point concerns about reallocation and fragmentation go away.
On a stand-alone Java invocation, that involve specifying something like -Xmx500m and -Xms500m for a 500Mb heap. You'd have to translate that into what matlab wants.
If you just want to get the JVM heap to shrink you could try playing with the gc parameters such as -XX:MaxHeapFreeRatio (see http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp)
I don't think you can get a pointer to the Java heap with JNI. However, the Java heap is just memory allocated to the process by Windows from one of the process heaps.
You can get at the process heaps from your C++ code using the GetProcessHeaps function (http://msdn.microsoft.com/en-us/library/aa366571(VS.85).aspx) and then start walking through them with the HeapWalk function. There's a good example at http://www.abstraction.net/content/articles/analyzing%20the%20heaps%20of%20a%20win32%20process.htm. You might be able to spot which allocated blocks are used by the Java heap by looking for certain patterns of bytes (the JVM source code might give you some clues as to what to look for, but good luck figuring that out!)
The GC can move data at any time to compact the memory. So there is no fixed point for objects in the Java Heap. You can use a ByteBuffer.allocateDirect() which allocates memory in the "C" space rather than the heap and it is fixed in memory.
I don't think that there's a way do to what you want, without using a customized JVM. You could theoretically use OpenJDK and patch it or enable some tracing (not sure whether there's one that fits your needs). I think n external monitoring tool such as the process explorer could solve your problem
I have a piece of an application that is written in C, it spawns a JVM and uses JNI to interact with a Java application. My memory footprint via Process Explorer gets upto 1GB and runs out of memory. Now as far as I know it should be able to get upto 2GB. One thing I believe is that the memory the JVM is using isn't visible in the Process Explorer. My xmx is set to 256, I added some statements to watch the java side memory and it is peaking at 256 and GC is doing its job and it is all good on that side. So my question is, where is the other 700+ MB being consumed? Anyone out there a Java/JNI/C Memory expert?
There could be a leak in the JNI code.
Remember to use (*jni)->DeleteLocalRef() for any object references you get once you are done with them. If you use any native C buffers to create new Java objects, make sure you free them off once the object is created. Check the JNI Specification for further guidelines.
Depending on the VM you are using you might be able to turn on JNI checking. For example, on the IBM JDK you can specify "-Xcheck:jni".
Try a test app in C that doesn't spawn the JVM but instead tries to allocate more and more memory. See whether the test app can reach the 2 GB barrier.
The C and JNI code can allocate memory as well (malloc/free/new/etc), which is outside of the VM's 256m. The xMX only restricts what the VM will allocate itself. Depending on what you're allocating in the C code, and what other things are loaded in memory you may or may not be able to get up to 2GB.
If you say that it's the Windows process that runs out of memory as opposed to the JVM, then my initial guess is that you probably invoke some (your own) native methods from the JVM and those native methods leak memory. So, I concur with #John Gardner here.
Well thanks to all of your help especially #alexander I have discovered that all the extra memory that isn't visible via Process Explorer is being used by the Java Heap. In fact via other tests that I have run the JVM's memory consumption is included in what I see from the Process Explorer. So the heap is taking large amounts of memory, I will have to do some more research about that and maybe ask a separate question.
Write a C test harness and use valgrind/alleyoop to check for leakage in your C code, and similarly use the java jvisualvm tool.