I just proposed an algorithm, and I want to prove its superiority compared with another algorithm in terms of time&space consumption. I will implement my algorithm in Java, does anyone know how to monitor the memory consumption of a Java program when the program is executing? Thanks.
from JDK 6 onwards you have a tool in the bin directory which monitors CPU, memory consumption as well as the number of threads spawned. its called visualvm. Just start it and then start your java process. You will see the java process in the tool on the left hand side. Double click on the process and view the statistics. Hope this helps. :)
While the tools mentioned in other answers will tell you how much memory Java allocated to your heap, this can only be taken as an upper bound on your program's memory consumption - the JVM does not garbage collect objects in the moment they're no longer needed, it can even increase heap size instead of garbage collecting and only perform garbage collection when it reaches the given heap limit (-Xmx).
So, the graphs you'll see may reach much higher than what the program really needs.
One way to deal with this is to lower the heap size limit (-Xmx) incrementally until the program crashes on OutOfMemory exception.
Another way can be used if you know at which point in your algorithm it will consume the most memory - place a breakpoint at that point, make a heap dump and then examine it to see how much living objects occupy.
You can use
long memory = runtime.totalMemory() - runtime.freeMemory();
This place also gives you runtime so that might be pretty useful for what you're trying to do as well!
http://www.vogella.com/tutorials/JavaPerformance/article.html#runtimeinfo
Answer is found with a quick Google search.
Use the totalMemory( ) and freeMemory( ) methods [...] The java.lang.Runtime.totalMemory()
from http://crunchify.com/java-runtime-get-free-used-and-total-memory-in-java/
so
long totalmem = java.lang.Runtime.totalMemory();
long freemem = java.lang.Runtime.freeMemory();
long consump = totalmem - freemem;
Related
I have troubles with Java memory consumption.
I'd like to say to Java something like this: "you have 8GB of memory, please use it, and only it. Only if you really can't put all your resources in this memory pool, then fail with OOM".
I know, there are default parameters like -Xmx - they limit only the heap. There are also plenty of other parameters, I know. The problems with these parameters are:
They aren't relevant. I don't want to limit the heap size to 6GB (and trust that native memory won't take more than 2GB). I do want to limit all the memory (heap, native, whatever). And do that effectively, not just saying "-Xmx1GB" - to be safe.
There is too many different parameters related to memory, and I don't know how to configure all of them to achieve the goal.
So, I don't want to go there and care about heap, perm and whatever types of memory. My high-level expectation is: since there is only 8GB, and some static memory is needed - take the static memory from the 8GB, and carefully split the remaining memory between other dynamic memory entities.
Also, ulimit and similar things don't work. I don't want to kill the java process once it consumes more memory than expected. I want Java does its best to not reach the limit firstly, and only if it really, really can't - kill the process.
And I'm OK to define even 100 java parameters, why not. :) But then I need assistance with the full list of needed parameters (for, say, Java 8).
Have you tried -XX:MetaspaceSize?
Is this what you need?
Please, read this article: http://karunsubramanian.com/websphere/one-important-change-in-memory-management-in-java-8/
Keep in mind that this is only valid to Java 8.
AFAIK, there is no java command line parameter or set of parameters that will do that.
Your best bet (IMO) is to set the max heap size and the max metaspace size and hope that other things are going to be pretty static / predictable for your application. (It won't cover the size of the JVM binary and it probably won't cover native libraries, memory mapped files, stacks and so on.)
In a comment you said:
So I'm forced to have a significant amount of memory unused to be safe.
I think you are worrying about the wrong thing here. Assuming that you are not constrained by address space or swap space limitations, memory that is never used doesn't matter.
If a page of your address space is not used, the OS will (in the long term) swap it out, and give the physical RAM page to something else.
Pages in the heap won't be in that situation in a typical Java application. (Address space pages will cycle between in-use and free as the GC moves objects within and between "spaces".)
However, the flip-side is that a GC needs the total heap size to be significantly larger than the sum of the live objects. If too much of the heap is occupied with reachable objects, the interval between garbage collection runs decreases, and your GC ergonomics suffer. In the worst case, a JVM can grind to a halt as the time spent in the GC tends to 100%. Ugly. The GC overhead limit mechanism prevents this, but that just means that your JVM gets an OOME sooner.
So, in the normal heap case, a better way to think about it is that you need to keep a portion of memory "unused" so that the GC can operate efficiently.
This is related to my question Java Excel POI stops after multiple execution by quartz.
My program stops unexpectedly after a few iterations. I tried profiling and found that I was consuming a lot of heap memory per iteration (And a memory leak somewhere.. havn't found the bugger yet). So, as a temporary solution, I tried inserting System.gc(); at the end of each complete execution of the program (kindly read the linked question for a brief description of the program). I was not expecting much, maybe a few more heap space available after each iteration. But, it appears that the program uses less heap memory when I inserted System.gc();.
The top graph shows the program running with the System.gc(); while the bottom graph is the one without. As you can see the top graph shows that I'm only using less than a 100mb after 4 iteratioins of the program while the bottom graph shows over 100mb in usage for the same amount of iterations. Can anyone clarify how and why System.gc(); causes this effect in my heap? If there are any disadvantages if I were to use this in my program? Or I'm completely hopless in programming and take up photography instead?
Note that I inserted GC at the end of each program iteration. So I assume that heap usage must be the same as without the GC inserted until it meets the the System.gc(); command
Thanks!
Can anyone clarify how and why System.gc(); causes this effect in my heap?
System.gc is kind of a request service for the Garbage Collector to Run. Note that I have used request and not trigger in my statement. GC based upon the heap state might/not carry on collection.
If there are any disadvantages if I were to use this in my program?
From experience, GC works best when left alone. In your example you shouldn't worry or use System.gc. Because GC will run when it is best to run and manually requesting it might reduce the performance. Even though only a small difference, you can observe that "time spent on gc" is better in the below graph than the first one.
As per memory, both the graphs are OK. Seems like your max heap is a bit high. Hence GC did-not run it in second graph. If it was really required, it would have ran it.
As per the Java specs, calling gc() does not guarantee that it will run, you only hint to the JVM that you need it to run, so the result is unreliable (You should avoid calling gc() at not matter what). But, in your case here and since the heap is reaching critical limits incrementally, that's why perhaps your hints are being executed.
GC usually runs based on specific algorithms to prevent the heap from being exhausted and when it fails to reclaim the much needed space while having no more heap for you app to survive, you'll face the OutOfMemoryException.
While the GC is running, your application will experience some pauses as a result of its activities, so you won't really want it to run more often!
Your best bet is to solve the leak and practice better memory management for a healthy runtime experience.
Using System.gc() shouldn't impact the heap size allocated to JVM. Heap size is dependent only on startup arguments we provide to our JVM. I will recommend you to run the same program 3-4 times and take average values with System.gc() and without.
Coming back to the problem of finding the memory leak; I will recommend to use JProfiler or other tools which would tell you exact memory footprint; and different objects in the heap.
Last but not the least; you are a reasonable programmer. No need of going for a photo shoot :)
I'm writing a Java/Swing application with ~30 class my probleme is when i run my programe it load more than 150 M of the memory, is that normal ? since the application have 4 threads, parse some XML files, load some icon file, and drow some Jfreechat charts.
if not how can i do to minimize the amount of memory used by the application, is affecting some variables to null help? is loading the XML files once to use them in all the application life cycle help or i have to load them evry time i need them? is there some other tips that help me?
PS: im devlopping with a 8G memory computer in case that can affect the memory used by my program.
EDIT: it appeared that the program don't occupy all the 150MB because i get this value from the top command on linux, by running this code in my application as vilmantas advises me:
long free = Runtime.getRuntime().freeMemory();
long total = Runtime.getRuntime().totalMemory();
long max = Runtime.getRuntime().maxMemory();
long used = total - free;
I found that he occupy much less than that (~40MB) so i decide to run it with "-Xmx40M" argument and i reduce more than 40% of memory usage in the Top command.
The problem who are occupying the rest of memory since JVM (as i know) have his own process ? and how to make this operation automatic**?** because when choosing a not appropriate value you can get a memory exception as i have by running with "-Xmx30M" argument:
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
It is. This is Java, usually your VM/GC will do the job for you. Worry about memory usage when and if it becomes a problem.
If you want, there are several tools that can help you analyze what is going on. How to monitor Java memory usage?
Setting variables to null can help preventing memory leaks, if the referring variable's life cycle is greater than the referred instance. So that variables that should hold-on through the whole application life cycle are better not hold references to temporary objects that are used for a short time.
Loading the XMLs only once can help if you're good with loading its information only once. Meaning, if the XML is changed otherwise than through your application and you need to get the update - you'll have to reload the XML (and if the deprecated XML info is no longer needed - get rid of it).
You could use java memory heap analyzer like http://www.eclipse.org/mat/ to identify the parts of your application that use up most of the memory. You can then either optimize your data structures, or decide release parts of the data by setting all references to it to null.
Unintended references to data that is not needed anymore are also refered as "memory leaks". Settings those references to null will cause the garbage collector to remove it from java memory heap.
Along that line, you might find WeakReferences helpful.
Where do you observe those 150M? Is that how much your JVM process occupies (e.g. visible in the top command on linux/unix) or is it really the memory used (and necessary) by your application?
Try writing the following 4 values when your application runs:
long free = Runtime.getRuntime().freeMemory();
long total = Runtime.getRuntime().totalMemory();
long max = Runtime.getRuntime().maxMemory();
long used = total - free;
If the value for "used" is much lower than 150M, you may add java start parameter e.g. "-Xmx30M" to limit the heap size of your application to 30MB. Note that the JVM process will still occupy a little bit more than 30MB in such case.
The memory usage by JVM is somewhat tricky.
I want to limit the maximum memory used by the JVM. Note, this is not just the heap, I want to limit the total memory used by this process.
use the arguments -Xms<memory> -Xmx<memory>. Use M or G after the numbers for indicating Megs and Gigs of bytes respectively. -Xms indicates the minimum and -Xmx the maximum.
You shouldn't have to worry about the stack leaking memory (it is highly uncommon). The only time you can have the stack get out of control is with infinite (or really deep) recursion.
This is just the heap. Sorry, didn't read your question fully at first.
You need to run the JVM with the following command line argument.
-Xmx<ammount of memory>
Example:
-Xmx1024m
That will allow a max of 1GB of memory for the JVM.
If you want to limit memory for jvm (not the heap size )
ulimit -v
To get an idea of the difference between jvm and heap memory , take a look at this excellent article
http://blogs.vmware.com/apps/2011/06/taking-a-closer-look-at-sizing-the-java-process.html
The answer above is kind of correct, you can't gracefully control how much native memory a java process allocates. It depends on what your application is doing.
That said, depending on platform, you may be able to do use some mechanism, ulimit for example, to limit the size of a java or any other process.
Just don't expect it to fail gracefully if it hits that limit. Native memory allocation failures are much harder to handle than allocation failures on the java heap. There's a fairly good chance the application will crash but depending on how critical it is to the system to keep the process size down that might still suit you.
The NativeHeap can be increasded by -XX:MaxDirectMemorySize=256M
(default is 128)
I've never used it. Maybe you'll find it useful.
Following is the scenario i need to solve. I have struck with two solutions.
I need to maintain a cache of data fetched from database to be shown on a Swing GUI.
Whenever my JVM memory exceeds 70% of its allocated memory, i need to warn user regarding excessive usage. And once JVM memory usage exceeds 80%, then i have to halt all the database querying and clean up the existing cache fetched as part of the user operations and notifying the user. During cleanup process, i will manually handle deleting some data based up on some rules and instructs JVM for a GC. Whenever GC occurs, if memory cleans up and reaches 60% of the allocated memory, I need to restart all the Database handling and giving back control to the user.
For checking JVM memory statistics i found following two solutions. Could not able to decide which is best way and why.
Runtime.freeMemory() - Thread created to run every 10 seconds and check for the free memory and if memory exceeds the limits mentioned, necessary popups will intimate user and will call the methods to halt the operations and freeing up the memory.
MemoryPoolMXBean.getUsage() - Java 5 has introduced JMX to get the snapshot of the memory at runtime. In, JMX i cannot use Threshold notification since it will only notify when memory reaches/exceeds the given threshhold. Only way to use is Polling in MemoryMXBean and check the memory statistics over a period.
In case of using polling, it seems for me both the implementations are going to be same.
Please suggest the advantages of the methods and if there are any other alternatives/any corrections to the methods using.
Just a side note: Runtime.freeMemory() doesn't state the amount of memory that's left of allocating, it's just the amount of memory that's free within the currently allocated memory (which is initially smaller than the maximum memory the VM is configured to use), but grows over time.
When starting a VM, the max memory (Runtime.maxMemory()) just defines the upper limit of memory that the VM may allocate (configurable using the -Xmx VM option).
The total memory (Runtime.totalMemory()) is the initial size of the memory allocated for the VM process (configurable using the -Xms VM option), and will dynamically grow every time you allocate more than the currently free portion of it (Runtime.freeMemory()), until it reaches the max memory.
The metric you're interested in is the memory available for further allocation:
long usableFreeMemory= Runtime.getRuntime().maxMemory()
-Runtime.getRuntime().totalMemory()
+Runtime.getRuntime().freeMemory()
or:
double usedPercent=(double)(Runtime.getRuntime().totalMemory()
-Runtime.getRuntime().freeMemory())/Runtime.getRuntime().maxMemory()
The usual way to handle this sort of thing is to use WeakReferences and SoftReferences. You need to use both - the weak reference means you are not holding multiple copies of things, and the soft references mean that the GC will hang onto things until it starts running out of memory.
If you need to do additional cleanup, then you can add references to queues, and override the queue notification methods to trigger the cleanup. It's all good fun, but you do need to understand what these classes do.
It is entirely normal for a JVM to go up to 100% memory usage and them back to say 10% after a GC and do this every few second.
You shouldn't need to try managing the memory in this way.
You cannot say how much memory is being retained until a full GC has been run.
I suggest you work out what you are really trying to achieve and look at the problem another way.
The requirements you mention are a clear contradiction with how Garbage Collection works in a JVM.
because of the behaviour of the JVM it will be very hard to warn you users in a correct way.
Altogether stopping als database manipulation , cleaning stuff up and starting again really is not the way to go.
Let the JVM do what it is supposed to do, handle all memory related for you.
Modern generations of the JVM are very good at it and with some finetuning of the GC parameters you will get a a much cleaner memory handling then forcing things yourself
Articles like http://www.kodewerk.com/advice_on_jvm_heap_tuning_dont_touch_that_dial.htm mention the pros and cons and offer a nice explanation of what the VM does for you
I've only used the first method for similar task and it was OK.
One thing you should note, for both methods, is to implement some kind of debouncing - i.e. once you recognize you've hit 70% of memory, wait for a minute (or any other time you find appropriate) - GC can run at that time and clean up lots of memory.
If you implement a Runtime.freeMemory() graph in your system you'll see how the memory is constantly going up and down, up and down.
VisualVM is a bit nicer than JConsole because it gives you a nice visual Garbage Collector view.
Look into JConsole. It graphs the information you need so it is a matter of adapting this to your needs (given that you run on a Sun Java 6).
This also allows you to detach the surveiling process from what you want to look at.
Very late after the original post, I know, but I thought I'd post an example of how I've done it. Hopefully it'll be of some use to someone (I stress, it's a proof of principal example, nothing else... not particularly elegant either :) )
Just stick these two functions in a class, and it should work.
EDIT: Oh, andimport java.util.ArrayList;
import java.util.List;
public static int MEM(){
return (int)(Runtime.getRuntime().maxMemory()-Runtime.getRuntime().totalMemory() +Runtime.getRuntime().freeMemory())/1024/1024;
}
public static void main(String[] args) throws InterruptedException
{
List list = new ArrayList();
//get available memory before filling list
int initMem = MEM();
int lowMemWarning = (int) (initMem * 0.2);
int highMem = (int) (initMem *0.8);
int iteration =0;
while(true)
{
//use up some memory
list.add(Math.random());
//report
if(++iteration%10000==0)
{
System.out.printf("Available Memory: %dMb \tListSize: %d\n", MEM(),list.size());
//if low on memory, clear list and await garbage collection before continuing
if(MEM()<lowMemWarning)
{
System.out.printf("Warning! Low memory (%dMb remaining). Clearing list and cleaning up.\n",MEM());
//clear list
list = new ArrayList(); //obviously, here is a good place to put your warning logic
//ensure garbage collection occurs before continuing to re-add to list, to avoid immediately entering this block again
while(MEM()<highMem)
{
System.out.printf("Awaiting gc...(%dMb remaining)\n",MEM());
//give it a nudge
Runtime.getRuntime().gc();
Thread.sleep(250);
}
System.out.printf("gc successful! Continuing to fill list (%dMb remaining). List size: %d\n",MEM(),list.size());
Thread.sleep(3000); //just to view output
}
}
}
}
EDIT: This approach still relies on sensible setting of memory in the jvm using -Xmx, however.
EDIT2: It seems that the gc request line really does help things along, at least on my jvm. ymmv.