We Keep Coding

Java memory usage: Can someone explain the difference between memory reported by jconsole, ps, and prstat?

I'm investigating some memory bloat in a Java project. Confounded by the different statistics reported by different tools (we are using Java 8 on Solaris 10).
jconsole gives me three numbers:
Committed: the amount reserved for this process by the OS
Used: the amount actually being used by this process
Max: the amount available to the process (in our case it is limited to 128MB via Java command line option -Xmx128m).
For my project, jconsole reports 119.5MB max, 61.9MB committed, 35.5MB used.
The OS tools report something totally different:
ps -o vsz,rss and prstat -s rss and pmap -x all report that this process is using around 310MB virtual, 260MB physical
So my questions are:
Why does the OS report that I'm using around 5x as much as jconsole says is "committed" to my process?
Which of these measurements is actually accurate? (By "accurate", I mean, if I have 12GB of memory, can I run 40 of these (# 300MB) before I hit OutOfMemoryException? Or can I run 200 of them (# 60MB)? (Yes, I know I can't use all 12GB of memory, and yes I understand that virtual memory exists; I'm just using that number to illuminate the question better.)

This question goes quite deep. I'm just going to mention 3 of the many reasons:
VMs
Shared libraries
Stacks and permgen
VMs
Java is like a virtual mini computer. Imagine you ran an emulator on your computer that emulates an old macintosh computer, for example. The emulator app has a config screen where you set how much RAM is in the virtual computer. If you pick 1GB and start the emulator, your OS is going to say the 'Old Mac Emulator' application is taking 1GB. Eventhough inside the virtual machine, that virtual old mac might be reporting 800MB of 1GB free.
A JVM is the same thing. The JVM has its own memory management. As far as the OS is concerned, java.exe is an app that takes 1GB. As far as the JVM is concerned, there's 400MB available on the heap right now.
A JVM is slightly more convoluted, in that the total amount of memory a JVM 'claims' from the OS can fluctuate. Out of the box, a JVM will generally not ask for the maximum right away, but will ask for more over time before kicking in the garbage collector, or a combination thereof: Heap full? Garbage collect. That only freed up maybe 20% or so? Ask the OS for more. -Xms and -Xmx control this; set them to the same, and the JVM will on bootup ask for that much memory and will never ask for more. In general a JVM will never relinquish any memory it claimed.
JVMs, still, are primarily aimed at server deployments, where you want the RAM dedicated to your VM to be constant. There's little point in having each app take whatever they want when they want it, generally. In contrast to desktop apps where you tend to have a ton of apps running and given that a human is 'operating' it, generally only one app has particularly significant ram requirements.
This explains jconsole, which is akin to reporting the free memory inside the virtual old mac app: It's reporting on the state of the heap as the JVM sees it.
Whereas ps -o and friends are memory introspection tools at the OS level, and they just see the JVM as a big black box.
Which one is actually accurate
They both are. From their perspective, they are correct.
Shared library
OSes are highly complex beasts, these days. To put things in java terms, you can have a single JVM that is concurrently handling 100 simultaneous incoming https calls. One could want to see a breakdown of how much memory each of the currently 100 running 'handlers' is taking up. Okay... so how do we 'file' the memory load of String, the class itself (not any particular instance of String - the code. e.g. the instructions for how .toLowerCase() runs. Those are in memory too, someplace!). The web framework needs it, so does the core JVM, and so does probably every single last one of those 100 concurrent handlers. So how do we 'bookkeep' this?
In other words, the memory load on an entire system cannot be strictly divided up as 'that memory is 100% part of that app, and this memory is 10)% part of this app'. Shared libraries make that difficult.
The JVM is technically capable of rendering UIs, processing images, opening files both using the synchronous as well as the asynchronous API, and even the random access API if your OS offers a separate access library for it, sending network requests in async mode, in sync mode, and more. In effect, a JVM will immediately tell the OS: I can do allllll these things.
In my experience/recollection, most OSes report the total memory load of a single application as the sum of the memory they need as well as all the memory any (shared) library they load, in full.
That means ps and friends overreport JVMs considerably: The JVM loads in a ton of libraries. This doesn't actually cost RAM (The OS also loaded these libraries, the JVM doesn't use any large DLLs/.SO/.JNILIB files of its own, just hooks up the ones the OS provides, pretty much all of them), but is often 'bookkept' as such. You know this is happening if this trivial app:
class Test { public static void main(String[] args) throws Exception {
System.out.println("Hello!");
Thread.sleep(100000L);
}}
Already takes more than ~60MB or so.
I mean, if I have 12GB of memory, can I run 40 of these (# 300MB)
That shared library stuff means each VM's memory load according to ps and friends are over-inflated by however much the shared libraries 'cost', because each JVM is going to share that library - the OS only loads it once, not 40 times.
Stacks and permgen
The 'heap', which is where newly created objects go, is the largest chunk of any JVM's memory load. It's also generally the only one JVM introspection tools like jconsole show you. However, it's not the only memory a JVM needs. There's a small slice it needs for its core self (the 'C code', so to speak). Each active thread has a stack and each stack also needs memory. By default it's whatever you pass to -Xss, but times the number of concurrent threads. But that's not a certainty: You can construct a new thread with an alternate size (check the constructors of j.l.Thread). There used to be 'permgen' which is where class code lived. Modern JVM versions got rid of it; in general newer JVM versions try to do more and more on heap instead of in magic hard-to-introspect things like permgen.
I mean, if I have 12GB of memory, can I run 40 of these (# 300MB) before I hit OutOfMemoryException?
Run all 40 at once, and always specify both -Xms and -Xmx, setting them to equal sizes. Assuming all those 40 JVMs are relatively stable in terms of how many concurrent threads they ever run, if you're ever going to run into memory issues, it'll happen immediately (due to -Xms and -Xmx being equal you've removed the dynamism from this situation. All JVMs pretty much instaclaim all the memory they will ever claim, so it either 'works' or it won't. Stacks mess with the cleanliness of this somewhat, hence the caveat of stable-ish thread counts).

How I can set cpu and memory limit for a process in java?

Hi
I want to make a website using java and tomcat server.
The website will set cpu and memory limit for each user for their all process on a server. For example for each user %3 cpu and 100mb memory limit.
How can I make it?
Thanks

It is not possible with one Tomcat process. You need to spawn multiple child processes, each with it's own (-Xmx100m OR OS-specific memory quota) AND OS-specific settings to control CPU quota.
Depending on how malicious your users may become, you may also restrict number of available file descriptors, port ranges, disk quota, etc. At the end it may worth it to place each user's process into a VM and/or jail.

There is no mechanism available in Java for enforcing limits on threads resource usage. The only way you can do this, is by getting the underlying operating system to do it which will most likely require you to spawn a separate JVM for each restriction, and it will not be portable across platforms.
Note that you can put a timeout limit on a function call (using Executor) and have most operations interrupted. This might be good enough for you. If you need more, you again need the operating system to step in.

Java Visual VM skewing CPU

i am trying to analyze the CPU usage for a Java UI application running on Windows. I connected it to VisualVM, but it looks like the highest percentage for CPU usage is being used by
sum.rmi.transport.tcp.TCPTransport$ConnectionHandler.run();
I believe this is being used to supply information to VisualVM and hence VisualVM is skewing the results that i'm trying to investigate. Does any one have a way to get a better indication of what is occurring or a better method to determine what in a running java application is taking up so much CPU.

Try to use sampler first.
For detailed information use the profiler and set root methods. See Profiling With VisualVM, Part 1 and Profiling With VisualVM, Part 2 for more information about CPU and Memory profiling.

That sounds awfully suspicious. Try cross referencing the data with results from hprof. You won't need any external applications running, and the data will simply be dumped to a text file from your own process. Are you connecting to your process remotely?

Can current Java/Tomcat 6 application utilize 64bit windows platform advantage in term of performance and memory usage

Currently we have developed application using Java 6 based on windows 32bit(Dual core & 3G Ram).
If we install into 64bit windows OS, does it will perform better because of the resources advantage that having in the 64bit(Same OS diff. bit)? The 64bit machine is having Quad core processor and Ram more than 4g. Is the any different for JVM between 32bit vs 64bit.
Thank you in advance for your feedback.
Extra info
I am doing Security Information Event management Sys.(SIEM) - log management.
We have 4 important parts ,
Collector -to collect logs from devices/system,
Aggregator -To aggregate the syslog to be meta data for reporting,
Real Time Monitoring-To display realtime analisys report/charts and dashboard that must run every second
GUI - Struts2 apps. that runs the web GUI, log analytics, backup and other things
So far the most resources cpu and memory are used by 1-Collector, 2-RealTime, 3-Aggregator.
Right now in 32bit, collector can recieved up to 2000logs per seconds. If more than that it will crash to memory heap. So we used tanuki software to auto restart back the collector service. We use the Tanuki to split the memory usage and auto restart once detected memory heap.
Our objective is to increase event per second from 2000logs to maximum if possible by using 64bit advantages.
For the GC we let the Java handle automatically, more important we can process the more logs in 1 second without any problem.

Switching to a 64-bit JVM doesn't guarantee any performance differences. You will, however see a huge difference in the amount of RAM that can be allocated. On 32-bit Windows, the maximum amount of RAM that could be allocated for the heap maxed out at around 1.6 GB.
If you see a lot of swapping with your application on the 32-bit machine, then switching to the 64-bit machine and adding sufficient RAM is likely to improve your performance. You might also be able to make design choices that favor faster, but more memory hungry algorithms where such choices exist.
As of this writing, you will probably not see significant difference between running your app on a 32-bit JVM and a 64-bit JVM on the exact same hardware. Eventually, support for 32-bit operating systems and JVMs will probably be discontinued, but that's a different concern than performance.
I strongly recommend you start out by profiling your app first to see where your performance hot spots are.
It's a common misconception that 64-bit automatically means better performance than 32-bit. See e.g. this JVM faq and this MS Windows 7 FAQ.

It really depends on the nature of your application and where your performance bottlenecks are.
If you have relatively un-tuned garbage collection, and your application is latency sensitive (i.e. must respond to a user request such as an http request quickly), adding more memory can actually worsen your GC pauses.
Is your application multi-threaded, as most web servers are? If so, going from 2 to 4 cores will very likely help if you don't have significant locking / contention issues.
If you look into GC tuning, you might want to try parallel GC on the 4 core cpu. This can significantly reduce GC pause times while incurring some extra overhead. For a latency sensitive app I worked on this was definitely worth it.
Please feel free to reply with more info - we could use some context on your app, it's workload, in-memory working set, etc.

How to measure CPU and memory during load test

We are using JMeter in a heterogeneous environment and we would like to measure CPU and memory while we are producing heavy load on a server where our service or web application is running. What would you recommend to use? Is there any software solution (open source or commercial) available which records CPU and memory usage in a file or to a database?
Thank you!

You can use our JMeter perfmon plugin to measure CPU, memory and a lot of other metrics:

From the java perspective, you can use jconsole or write your own code to monitor the memory usage.
Measure CPU and memory consumption of a Java application
javasysmon on github
dstat for linux and perfmon for windows. Read more

You could take a look at Cacti.
It could collect resource usages like CPU, MEM easily with built-in templates and display them in nice graphics. Both Windows and Uni* are supported.

One you install the Perfmon plugin, I also had to install the server agent, and open the firewall port 4444.
Here's a link which I found super useful.
http://grosan.co.uk/web-application-performance-testing-using-apache-jmeter-part-2-monitor-server-resources/

Perhaps you can take a look to collectd if you run in some unix/linux flavour. It will bring you not only CPU but also memory, Disk IO, network and a good number of plugins available.