We Keep Coding

Elastic APM : cpu usage when running application.jar in local system

I am running a sample application jar on local system using elasticAPM agent.
Elastic APM show 2 different cpu stats (system/process).
Metrics explanation on official site says the same thing for both stats
https://www.elastic.co/guide/en/apm/server/current/exported-fields-system.html
Please explain, Is the "system cpu stats" is of my system even when the agent is connected to application.jar only using java command? If so, how can I check on elastic apm what else on my system in consuming cpu since only application is running during the load test.
java -javaagent:<agent.jar> -jar <app.jar>
The CPU usage shown below

The metrics shown in Kibana are sent by the APM agent that like you said has limited access to your environment. It basically says anything that is collected by the JVM running your JAR.
If you want to get further visibility into the CPU details of your local environment then you must augment your setup using Elastic MetricBeats that ships O.S level details about your machine that sees beyond what the JVM can see.
In the presentation below I show how to configure logs, metrics, and APM altogether.
https://www.youtube.com/watch?v=aXbg9pZCjpk

High CPU consumption issue while Tomcat was configured as windows service

I have deployed one java application in tomcat server. And tomcat server was configured as windows service in one of my VMs.
Our VMs are windows servers with 64GB RAM and 8 core 2.4 GHz Intel Xeon Processors.
Below are the software details and JVM args configured.
JDK 1.7.0_67
Tomcat 7.0.90
JVM args for Tomcat :
-Xms2g -Xmx40g -XX:PermSize=1g -XX:MaxPermSize=2g
But still getting this issue, could you please any one help.

You can enable JMX (which is a technology to monitor java applications) by adding the -Dcom.sun.management ..... jvm options on the startup script and connect your application via JConsole with JTop Plugin which shows the top CPU consuming threads. Refer :https://arnhem.luminis.eu/top-threads-plugin-for-jconsole/

Unable to connect profiler (JMC) on VirtualBox VM

I'm running an application on Weblogic server hosted inside a VirtualBox VM (Windows 2008 Server). To connect JRockit Mission Control Profiler my server aruguments is as below :
-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=4000,server=y,suspend=n
When I start my server, in ProcessExplorer I can see the application listening to port 4000.
But in Mission Control UI, I'm neither able to see this application in default local connection nor connect as new using manual port configurations. I'm able to see connections for other Java applications in the same VM.
My configurations are like this:
Guest (VM) OS : Windows 2008 Server
Host OS : Windows 7 VM Network : Bridged Adapter (Promiscuous : Deny)
Server : Weblogic 12c
Java : 1.6
JVM : Oracle JRockit(R) (build
R28.2.5-20-152429-1.6.0_37-20120927-1915-windows-x86_64, compiled
mode)
VM : VirtualBox 4.2.4
The closest answer I could find which seeems to have worked for Linux VM is this.
I tried below :
Windows Firewall with Advanced Securtiy -> Inbound rules -> Added
port 4000 with full access
Windows Firewall with Advanced Securtiy ->
Outbound rules -> Added port 4000 with full access
Changed VM Network
-> Bridged Adapter -> Promiscuous : Allow All
Changed VM Network -> NAT
But no luck yet. Anyone had this problem before?

The JVM flags above is for JDWP style debugging and not connecting remotely using JMXRMI which is what JRockit Mission Control is using.
You should run with the -Dcom.sun.management.jmxremote properties,
http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html
or for JRockit you can also use the shorthand flag -Xmanagement
http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionx.htm#BABJDIGF
I would suggest
-Xmanagement:port=4000,ssl=false,authenticate=false
Then you should be able to create a custom jmc connection to server os hostname/ip and port 4000.
If you are running both WLS and JRMC on the same host, you should be able to see the running WLS in the JVM Browser though. If you don't, you can still use the same flags as above.

Visual GC in Java VisualVM via jstadt for remote Tomcat

I'm trying to get the memory layout of a remote Tomcat visualized with Visual GC in Java VisualVM.
On the Linux server which hosts the Tomcat jstatd is running (including tools.policy and the proper [-J]-Djava.rmi.server.hostname=[SERVER_IP], which is to be found in the jvm options of the Tomcat as well).
The connection from VisualVM to the server/jstatd works properly:
But all i can see in the Visual GC tab of the Jstatd connection are the memory details of the Jstatd JVM itself, not the data of the Tomcat.
Edit: And in the Visual GC tab of the Tomcat connection (prod apl01 jmx) I'm being told: "Not supported for this JVM"
Testing this with a system we run inhouse I get proper data here.
The remote Tomcat runs with it's own user "tomcat".
Must jstatd be started by this user as well?
Or is there another issue I might be missing?
Any working alternative like another visual tool that provides a visualization similar to the one below would be absolutely acceptable.
We're using Yourkit as well, but I couldn't find anything in there that resembles Visual GC easy to understand style.

Having done some more research and tests on our locale development environment i realized that we haven't had the right permissions on the production environment.
So i asked the admin to add jstatd to the sudoers list and voilà it works as expected:
sudo /usr/java/latest/bin/jstatd -J-Djava.security.policy=/home/empulse/tools.policy -J-Djava.rmi.server.hostname=[SERVER_IP]

On remote server create policy,
vim /tmp/tools.policy
Add Permission,
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
And start jstatsd,
jstatd -J-Djava.security.policy=/tmp/tools.policy -J-Djava.rmi.server.hostname=<public_ip>
and you ready are with Visual GC inside VisualVM on local machine.

Using the Netbeans 6.5 profiler to profile a remote web app?

I'm having trouble using the Netbeans 6.5 profiler to profile the memory usage of a java web app, remotely.
I have followed the steps in the tutorial, and was able to customize a Tomcat instance to start with the profiling agents. I can also attach the Netbeans profiler to the running server: for some seconds, the profiler shows graphs with the memory usage of the server.
However, I am not able to run my deployed web app. After a few seconds, the server is shut down gracefully as if it simply had reached the bottom of its method main (hardly the expected behavior of a web server).
During these seconds, the server is not responsible, and I can't connect to a web application in it, they are simply not deployed.
I do not want to import my app into a new Netbeans project just for profiling it. It's possible to profile a remote app without access to its source, right ? After all, for a few seconds, the server was running and being profiled.
I want to profile a web application in Netbeans 6.5. What did I forget ?

You can use VisualVM tool as well. Follow the below mentioned steps.
1) Run jstatd ( jstat demon process) ( Run on your remote machine where application is running)
jstatd -p 1098 -J-Djava.security.policy=tools.policy ( specify unused port ) if tools.policy file is not specified create it as follows.
tools.policy
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
2) Also specify the following properties before you start your JVM.
#JMX property
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote=true"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.port=10992"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.authenticate=false"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.ssl=false"
3) Start the visual VM on local machine.
a. Under remote tab add remote node.
b. Wait for couple of minutes you will see all the core applications instance will be seen under remote tab.
c. Click on remote node then select Add JMX connections specify node:port (port mentioned above in point 2.)
d. Wait for few seconds you will see JMX connection under remote node.
4) After few seconds you will see new tab appearing under remote node. Double click on the remote_node:10992 you will see the profile table on right hand side.
You can see memory , cpu , thread etc details ..

Are profiling the entire webserver or did you use the profiling filter to profile only your package?
Reducing the interaction between the profiler and the app server itself can solve it.

Check your jvm version. The version used by netbeans should be same as that in tomcat.
Use default ports. If you have changed any ports ensure they are above 1024.
Try to profile the example application in tomcat (remove all other applications). If it is running sucessfully it means there is a memory constraint.
Profiling requires huge memory. use -xms1024m -xmx1024m for tomcat.
If every thing else fail. think of switching to jdk6 and tomcat 6.
After many trials I discoverd that there are lot of limitations in jdk5 and tomcat5. switching to tomcat6 and jdk6 solved the problem for me.
Profiling with netbeans does not work jrockit, ibm jdk etc..
--kiran.kumar