I am running a sample application jar on local system using elasticAPM agent.
Elastic APM show 2 different cpu stats (system/process).
Metrics explanation on official site says the same thing for both stats
https://www.elastic.co/guide/en/apm/server/current/exported-fields-system.html
Please explain, Is the "system cpu stats" is of my system even when the agent is connected to application.jar only using java command? If so, how can I check on elastic apm what else on my system in consuming cpu since only application is running during the load test.
java -javaagent:<agent.jar> -jar <app.jar>
The CPU usage shown below
The metrics shown in Kibana are sent by the APM agent that like you said has limited access to your environment. It basically says anything that is collected by the JVM running your JAR.
If you want to get further visibility into the CPU details of your local environment then you must augment your setup using Elastic MetricBeats that ships O.S level details about your machine that sees beyond what the JVM can see.
In the presentation below I show how to configure logs, metrics, and APM altogether.
https://www.youtube.com/watch?v=aXbg9pZCjpk
Related
We're using Appdynamics Java agent for monitoring our production applications. We have noticed slow growth in memory and the application eventually stalls. We ran a head dump on one of the JVMs and got the below reports.
Problem Suspect 1:
The thread com.singularity.ee.agent.appagent.kernel.config.xml.a# 0x1267......
AD thread config Poller keeps local variable config size of 28546.79(15.89%) KB
Problem Suspect 2:
280561 Instances of
com.singularity.ee.agent.appagent.services.transactionmonitor.com.exitcall.p loaded by com.singularity.ee.agent.appagent.kernel.classloader.d# 0x6c000....
occupy 503413.3(28.05%) KB. These instances are referenced from one instance of java.util.HashMap$Node[]...
We figured that these classes were from the Appdynamics APM that hooks on to the running JVM and sends monitored events to the controller. There is so much convoluted process associated with reaching out to the vendor, so I am wondering if there are any work arounds for this like we enabling our java apps with JMX and Appd getting the monitoring events from JMX rather than directly hooking on to the applications' JVM. Thanks for your suggestions.
I've this JMX script that when I run for 100 virtual users, using my laptop, it works just fine without any errors. Here is the BlazeMeter report for the run. But when I increase the virtual users to 200, it is giving me this SSLHandshakeException error. Please find the BlazeMeter error report here.
To verify, the system can handle 200 virtual users, I've used four AWS JMeter machines (giving each to load 50 virtual users) and ran the same script, found no issues. Here is the report for this run.
I'm wondering what might go wrong when I run it for 200 virtual users from my laptop. Can anyone please help me? What can cause this?
Please note, I'm using a 16GB RAM machine (assigned 10GB of ram to JMeter), updated Jmeter properties file ("httpclient4.retrycount=1") and 'hc.parameters' (enabled "http.connection.stalecheck$Boolean=true" and increased socket timeout "http.socket.timeout$Integer=600000") file in the \bin folder.
Thanks in advance!
--Ishti
Given response time differences for local and AWS runs (1.55s vs 0.6s) and the fact load does not increase as virtual users number grows I would expect that your laptop either doesn't have enough hardware resources in order to kick off 200 concurrent users or you don't have enough network bandwidth. Check JMeter JVM stats using JConsole or something like it and main OS metrics using i.e. Windows Performance Monitor or equivalent.
Also if you're not load testing stgbasalt.peopleanswers.com domain, it's better to exclude it (as well as other external domains) from your test plan. If it comes through embedded resources - filter it via "URLs must match" section, see Excluding Domains From The Load Test guide for more information.
Respected Experts,
I have a stand alone java application and want to monitor the threads created by it. I am planning to use a tool like JConsole or JVisualVM. However, I am not able to connect these tools locally to my Java Application.
I am using Windows machine. JConsole and Java program are running locally. I have tried to run Java application with following JMV arguments with no success:
java -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false LinkedListTest
When I try to connect using JConsole, my process id is greyed and following message appears:
Note: The management agent is not enabled on this process
I think I should be able to connect JConsole to a stand alone java application. Any thoughts what I am missing here
Thanks and Regards
Thanks for the inputs. I was able to solve the problem and the details are as follows:
As #Holger mentioned, both JConsole and JVisualVM can connect to the local Java Application running without any JMX arguments. The problem that I was facing was machine specific. I restarted the machine and deleted the following directory:
%TMP%\hsperfdata_User.Name
(I was not able to delete this directory without doing a restart)
Restarted JConsole/JVisualVM and was able to connect to local java processes using the process id.
In fact, the use of JVisualVM pointed me towards this fix. On start of JVisualVM, I got an error message stating something like local processes/applications can't be monitored. The message had a link to Troubleshooting guide. I am reproducing the relevant snippet:
Local Applications Cannot Be Monitored (Error Dialog On Startup)
Description: An error dialog saying that local applications cannot be monitored is shown >immediately after VisualVM startup. Locally running Java applications are displayed as Application> (pid ###).
Resolution: This can happen on Windows systems if the username contains capitalized letters. In >this case, username is UserName but the jvmstat directory created by JDK is >%TMP%\hsperfdata_username. To workaround the problem, exit all Java applications, delete the >%TMP%\hsperfdata_username directory and create new %TMP%\hsperfdata_UserName directory.
However, on my machine directory had the following format:
%TMP%\hsperfdata_User.Name
So, my recommendations are:
-check the name of the above mentioned directory for presence of camel case user name
-If not, follow the steps from the Trouble Shooting guide
-If the problem persists, delete the directory (may require machine reboot, as in my case)
-Restart JConsole/JVisualVM
Hopefully, the problem would be resolved.
Thanks and Regards
Hi I noticed that different tools for monitoring JVM require that the process will be java.exe in the taskbar. Its process id is supplied to them and then statistics regarding garbage collection and etc are returned. When running tomcat as a service, there is no a process named jave.exe but it is tomcat6.exe.
How can I get statistics regarding the JVM tomcat6 uses ?
Enable JMX in Tomcat and then connect via JConsole or JVisualVM. The name of the process is not relevant.
I'm having trouble using the Netbeans 6.5 profiler to profile the memory usage of a java web app, remotely.
I have followed the steps in the tutorial, and was able to customize a Tomcat instance to start with the profiling agents. I can also attach the Netbeans profiler to the running server: for some seconds, the profiler shows graphs with the memory usage of the server.
However, I am not able to run my deployed web app. After a few seconds, the server is shut down gracefully as if it simply had reached the bottom of its method main (hardly the expected behavior of a web server).
During these seconds, the server is not responsible, and I can't connect to a web application in it, they are simply not deployed.
I do not want to import my app into a new Netbeans project just for profiling it. It's possible to profile a remote app without access to its source, right ? After all, for a few seconds, the server was running and being profiled.
I want to profile a web application in Netbeans 6.5. What did I forget ?
You can use VisualVM tool as well. Follow the below mentioned steps.
1) Run jstatd ( jstat demon process) ( Run on your remote machine where application is running)
jstatd -p 1098 -J-Djava.security.policy=tools.policy ( specify unused port ) if tools.policy file is not specified create it as follows.
tools.policy
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
2) Also specify the following properties before you start your JVM.
#JMX property
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote=true"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.port=10992"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.authenticate=false"
JVM_ARGS="${JVM_ARGS} -Dcom.sun.management.jmxremote.ssl=false"
3) Start the visual VM on local machine.
a. Under remote tab add remote node.
b. Wait for couple of minutes you will see all the core applications instance will be seen under remote tab.
c. Click on remote node then select Add JMX connections specify node:port (port mentioned above in point 2.)
d. Wait for few seconds you will see JMX connection under remote node.
4) After few seconds you will see new tab appearing under remote node. Double click on the remote_node:10992 you will see the profile table on right hand side.
You can see memory , cpu , thread etc details ..
Are profiling the entire webserver or did you use the profiling filter to profile only your package?
Reducing the interaction between the profiler and the app server itself can solve it.
Check your jvm version. The version used by netbeans should be same as that in tomcat.
Use default ports. If you have changed any ports ensure they are above 1024.
Try to profile the example application in tomcat (remove all other applications). If it is running sucessfully it means there is a memory constraint.
Profiling requires huge memory. use -xms1024m -xmx1024m for tomcat.
If every thing else fail. think of switching to jdk6 and tomcat 6.
After many trials I discoverd that there are lot of limitations in jdk5 and tomcat5. switching to tomcat6 and jdk6 solved the problem for me.
Profiling with netbeans does not work jrockit, ibm jdk etc..
--kiran.kumar