We are running some heavy deployments on weblogic setup and it takes around an hour. During that time, we want to take a memory snapshots/heap dumps to see how much headroom we have wrt memory to avoid crash. Is there any optional jvm arg that we can provide while starting the server which will do the job? I checked below link but nothing is fitting the requirement -
http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm
If acceptable to drive the snapshots from the outside then you can use jrcmd to send commands to your JVM.
To get the PID use
jrcmd -P
and then you can use
jrcmd PID hprofdump dumpfile.bin
See http://docs.oracle.com/cd/E15289_01/doc.40/e15062/diagnostic.htm#BABIACCC for hrpofdump and http://docs.oracle.com/cd/E15289_01/doc.40/e15061/ctrlbreakhndlr.htm#i1001760 for jrcmd.
Related
We all know good old flag +HeapDumpOnOutOfMemoryError for taking heap dumps when JVM runs out of memory. Problem with this is that for large heaps this takes more and more time.
There is a way to take fast heap dumps using GNU Debugger...
You effectively take the core file of the process (very fast) then convert it to heapdump format using jmap... this is the slowest part of the work.
However this is only if you take it manually, when your java apps run in containers there is usually a fixed timeout until your app will be killed non gracefully... for kube i believe it is 30 seconds by default.
For many reasons i do not want to extend this timeout to larger number. Is there a way to trigger only core file dump when out of java runs out of memory? or we are just limited with whatever +HeapDumpOnOutOfMemoryError flag offers?
I can think of 2 possible solutions but it wont be only for out of memory situations but crashes too:
You can use java's -XX:OnError option to run your own script or gcore, (gdb) generate-core-file(depends to your OS) to create an core dump that you can later use a debugger(like gdb) to attach to it.
You can enable auto core dumps in your OS in the way it provides. For Redhat:
To enable: Edit the related line in file /etc/systemd/system.conf as DefaultLimitCORE=infinity
Reboot and remove the limits of core dump by ulimit -c unlimited.
When your application crashes, the dumb must be created in its working directory.
How to troubleshoot/Optimize CPU usage in a Springboot application. Are the allocated resources sufficient for an application having a total of around a 300k user base? The application isn't heavy at all. It just calls third-party APIs and do the necessary checks and gives the response.
How to identify exact codes that could have been using more resources than normally required? I found out somewhere that tracking the processes id from top command and reaching to thread dump and looking up for the corresponding hexadecimal value of processid that could have been using more CPU is one way to figure out. This wasn't easily achievable as some of the commands suggested didn't work. I would appreciate any help or suggestions.
Thanks in advance.
Htop command output
Htop when it's normal
The process of Collection of Thread Stack is no different for a spring boot app. Before a boot app is containerized it is still a Jar. If you suspect that its your application that is actually contributing to the high CPUT then run your jar and attach a profiler to it and trace the code contributing to the high CPU on load. If you can not do it then take the thread dump of the running jar/java process and use any free or opensource tool to analyze the trace. The second logic explained is applicable for the containerized application as well.
Follow this steps to take the thread dump of a java app/boot app running inside a docker container :-
docker exec -it <containerName> jstack > someFile.txt
Take multiple snapshot of it for better visiblity and comparision.
If you have not added JMX enable options to the JVM commandline, do it to begin with:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=10000
-Dcom.sun.management.jmxremote.rmi.port=10000
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then on your local machine you start "jmc" from your JDK bin folder and connect to your spring boot server.
You will then be able to see all the threads and enable both CPU load and thread locks on all active threads.
Be aware though that the above opens up JVM for unauthenticated entry so keep the port safe.
Next if your JVM dies send a "kill -3" SIGHUP which will tell the JVM to dump the core. that can then be read via the Eclipse MAT plugin in order to analyze the JVM inner doings.
Another way is to install jolokia into your server for other ways to retrieve the same info.
I am currently working on creating a performance framework using jenkins and execute the performance test from Jenkins. I am using https://github.com/jmeter-maven-plugin/jmeter-maven-plugin this plugin. The sanity test with single user in this performance framework worked well and went ahead with an actual performance test of 200 users and within 2 mins received the error
java.lang.OutOfMemoryError: GC overhead limit exceeded
I tried the following in jenkins.xml
<arguments>-Xrs -Xmx2048m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8080 --prefix=/jenkins --webroot="%BASE%\war"</arguments>
but it didn't work and also noted that whenever I increased the memory the jenkins service stops and had to reduce the memory to 1Gb and then the service restarts.
Had increased the memory for jmeter and java as well but no help.
In the .jmx file view results tree and every other listener is disabled but still the issue persists.
Since I am doing a POC jenkins is hosted in my laptop and high level specs as follows
System Model : Latitude E7270 Processor : Intel(R) Core(TM) i5-6300U CPU # 2.40GHZ(4CPU's), ~2.5GHZ Memory : 8192MB RAM
Any help please ?
The error about GC overhead implies that Jenkins is thrashing in Garbage Collection. This means it's probably spending more time doing Garbage Collection than doing useful work.
This situation normally comes about when the heap is too small for the application. With modern multi generational heap layouts it's difficult to say what exactly needs changing.
I would suggest you enable Verbose GC with the following options "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
Then follow the advice here: http://www.oracle.com/technetwork/articles/javase/gcportal-136937.html
Few points to note
You are using the integrated maven goal to run your jmeter tests. This will use Jenkins as the container to launch your jmeter tests thereby not only impacting your work but also other users of jenkins
It is better to defer the execution to a different client machine like a dedicated jmeter machine which uses its own JVM with parameters to launch your tests (OR) use the one that you provide
In summary,
1. Move the test execution out of jenkins
2. Provide the output of the report as an input to your performance plug-in [ this can also crash since it will need more JVM memory when you process endurance test results like an 8 hour result file]
This way, your tests will have better chance of scaling. Also, you haven't mentioned what type of scripting engine that you are using. AS per Jmeter documentation, JSR223 with groovy has a memory leak. Please refer
http://jmeter.apache.org/usermanual/component_reference.html#JSR223_Sampler
Try adding -Dgroovy.use.classvalue=true to see if that helps (provided you are using groovy). If you are using Java 8, there is a high chance that it is creating unique class for all your scripts in jmeter and it is increasing the meta space which is outside your JVM. In that case, restrict the meta space and use class unloading and a 64 bit JVM like
-d64 -XX:+CMSClassUnloadingEnabled.
Also, what is your new generation size. -XX:NewSize=1024m -XX:MaxNewSize=1024m ? Please note jmeter loads all the files permanently and it will go directly to the old generation thereby shrinking any available space for new generation.
I want to write a simple Java code to monitor a Cassandra database with JMX. Now I'm stuck with retrieving the CPU usage of the database. As far as I figured out, a possible MBean would be the java.lang:type=OperatingSystem with attribute ProcessCpuLoad.
However, it seems in this case the result would be the CPU usage of all processes running in the JVM, not only the Cassandra threads. Is this assumption correct?
I also wonder what data is shown as CPU usage when connecting with JConsole to the database. Is it possible to get direct access to these values(I mean without JConsole)? Or is there another Mbean which gives exactly the desired values?
Thanks,
Nico
ProcessCpuLoad in the OS mbean is correct. Its not all JVMs, just the one JVM thats reporting it. You do not have multiple processes running within a single JVM, the JVM runs as a single process per java application.
You can use java.lang:type=Threading to monitor cpu time spent on individual threads but there are a ton of threads in Cassandra and it will probably never be totally right (miss things like GC time).
If dont want to use jconsole you can check:
ps -p <whatever-your-cassandra-pid-is> -o %cpu
# or depending on OS/installer
ps -p `cat /var/run/cassandra.pid` -o %cpu
I know that this is not "best practice" but I would like to know if I can auto restart tomcat if my deployed app throws an outofmemory exception
You can try to use the OnOutOfMemoryError JVM option
-XX:OnOutOfMemoryError="/yourscripts/tomcat-restart"
It is also possible to generate the heap dump for later analysis:
-XX:+HeapDumpOnOutOfMemoryError
Be careful with combining these two options. If you force killing the process in "tomcat-restart" the heap dump might not be complete.
I know this isn't what you asked, but have you tried looking through a heap dump to see where you may be leaking memory?
Some very useful tools for tracking down memory leaks:
jdk/bin/jmap -histo:live pid
This will give you a histogram of all live objects currently in the JVM. Look for any odd object counts. You'll have to know your application pretty well to be able to determine what object counts are odd.
jdk/bin/jmap -dump:live,file=heap.hprof pid
This will dump the entire heap of the JVM identified by pid. You can then use the great Eclipse Memory Analyzer to inspect it and find out who is holding on to references of your objects. Your two biggest friends in Eclipse Memory Analyzer are the histo gram and a right click -> references -> exclude weak/soft references to see what is referencing your object.
jconsole is of course another good tool.
not easily, and definitely not through the JVM that just suffered the out of memory exception. Your best bet would be some combination of tomcat status monitor coupled with cron scripts or related scheduled system administrator scripts; something to check the status of the server and automatically stop and restart the service if it has failed.
Unfortunately when you kill the java process. Your script will keep a reference to the tomcat ports 8080 8005 8009 and you will not be able to start it again from the same script. The only way it works for me is:
-XX:OnOutOfMemoryError="kill -9 %p" and then another cron or monit or something similar to ensure you have the tomcat running again.
%p is actually the JVM pid , something the JVM provides for you.
Generally, no. The VM is a bad state, and cannot be completely trusted.
Typically, one can use a configurable wrapper process that starts and stops the "real" server VM you want. An example I've worked with is "Java Service Wrapper" from Tanuki Software http://wrapper.tanukisoftware.com/doc/english/download.jsp
I know there are others.
To guard against OOMs in the first place, there are ways to instrument modern VMs via interface beans to query the status of the heap and other memory structures. These can be used to, say, warn in a log or an email if some app specific operations are pushing some established limits.
I use
-XX:OnOutOfMemoryError='pkill java;/usr/local/tomcat/bin/start.sh'
What about something like this? -XX:OnOutOfMemoryError="exec \`ps --no-heading -p $$ -o cmd\`"