We are using Groovy, Grill, web socket and REST API in our application. Our production server shows high CPU as soon as there are 10+ users simultaneously accessing the app. On checking the server health I can see some logging activity showing high CPU (screenshot attached). I wanted to know if excess of logging contributes to high CPU utilization. Definitely there could be more other reasons for high CPU.
Related
I have a java application (based on dropwizard) that accepts HTTP requests, performs some in memory computations and returns HTTP responses.
I have another java application (jmeter) which load tests the first one by sending requests with random parameters.
When I run the test (both the test and applications on the same laptop) I get bad performance: throughput is low (25 requests per second (rps)), latency is high (150ms). What interesting is that CPU usage of the application is low too (10%). It seems that the application spends much time waiting for network calls.
But when some other network activity (not all, playing YouTube video doesn't help) is going on, for instance when I do a Skype call or open local auto-refreshing grafana dashboard the situation changes. CPU usage by the application increases to 50%, throughput increases to 70-90 rps, latency decreases to 50ms.
It seems that irrelevant network activity somehow speed-ups network calls between the application and test. Can anybody explain that behavior?
Win 10, Java 8.
Are you on a computer with power saving settings? (Such as a laptop on battery power?)
Try to turn off all kinds of power saving, on CPU as well as other devices.
We are using the GAE flexible environment for java (with openjdk8), with a memory limit of 2GB, running a spring boot app which connects to google cloudsql and a separate mongodb vm instance.
Now every once in a while, the instance running the service restarts (which means a downtime of our service for several minutes). The logs show that the oom_kill_process routine is called, which leads to the restart. After some research we started to monitor not just the memory usage of our service, but also of the other services running on the instance (https://github.com/GoogleCloudPlatform/appengine-sidecars-docker).
We found that the fluentd-logger running in a ruby environment uses a larger junk of the memory, more than the 400MB additionally reserved for the system according to the gae docs. One snapshot showed a usage of >650MB and it was still rising, minutes later the instance crashed and then restarted.
Is there a way to limit the memory usage of the fluentd-logger?
This might also be related to this: Google Cloud Platform App Engine Node Flexible Instance Ruby sitting at 50% RAM usage
As an intermediary solution we increased the total memory to 4GB, which made the gap between the restarts larger, but that shouldn't be the long term solution
I have a java+camel bases application which is consuming huge amount of CPU. it various from 40-90% of CPU in production. I tried to replicate the issue in my local environment and started 700 routes (file endpoints) and it is consistently taking 70-80% CPU.
I want to know is there any way I can reduce the CPU utilization by configuring some setting while starting up the routes?
regards
sanjay
We're currently testing out Alfresco Community on an old server (only 1GB of RAM). Because this is the Community version we need to restart it every time we change the configuration (we're trying to add some features like generating previews of DWG files etc). However, restarting takes a very long time (about 4 minutes I think). This is probably due to the limit amount of memory available. Does anybody know some features or settings that can improve this restart time?
As with all performance issues there is rarely a magic bullet.
Memory pressure - the app is starting up but the 512m heap is only just enough to fit the applications in and it is spending half of the start up time running GC.
Have a look at any of the following:
1. -verbose:gc
2. jstat -gcutil
2. jvisualvm - much nicer UI
You are trying to see how much time is being spent in GC, look for many full garbage collection events that don't reclaim much of the heap ie 99% -> 95%.
Solution - more heap, nothing else for it really.
You may want to try -XX:+AggressiveHeap in order to get the JVM to max out it's memory usage on the box, only trouble is with only 1gb of memory it's going to be limited. List of all JVM options
Disk IO - the box it's self is not running at close to 100% CPU during startup (assuming 100% of a single core, startup is normally single threaded) then there may be some disk IO that the application is doing that is the bottle neck.
Use the operating system tools such as Windows Performance monitor to check for disk IO. It maybe that it isn't the application causing the IO it could be swap activity (page faulting)
Solution: either fix the app (not to likely) or get faster disks/computer or more physical memory for the box
Two of the most common reasons why Tomcat loads slowly:
You have a lot of web applications. Tomcat takes some time to create the web context for each of those.
Your webapp have a large number of files in a web application directory. Tomcat scans the web application directories at startup
also have a look at java performance tuning whitepaper, further I would recomend to you Lambda Probe www.lambdaprobe.org/d/index.htm to see if you are satisfied with your gcc settings, it has nice realtime gcc and memory tracking for tomcat.
I myself have Alfresco running with the example 4.2.6 from java performance tuning whitepaper:
4.2.6 Tuning Example 6: Tuning for low pause times and high throughput
Memory settings are also very nicely explained in that paper.
kind regards Mahatmanich
I've got a somewhat dated Java EE application running on Sun Application Server 8.1 (aka SJSAS, precursor to Glassfish). With 500+ simultaneous users the application becomes unacceptably slow and I'm trying to assist in identifying where most of the execution time is spent and what can be done to speed it up. So far, we've been experimenting and measuring with LoadRunner, the app server logs, Oracle statpack, snoop, adjusting the app server acceptor and session (worker) threads, adjusting Hibernate batch size and join fetch use, etc but after some initial gains we're struggling to improve matters more.
Ok, with that introduction to the problem, here's the real question: If you had a slow Java EE application running on a box whose CPU and memory use never went above 20% and while running with 500+ users you showed two things: 1) that requesting even static files within the same app server JVM process was exceedingly slow, and 2) that requesting a static file outside of the app server JVM process but on the same box was fast, what would you investigate?
My thoughts initially jumped to the application server threads, both acceptor and session threads, thinking that even requests for static files were being queued, waiting for an available thread, and if the CPU/memory weren't really taxed then more threads were in order. But then we upped both the acceptor and session threads substantially and there was no improvement.
Clarification Edits:
1) Static files should be served by a web server rather than an app server. I am using the fact that in our case this (unfortunately) is not the configuration so that I can see the app server performance for files that it doesn't execute -- therefore excluding any database performance costs, etc.
2) I don't think there is a proxy between the requesters and the app server but even if there was it doesn't seem to be overloaded because static files requested from the same application server machine but outside of the application's JVM instance return immediately.
3) The JVM heap size (Xmx) is set to 1GB.
Thanks for any help!
SunONE itself is a pain in the ass. I have a very same problem, and you know what? A simple redeploy of the same application to Weblogic reduced the memory consumption and CPU consumption by about 30%.
SunONE is a reference implementation server, and shouldn't be used for production (don't know about Glassfish).
I know, this answer doesn't really helps, but I've noticed considerable pauses even in a very simple operations, such as getting a bean instance from a pool.
May be, trying to deploy JBoss or Weblogic on the same machine would give you a hint?
P.S. You shouldn't serve static content from under application server (though I do it too sometimes, when CPU is abundant).
P.P.S. 500 concurrent users is quite high a load, I'd definetely put SunONE behind a caching proxy or Apache which serves static content.
After using a Sun performance monitoring tool we found that the garbage collector was running every couple seconds and that only about 100MB out of the 1GB heap was being used. So we tried adding the following JVM options and, so far, this new configuration as greatly improved performance.
-XX:+DisableExplicitGC -XX:+AggressiveHeap
See http://java.sun.com/docs/performance/appserver/AppServerPerfFaq.html
Our lesson: don't leave JVM option tuning and garbage collection adjustments to the end. If you're having performance trouble, look at these settings early in your troubleshooting process.