My goal
I am trying to understand and figure out best practices or optimal ways or best methods how to properly set up resources for low performance-hungry and high performance-hungry Java Springboot app.
Examples
First example:
Let have low performance-hungry Springboot App which only computes a key for cache and calls Redis for data with the key. I have tried two configurations of resources and java opts.
replicaCount: 4
memory: 1.9Gi
cpuRequest: 800m
cpuLimit: 1200m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAM=1792m -Xmx768m -Xms768m -XX:MaxMetaspaceSize=1024m -XshowSettings:vm -XX:ActiveProcessorCount=2"
Performance is good and the application has a response time around 3.5 ms in the median. And for 0.99 percentile it is 90ms. GC pause count was 0.4 per minute and pause duration 20ms. Also little throttling.
Then I have tried this setup:
replicaCount: 4
memory: 3Gi
cpuRequest: 800m
cpuLimit: 10000m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=80 -XshowSettings:vm"
The application was more memory-hungry but the response time was the same, 3.5ms in the median. But only on 0.99 percentile was 72 ms. GC pause count was lower, something about 0.1 per minute, and pause duration 5 ms. Throttling during startup but no throttling during the run.
Second example:
Performance hungry Springboot application which loads data from DB calculates multiple things like the distance between two points, calculation of price for multiple deliveries, filtering possible transport or pickup points for the package.
It was running on 4 VPS with 4 CPUs and 4 GB. But on Kubernetes, it needs more resources than I expected.
replicaCount: 4
memory: 7.5Gi
cpuRequest: 2000m
cpuLimit: 50000m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=80 -XX:+UseStringDeduplication -XshowSettings:vm"
Performance is good, but vertical scaling was doing nothing, but only horizontal scaling provided better performance. CPU usage reported by Kubernetes is about 1 and has no throttling.
What I have done so far
Read multiple articles found via Google, but any not give me a proper
answer or explanation.
I have tried various settings for CPU and
memory resource limits on Kubernetes spec, but it was not doing what I expected. I expected, which is lowering response time and the ability to process more requests.
Scaling vertically does not help either, everything was still slow.
Scaled horizontally pods with low CPU and memory with specified Xms, Xmx, ... it does that, pods were stable, but not performant as possible. (I think.) And also some throttling of CPU also if CPU was not fully used.
Question
I have a big problem with properly setting memory and CPU on Kubernetes. I do not understand why memory usage is increasing when I give it more CPU (Xss is the default value) and usage is the same. The pod is not OOM killed only if a gap between committed memory and used memory is about 1 GB (for the second application example) or 500MB (for the first application example).
If I set Xmx, Xms, MetspaceSite, and Xss, then I can achieve lower memory usage and CPU usage. But in this case, increasing the pod memory limit is complicated because it is not defined as a percentage and I must every time calculate each Java opt.
Also If give application too much memory, at the begging it will start at some level, but after some time it every times goes to limit (until gap between committed and heap memory is 1GB - 500MB) and stays there.
So, how is the proper way to find the best resource settings and Java opts for Springboot applications running on Kubernetes? Should I give the application big resources and after some time, something like 7 days, lower it by maximal values on metrics?
Related
i am running some performance tests with Jmeter 5.0.
parsing a csv file with my parameters and sending a simple post request, nothing special.
100 threads configured in my thread group concurrency with 100 RPS(not a lot).
heap size set to max 15GB and min 1GB.
my machine physical RAM memory is 32GB.
i have tried to raise the heap size to 20GB and even 25GB but it still crashed due to lack of memory.
i have analyzed the gc logs and saw that for the first 15GB max heap size configuration and for the second most of the heap size (about 95%) is in use by the old generation, somehow the GC in unable to clean them.
i am not using GUI mode and non of the listeners.
the first 15GB configuration crashed after 1 hour and 20 minutes, the second lasted for just more than 2 hours.
here is the GCeasy log analysis for the 20GB:
20GB heap size
It's hard to say anything meaningful without seeing:
The full stacktrace (because OutOfMemory may have many faces)
The contents of your HProf file
The structure of your test plan
In general OutOfMemory error indicates a memory leak, so make sure to follow recommendations from the 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article.
A "blind shot". You can have 100 RPS with 100 virtual users only if your application response time is 1 second or less.
If application response time becomes > 1 second and you're using Precise Throughput Timer or Throughput Shaping Timer in order to force JMeter to reach 100 TPS, it might be the case the application simply cannot respond faster due to being overloaded and JMeter adds more and more threads trying to reach 100 RPS until it runs out of the resources.
I am using Jmeter in non GUI and not using any listener as of now. Jmeter is taking 14 g of memory at the time of start up in 4.0 version and 9g of memory in 5.0 version with out passing any configuration file and as the load is increasing (in 1 sec 250 request with loop count of 100) memory utilization is 32g. I do not want to increase heap size, can you tell me the solution how i can reduce the memory consumption?
Virtual memory usgae through top command
Simple jmx file
Most probably you cannot as heap utilization depends on the nature of your test, request and response sizes, number of Pre/Post Processors/Assertions, etc.
Make sure to follow JMeter Best Practices
Consider using a profiler tool i.e. JVisualVM or JProfiler or Heap Dump / Heap Analyzer to figure out which component(s) is consuming the heap
If you don't have enough RAM and suffer from intensive GC activities the only way of speeding your test up is going for distributed testing - this way you will be able to decrease the number of threads for each JMeter engine.
I have a program which essentially reads many files (mixed PDF usually below 10 MB + XML size of Kb) from disk and upload them one at a time into a legacy system. I let it run under visualvm when going on weekend, and this morning I came back to a graph showing that the heap usage was rather uneven in the hour it took the program to run. I was expecting a roughly level amount of heap usage over time.
I am looking for an explanation for this. This is with 64-bit Oracle JVM under Ubuntu 17.04 (Desktop, not server) with 32 GB RAM on a 4-core i5-2400 (no hyperthreading). The program is essentially single-threaded utilizing about 50% of a core and took an expected time to run.
I understand that if memory is not fully used it is released over time. I do not understand that the usage goes down over time, as the load should be quite evenly distributed. Am I seeing the result of the CPU throttling as the system is otherwise idle? Is some JVM optimization kicking in?
Actual GC logs would be more helpful, but it's probably just the GC heuristics adjusting their decisions about young and old heap size to meet throughput and pause time goals.
As long as you don't place limits on the JVM it will happily use more memory than the bare minimum to keep the program running. If it did not you would experience very very frequent and inefficient GCs. In other words, heap size = n * live object set size with n > 1. Depending on the collector used there may be no upper limit for n other than available memory since ParallelGC defaults MaxHeapFreeRatio to 100 while G1 uses 70.
The heuristics are far from perfect, they can paint themselves into a corner in some edge-cases or unnecessarily fluctuate between different equilibrium points. But the graph itself does not indicate a problem.
I have a Java application running on a Tomcat7 instance. I am using Java 8.
Now within my application there is a webservice of the format : http://zzz.xxx.zzz.xxx/someWebservice?somepar1=rrr&somePar2=yyy. This returns a String value of max 10 characters.
I have now started load testing this service using Jmeter. I am putting a load of 100 concurrent connections and getting a throughput of roughly 150 requests/second. The server is a 4 core-6GB machine and is only running Tomcat (application instance). The database instance is running on a separate machine. JVM is running with min 2GB and Max 4 GB memory allocation. Max Perm Size is 512 MB. Tomcat has enough threads to cater to my load (Max connections/threads/executor values have been setup correctly)
I am now trying to optimize this service and in order to do so I am trying to analyze memory consumption. I am using JConsole for the same. My CPU usage is not a concern , but when I look at the memory (HEAP) usage, I feel something is not right. What I observe is a sawtooth shaped graph, which I know is correct as regular GC clears heap memory.
My concern is that this sawtooth shaped graph has an upward trend. I mean that the troughs in the sawtooth seem to be increasing over time. With this trend eventually my server reaches max heap memory in an hour or so and then stabilizes at 4GB. I believed that if I am putting a CONSTANT load, then the heap memory utilization should have a constant trend, i.e. a saw tooth graph with its peaks and troughs aligned. If there is an upward trend I am suspecting that there is a memory leak of variables which are collecting over time and since GC isn't able to clear them there is an increase over a period of time. I am attaching a screenshot.
Heap Usage
Questions:
1). Is this normal behavior? If yes, then why does the heap continuously increase despite no change in load? I don't believe that a load of 100 threads should saturate 4GB heap in roughly 30 minutes.
2). What could be the potential reasons here? Do I need to look at memory leaks? Any JVM analyzer apart from JConsole which can help me pinpoint the variables which the GC is unable to clear?
The see-saw pattern most likely stems from minor-collections, the dip around 14:30 then is a major collection, which you did not take into account when doing your reasoning.
Your load may simply be so low that it needs a long time to reach a stable state.
With this trend eventually my server reaches max heap memory in an hour or so and then stabilizes at 4GB.
Supports that conclusion if you're not seeing any OOMEs.
But there's only so much one can deduce from such charts. If you want to know more you should enable GC logging and inspect the log outputs instead.
I often run workloads against my own web applications to try find performance issues.
Sometimes I see memory leaks etc at variable duration.
So I created a bash script to take javacores, kill -3 pid , every minute for 10 minutes and script is executed on the hour.
For a workload that runs for 120 hours, this will produce 1200 javacores.
I'm wondering,
Is this overkill? I'd like a continuous view of system (javacore every 5 minutes, for 120 hours), but don't want to impact perf
what is a reasonable frequency to automatically capture javacores against servlet based app?
Looks like we are looking at two issues:
Performance
OutOfMemoryError
Performance: for performance, determine the longest request you can tolerate and generate the javacores when its 3 to 5 times that amount. (Anything below 5 minutes to me is fine tuning and can difficult)
Let's say your longest request you want is 3 minutes, I would generate 3 javacores at evenly from 9 minutes to 15 minutes.
I usually suggest the link (collect manually) below but if you already wrote your own script use it
"MustGather: Performance, Hang or High CPU Issues on Linux"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21115785
OutOfMemoryError: see if your product is leaking, follow the steps in URL below and go to collect manually and google IBM heap analyzer (stand alone and free) and review the heap dump for potential leak suspects.
"MustGather: Native Memory Issues on Linux"
http://www.ibm.com/support/docview.wss?rs=180&uid=swg21138462
Personnaly, I prefer looking at heap dumps memory use to equal the XMX or nearly that.
Since this is an IBM JVM you could try using Health Center instead of taking javacores regularly:
http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/
This has profiling and memory monitoring views so should give you the data you are looking and save you analysing the javacore files yourself.