Since moving to Tomcat8/Java8, now and then the Tomcat server is OOM-killed. OOM = Out-of-memory kill by the Linux kernel.
How can I prevent the Tomcat server be OOM-killed?
Can this be the result of a memory leak? I guess I would get a normal Out-of-memory message, but no OOM-kill. Correct?
Should I change settings in the HEAP size?
Should I change settings for the MetaSpace size?
Knowing which Tomcat process is killed, how to retrieve info so that I can reconfigure the Tomcat server?
Firstly check that the oomkill isn't being triggered by another process in the system, or that the server isn't overloaded with other processes. It could be that Tomcat is being unfairly targeted by oomkill when some other greedy process is the culprit.
Heap should be set as a maximum size (-Xmx) to be smaller than the physical RAM on the server. If it is more than this, then paging will cause desperately poor performance when garbage collecting.
If it's caused by the metaspace growing in an unbounded fashion, then you need to find out why that is happening. Simply setting the maximum size of metaspace will cause an outofmemory error once you reach the limit you've set. And raising the limit will be pointless, because eventually you'll hit any higher limit you set.
Run your application and before it crashes (not easy of course but you'll need to judge it), kill -3 the tomcat process. Then analyse the heap and try to find out why metaspace is growing big. It's usually caused by dynamically loading classes. Is this something your application is doing? More likely, it's some framework doing this. (Nb oom killer will kill -9 the tomcat process, and you won't be able to diagnostics after that, so you need to let the app run and intervene before this happens).
Check out also this question - there's an intriguing answer which claims an obscure fix to an XML binding setting cleared the problem (highly questionable but may be worth a try) java8 "java.lang.OutOfMemoryError: Metaspace"
Another very good solution is transforming your application to a Spring Boot JAR (Docker) application. Normally this application has a lot less memory consumption.
So steps the get huge improvements (if you can move to Spring Boot application):
Migrate to Spring Boot application. In my case, this took 3 simple actions.
Use a light-weight base image. See below.
VERY IMPORTANT - use the Java memory balancing options. See the last line of the Dockerfile below. This reduced my running container RAM usage from over 650MB to ONLY 240MB. Running smoothly. So, SAVING over 400MB on 650MB!!
This is my Dockerfile:
FROM openjdk:8-jdk-alpine
ENV JAVA_APP_JAR your.jar
ENV AB_OFF true
EXPOSE 8080
ADD target/$JAVA_APP_JAR /deployments/
CMD ["java","-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-jar","/deployments/your.jar"]
Related
I have ran nohup java -jar sample.jar & on my linux server as a backend service of web application but service is automatically killed after few hours and need to re-run the command.
I want to know what is the reason and how to resolve the issue
On Linux, one possibility is that it is the oomkiller process. This watches for processes that appears to be triggering excessive paging activity, and sends them a SIGKILL signal. The oom killer's purpose is to prevent the paging system from thrashing ... which can bring the entire system to its knees.
If it is the oom killer killing your Java app, you should see a log message in /var/log/messages or /var/log/syslog.
The solution may be as simple as reducing the Java app's heap size so that it all fits in the available physical memory. Alternatively, if your Java app's heap usage is growing all of the time, you may need to look for a possible memory leak in your app. (Search for articles on finding Java memory leaks ...)
I run Spring Boot via the same command and only a handful of times have I seen this behavior. It turned out that the JVM ran out of memory and it crashed. When I changed the heap size settings it was rock solid again.
We are running a set of Java applications in docker containers on OpenShift. On a regular basis we experience oom kills for our containers.
To analyse this issue we set up Instana and Grafana for monitoring. In Grafana we have graphs for each of our containers showing memory metrics e.g. JVM heap, memory.usage and memory.total_rss. From these graphs we know that the heap as well as the memory.total_rss of our containers is pretty stable on a certain level over a week. So we assume that we do not have a memory leak in our Java application. However, the memeory.total is constantly increasing over the time and after a couple of days it goes beyond the configured memory limit of the docker container. As far as we can see this doesn't cause Openshift to kill the container immediately but sooner or later it happens.
We increased the memory limit of all our containers and this seems to help since Openshift is not killing our containers that often anymore. However we still see in Grafana that the memeory.total is exceeding the configures memory limit of our containers significantly after a couple of days (rss memory is fine).
To better understand Openshifts OOM killer, does anybody know which memory metric Openshift takes into account to decide if a container has to be killed or not? Is the configured container memory limit related to the memory.usage or the memory.total_rss or something completely different?
Thanks for help in advance.
I need to deploy multiple instances of the same spring application.
I'm actually using docker as a container for my instances, and have got 1 container for each instance.
However noticed that one container consume up to 500mb, which is much for me, as i need my VPS to catch up as many instances as possible, and honestly, the containers are used just for the JVM and nothing else, would be a great waste of memory to dedicate a whole virtual environnement for a simple JVM instance and i don't really need that at all.
What i need is more like a tool to help me to administrate the instances (auto start if the processes are killed for some reasons, update easily the instances if a new version of the app has been developped, managing the instances through line commands for debugging or whatever.) just as i can do with Docker but without a whole virtualisation environnement which is too " greedy "
By the way, i've got a VPS from OVH Cloud Services, who doesn't provide any kind of spring like deployment services. I'm working on a traditionnal Ubuntu 18.04.
Thanks in advance
You should approach this from the bottom to the top of the stack. I'll try to answer each topic but I'd advise you to divide this into several questions, so people can give more detailed answer to each problem.
I'll give you just an overview so you can have a starting point.
JVM Memory Limit
First, control the memory allocation by setting coherent limits to the JVM. You can do so by setting the Max Heap Size, and the Max Metaspace size (Java 8).
You can set the max heap size by appending the flag -Xmx. -Xmx1G will limit the heap size to 1 Gigabyte.
The metaspace can be set using -MaxMetaspaceSize. The metaspace was introduced in Java 8. If you are using Java 6 or 7, you should take a look at PermGen.
Container Limit
You can also set the max memory of your container using appending the -m flag to the docker run command. You can take a look at docker's doc here: https://docs.docker.com/config/containers/resource_constraints/
Container Management
This deserves its own detailed answer. First of all, Spring Framework it is not related to what you are trying to achieve. It won't solve automatic restarts or anything like that.
What you are looking for is a Container Management Tool. You can take a look at docker swarm, portainer, or kubernetes.
They'll allow you start multiple instances of the same service without changing anything at code level.
For a quick and dirty implementation, you can use docker swarm, which is a no-brainer and integrates seamlessly with docker containers.
I was playing around with the Java memory heap allocation and I think I did something that set it not just for any specific tomcat folder but for the entire system as I can't run my application in Spring or with a custom tomcat folder that I had been also using for testing. When I try to run the application, it just hangs at "INFO: Initializing Spring root WebApplicationContext".
I am not fully clear on how I could have set this on the Linux command line by doing something like "export CATALINA_OPTS="-Xms2000m -Xmx4500m" or "JAVA_OPTS="-Xms2000m -Xmx4500m -XX:MaxPermSize=4500m"
I think I accidentally put the settings I had meant for my cloud server (which has a higher amount of memory) so I gave it more memory allocation than my entire system has.
I would appreciate any help if anyone can let me know if it is possible I did this and if this could be causing the issue and how I can get an output from my system as to the current allocation so I can check / change it, I have tried "export CATALINA_OPTS="-Xms2000m -Xmx4500m" but it still won't work for me.
I would like to restore everything to default settings. I normally set the memory heap allocation in the /bin/startup.sh file in the tomcat folder but I had forgotten this, I think, and was experimenting with things on the command line.
Thanks
Check all of the log files for Tomcat. It's possible that the startup of the Spring WebApplicationContext is encountering errors, but logging it elsewhere.
Setting the max heap size too large for Tomcat sounds like a red herring to me. If the initial heap size argument is larger than the maximum size of memory on the system, the Java process will fail immediately - Tomcat will not get to the point of initializing your Spring application.
See if you can increase the server timeout if you are running it from an IDE. From command line, you'll have to wait.
If the problem persists, try cleaning /reloading the application
Isn't finding bugs what debuggers are for? Why not simply start your Tomcat in debug mode, wait until it "hangs", and find out what it is doing by suspending the relevant thread and looking at the stack trace?
Or even simpler, do a "thread dump" once the app hangs?
For me it was another service that was not reachable
spring was retrying again and again to reach and wating until it timeouts
Once I started another service it worked automatically.
Tomcat 5.5.x and 6.0.x
Grails 1.6.x
Java 1.6.x
OS CentOS 5.x (64bit)
VPS Server with memory as 384M
JAVA_OPTS : tried many combinations- including the following
export JAVA_OPTS='-Xms128M -Xmx512M -XX:MaxPermSize=1024m'
export JAVA_OPTS='-server -Xms128M -Xmx128M -XX:MaxPermSize=256M'
(As advised by http://www.grails.org/Deployment)
I have created a blank Grails application i.e simply by giving the command grails create-app and then WARed it
I am running Tomcat on a VPS Server
When I simply start the Tomcat server, with no apps deployed, the free memory is about 236M
and used memory is about 156M
When I deploy my "blank" application, the memory consumption spikes to 360M and finally the Tomcat instance is killed as soon as it takes up all free memory
As you have seen, my app is as light as it can be.
Not sure why the memory consumption is as high it is.
I am actually troubleshooting a real application, but have narrowed down to this scenario which is easier to share and explain.
UPDATE
I tested the same "blank" application on my local Tomcat 5.5.x on Windows and it worked fine
The memory consumption of the Java process shot from 32 M to 107M. But it did not crash and it remained under acceptable limits
So the hunt for answer continues... I wonder if something is wrong about my Linux box. Not sure what though...
UPDATE 2
Also see this http://www.grails.org/Grails+Test+On+Virtual+Server
It confirms my belief that my simple-blank app should work on my configuration.
It is a false economy to try to run a long running Java-based application in the minimal possible memory. The garbage collector, and hence the application will run much more efficiently if it has plenty of regular heap memory. Give an application too little heap and it will spend too much time garbage collecting.
(This may seem a bit counter-intuitive, but trust me: the effect is predictable in theory and observable in practice.)
EDIT
In practical terms, I'd suggest the following approach:
Start by running Tomcat + Grails with as much memory as you can possibly give it so that you have something that runs. (Set the permgen size to the default ... unless you have clear evidence that Tomcat + Grails are exhausting permgen.)
Run the app for a bit to get it to a steady state and figure out what its average working set is. You should be able to figure that out from a memory profiler, or by examining the GC logging.
Then set the Java heap size to be (say) twice the measured working set size or more. (This is the point I was trying to make above.)
Actually, there is another possible cause for your problems. Even though you are telling Java to use heaps of a given size, it may be that it is unable to do this. When the JVM requests memory from the OS, there are a couple of situations where the OS will refuse.
If the machine (real or virtual) that you are running the OS does not have any more unallocated "real" memory, and the OS's swap space is fully allocated, it will have to refuse requests for more memory.
It is also possible (though unlikely) that per-process memory limits are in force. That would cause the OS to refuse requests beyond that limit.
Finally, note that Java uses more virtual memory that can be accounted for by simply adding the stack, heap and permgen numbers together. There is the memory used by the executable + DLLs, memory used for I/O buffers, and possibly other stuff.
384MB is pretty small. I'm running a small Grails app in a 512MB VPS at enjoyvps.net (not affiliated in any way, just a happy customer) and it's been running for months at just under 200MB. I'm running a 32-bit Linux and JDK though, no sense wasting all that memory in 64-bit pointers if you don't have access to much memory anyway.
Can you try deploying a tomcat monitoring webapp e.g. psiprobe and see where the memory is being used?