When I start elastic search, I see "Killed" on the console and the process ends. I am unable to get the elastic search process to start. What am I missing?
:~/elasticsearch-5.5.2/bin$ ./elasticsearch
Killed
If it is relevant, I am installing this on a VPS. I don't see any other error message - making it hard to debug.
As of configuring RAM usage by Elasticsearch
Find the elasticsearch jvm.options file location, by default it is /etc/elasticsearch/jvm.options
Set the -Xms and -Xmx options in there to reflect the RAM amout available at your VPS as described here
By default, elasticsearch tries to occupy 1Gb of RAM at start, so if your VPS has less than 1Gb of RAM you need to configure elasticsearch to use less RAM accordingly
As an alternative to above file configuration you can try export of corresponding environment variable
export ES_JAVA_OPTS="-Xms256m -Xmx256m"
and then check if it helps
./elasticsearch
Regarding the exit state
Killed
It most frequently indicates the OoM Killer process activity which is aimed on emergent RAM freeing to let Linux survive the lack of available RAM event. OoM Killer, as per his name, sends kill signal to some most memory-consuming user process.
As of VPS and it's virtualization model, there are some custom container-based OoM settings in effect (check out example for OpenVZ), so if you're 100% sure you're configured ealsticsearch correctly, and there is enough RAM to start its instance - contact your VPS provider to clarify possible limits (like 10% of RAM must always be free or OoM Killer is triggered otherwise)
Some debug approach to OoM Killer events is described in this answer
Related
I have ran nohup java -jar sample.jar & on my linux server as a backend service of web application but service is automatically killed after few hours and need to re-run the command.
I want to know what is the reason and how to resolve the issue
On Linux, one possibility is that it is the oomkiller process. This watches for processes that appears to be triggering excessive paging activity, and sends them a SIGKILL signal. The oom killer's purpose is to prevent the paging system from thrashing ... which can bring the entire system to its knees.
If it is the oom killer killing your Java app, you should see a log message in /var/log/messages or /var/log/syslog.
The solution may be as simple as reducing the Java app's heap size so that it all fits in the available physical memory. Alternatively, if your Java app's heap usage is growing all of the time, you may need to look for a possible memory leak in your app. (Search for articles on finding Java memory leaks ...)
I run Spring Boot via the same command and only a handful of times have I seen this behavior. It turned out that the JVM ran out of memory and it crashed. When I changed the heap size settings it was rock solid again.
I'm trying to troubleshoot a Java program that requires increasingly more memory until it cannot allocate any more and then it crashes.
EDIT More information about the program. The program is an indexer going through thousands of documents and indexing them for search. The documents are read from MongoDB and written to MongoDB as well after some processing is performed. During the processing I'm using RocksDB (rocksdb-jni version 5.13.4 from Maven). There is some mentioning in this GitHub issue of RocksDB memory usage growing uncontrollably, but I'm not sure it could be related.
Monitoring the process with visualvm results in the following plot:
but running htop on the machine shows totally different stats:
There is a difference of several GBs of memory that I'm unable to trace the source of.
The program is launched with the following VM arguments:
jvm_args: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=<port> -Djava.rmi.server.hostname=<hostname> -Xmx12G -XX:+UseStringDeduplication
The system has 32GB of RAM and no swap. Out of this 32 GB, ~10GB are always taken by a tmpfs partition, ~8GB by MongoDB and the remaining 12GB are assigned to the program. EDIT The visualvm screenshot above shows 20GB of heap size, because it was from a previous run where I passed -Xmx20G; the behaviour, however, is the same whether I assign 12GB or 20GB to the heap. The behaviour also does not change if I remove the tmpfs partition, freeing 10 more GB of memory: it just takes longer but eventually it will get out of memory.
I have no idea where this memory usage that is not shown in visualvm but appears in htop is coming from. What tools should I use to understand what is going on? The application is running on a remote server, so I would like a tool that only works in the console or can be configured to work remotely, like visualvm.
I always use JProfiler but i hear jetbrains has a good one as well both can connect remotely.
If possible i would try to create a (local) setup where you can freely test it.
in the RocksDB several possible solutions are mentioned, do they work?
RocksDB seems to need some configuration, how did you configure it?
I am not familiar with RocksDB but i see there is some indexing and caching going on. How much data are you processing and what indexes/caching configuration do you have? are you sure this should fit in memory?
as far as i know the memory mismatch is because jni usage is not shown by default by most. There are some flags to improve this NativeMemoryTracking. i can't recall if this will add them to your visualvm overviews as well.
karbos-538 i just noticed this is quite an old issue, i hope this application is already working by now. what part of the question is still relevant to you?
Running a Java microservice in an environment you have to make/decide some configuration.
The configuration that I refer now is how to set the Xmx flag.
I consider that any type of instance you choose, you have to leave some RAM for the system and other programs.
I have a t2.medium (4 GB) AWS EB and I configured the Xmx flag for Java microservice to 3 GB.
After the load, the committed memory reaches a bit above 3.0 GB, around 3.1 GB. There are a lot of free memory. The problem is that AWS EB reports 97% RAM, this means that left memory is used for CloudWatch Agent, NGINX, Linux and maybe for other things.
If the values reaches or goes beyond 100%, something can be killed by Linux Kernel due to high usage of the RAM.
Question: Do you have any idea, the best practices in choosing the value for Xmx flag based on total available RAM of the machine? Do you decide it by doing experiments?
Thank you
There is no formula for tuning JVM. It is highly dependent on each specific machine, the application, application server, and usage pattern by end users. It is recommended that you set the same value for Xmx and Xms. You should try different values and profile JVM using JProfiler or Visual VM. This is an article for Liferay JVM tuning. Hopefully, it is useful and brings you some ideas.
Since moving to Tomcat8/Java8, now and then the Tomcat server is OOM-killed. OOM = Out-of-memory kill by the Linux kernel.
How can I prevent the Tomcat server be OOM-killed?
Can this be the result of a memory leak? I guess I would get a normal Out-of-memory message, but no OOM-kill. Correct?
Should I change settings in the HEAP size?
Should I change settings for the MetaSpace size?
Knowing which Tomcat process is killed, how to retrieve info so that I can reconfigure the Tomcat server?
Firstly check that the oomkill isn't being triggered by another process in the system, or that the server isn't overloaded with other processes. It could be that Tomcat is being unfairly targeted by oomkill when some other greedy process is the culprit.
Heap should be set as a maximum size (-Xmx) to be smaller than the physical RAM on the server. If it is more than this, then paging will cause desperately poor performance when garbage collecting.
If it's caused by the metaspace growing in an unbounded fashion, then you need to find out why that is happening. Simply setting the maximum size of metaspace will cause an outofmemory error once you reach the limit you've set. And raising the limit will be pointless, because eventually you'll hit any higher limit you set.
Run your application and before it crashes (not easy of course but you'll need to judge it), kill -3 the tomcat process. Then analyse the heap and try to find out why metaspace is growing big. It's usually caused by dynamically loading classes. Is this something your application is doing? More likely, it's some framework doing this. (Nb oom killer will kill -9 the tomcat process, and you won't be able to diagnostics after that, so you need to let the app run and intervene before this happens).
Check out also this question - there's an intriguing answer which claims an obscure fix to an XML binding setting cleared the problem (highly questionable but may be worth a try) java8 "java.lang.OutOfMemoryError: Metaspace"
Another very good solution is transforming your application to a Spring Boot JAR (Docker) application. Normally this application has a lot less memory consumption.
So steps the get huge improvements (if you can move to Spring Boot application):
Migrate to Spring Boot application. In my case, this took 3 simple actions.
Use a light-weight base image. See below.
VERY IMPORTANT - use the Java memory balancing options. See the last line of the Dockerfile below. This reduced my running container RAM usage from over 650MB to ONLY 240MB. Running smoothly. So, SAVING over 400MB on 650MB!!
This is my Dockerfile:
FROM openjdk:8-jdk-alpine
ENV JAVA_APP_JAR your.jar
ENV AB_OFF true
EXPOSE 8080
ADD target/$JAVA_APP_JAR /deployments/
CMD ["java","-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-jar","/deployments/your.jar"]
Recently came across and interesting scenario with Cloudera Hadoop and HDFS where we were unable to start our NameNode Service.
When attempting a restart of HDFS Services we were unable successfully restart NameNode Service in our cluster. Upon review of the logs, we did not observe any ERRORs but did see a few entries related to JvmPauseMonitor...
org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5015ms
We were observing these entries in the /var/log/hadoop-hdfs/NAMENODE.log.out and were not seeing any other errors including /var/log/messages.
CHECK YOUR JAVA HEAP SIZES
Ultimately, we were able to determine that we were running into a Java OOM Exception that wasn't being logged.
From a performance perspective as a general rule for every 1 Million Blocks in HDFS you should have configured at least 1GB of Java Heap Size.
In our case, the resolution was as simple as increasing the Java Heap Size for the NameNode and Secondary NameNode Services and Restarting... as we had grown to 1.5 Million Blocks but were only using the default 1GB setting for the java heap size.
After increasing the Java Heap Size to at least 2GB and restarting the HDFS Services we were green across the board.
Cheers!