I received OutOfMemoryError: PermGen space error.
I run tomcat using command:
/usr/local/etc/rc.d/tomcat7 start
I am trying to increase PermGen space (JAVA_OPTS="-XX:PermSize=256m -XX:MaxPermSize=256m").
I tried to add this string to next places:
catalina.sh
startup.sh
setenv.sh
tomcat7(/usr/local/etc/rc.d/)
But it is no effect. Default Tomcat server status (/manager/status) shows that only 82MB allowed for PS Perm Gen.
What I did wrong?
OS Name: FreeBSD
OS Version: 9.1-STABLE
# java -version
openjdk version "1.7.0_21"
OpenJDK Runtime Environment (build 1.7.0_21-b11)
OpenJDK 64-Bit Server VM (build 23.21-b01, mixed mode)
What helped me:
In the tomcat7(/usr/local/etc/rc.d/):
tomcat7_java_opts="-XX:PermSize=256m -XX:MaxPermSize=256m"
I suspect that the server process on the machine is not picking up right Java OPTS.
Please try to find the process on the server and check the parameters being passed.
In Linux, its usually be
ps -ef | grep tomcat
or ps -ef | grep java
to find out the process and verify the JVM parameters.
Edit 1: the is a sample output of the command to find process, which might indicate the java opt parameter values:
local-vm-1 [5]:ps -ef | grep tomcat
tomcat 4141 1 0 07:38 ? 00:01:33 /apps/mw/jdk/1.6.0.17-64bit/bin/java -Dnop -Xms1024m -Xmx1024m -server -DTC=testplatform -DWMC_ENV
=test -XX:MaxNewSize=112m -XX:NewSize=112m -XX:SurvivorRatio=6 -XX:PermSize=256m -XX:MaxPermSize=256m -Dsun.net.inetaddr.ttl=0 -DLISTEN_ADDRESS=wsx
-test-vm-dtcp-1.managed.com -Djavax.net.ssl.trustStore=/tech/tomcat/props/ldapstore.ts -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxre
mote.host=wsx-test-vm-dtcp-1 -Dcom.sun.management.jmxremote.port=25000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.auth
enticate=true -Dcom.sun.management.jmxremote.password.file=/tech/tomcat/props/jr-password.properties -Dcom.sun.management.jmxremote.access.file=/tec
h/tomcat/props/jr-access.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/tech/tomcat/apache-tomca
t-7.0.16/endorsed -classpath /tech/tomcat/apache-tomcat-7.0.16/bin/bootstrap.jar:/tech/tomcat/apache-tomcat-7.0.16/bin/tomcat-juli.jar -Dcatalina.ba
se=/tech/tomcat/instances/testplatform -Dcatalina.home=/tech/tomcat/apache-tomcat-7.0.16 -Djava.io.tmpdir=/tech/tomcat/instances/testplatform/te
mp org.apache.catalina.startup.Bootstrap start
Related
Here I have a command line to start a jarfile.
And it is correct and was able to start my application.jar file.
nohup java -jar -Xms512M -Xmx512M -Xmn300M -XX:SurvivorRatio=6 -Dspring.profiles.active=test /data/code/application.jar >> /data/code/logs/application.log &
When I tried to add metaspace parameters ‐XX:MetaspaceSize=256M ‐XX:MaxMetaspaceSize=256M into my start-up-command, which is showing below:
nohup java -jar -Xms512M -Xmx512M -Xmn300M -XX:SurvivorRatio=6 ‐XX:MetaspaceSize=256M ‐XX:MaxMetaspaceSize=256M -Dspring.profiles.active=test /data/code/application.jar >> /data/code/logs/application.log &
the JVM crashed with logError: Unable to access jarfile ‐XX:MetaspaceSize=256M.
Is there any thing that I missed? It seems that JVM not recognizing metaspace parameters in commandline, and treated it as an illegal jarfile.
The jdk version by java -version is showing below:
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
My bad, copy from google but never check the words,- was a wrong word(not in English) which leads to the error. Just type it manually again in English, the problem solved.....
We are seeing an issue were thread dumps are being generated every 2-3 seconds for java process where process does not crash or restart. Java process is working fine. This java process is running on docker container where datadog agent is also running to fetch matrices.
We see that JMXFetch is spawned by datadog process like below:
root 43334 43252 1 08:50 ? 00:03:35 java -Xmx200m -Xms50m -classpath /opt/datadog-agent/bin/agent/dist/jmx/jmxfetch.jar org.datadog.jmxfetch.App --ipc_host localhost --ipc_port 5001 --check_period 15000 --thread_pool_size 3 --collection_timeout 60 --reconnection_timeout 10 --reconnection_thread_pool_size 3 --log_level INFO --reporter statsd:localhost:8125 collect
Our process looks like:
ubuntu 18067 17637 0 Mar08 ? 00:04:37 java -XX:+UseParallelGC -XX:NativeMemoryTracking=summary -Xss256K -Dorg.xerial.snappy.tempdir=/home/ubuntu/tmp -Dlog4j2.debug=DEBUG -Dlogdir=/var/log/abclog -Dlog4j.configurationFile=/home/ubuntu/build-target/latency-metrics/latency-metrics.log4j2.xml -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.password.file=/home/ubuntu/build-target/latency-metrics/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/home/ubuntu/build-target/latency-metrics/jmxremote.access -Djava.rmi.server.hostname=10.79.199.11 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=11110 -Dcom.sun.management.jmxremote.password.toHashes=false -Xmx128m -Dreporters.configuration=/home/ubuntu/build-target/latency-metrics/reporters.configuration -Dreporters.application.name=App -Ddisable.task.reporters=true -Ddatadog.host=10.19.129.111 -Ddatadog.tags=env:APP-NEW-DEV -DApp.metrics.host=machine.10.79.199.11 -Dpostgres.configuration=/home/ubuntu/build-target/latency-metrics/postgres.configuration -Ddynamodb.configuration=/home/ubuntu/build-target/latency-metrics/dynamodb.configuration -Delasticsearch.configuration=/home/ubuntu/build-target/latency-metrics/elasticsearch.configuration -Ddpconfig.base.folder=/home/ubuntu/build-target/latency-metrics -Ddp.operational.config.base.folder=/home/ubuntu/build-target/latency-metrics -Dtaskmgr.spec.folder=/home/ubuntu/build-target/latency-metrics -DOvaParamFile=/etc/app/ova.params -Dcustomer.configuration=/home/ubuntu/build-target/latency-metrics/customer.configuration -Dpolicy.config.path=/home/ubuntu/build-target/latency-metrics/policy -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/heap-dumps/latency-metrics -XX:+ExitOnOutOfMemoryError --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-modules jdk.unsupported -Dio.netty.tryReflectionSetAccessible=true --illegal-access=warn -Djava.security.properties==/home/ubuntu/build-target/latency-metrics/java.security.fips -Dorg.bouncycastle.jca.enable_jks=true -cp /home/ubuntu/build-target/latency-metrics/App-0.001-SNAPSHOT.jar com.app.latency.Main
When we kill the datadog-agent (which kills JMXFetch too) the thread dumps stop.
When we spawn our docker container with "--privileged" flag, we dont see these dumps.
We are using OpenJDK11.
openjdk 11.0.13 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+14-SA (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.52+14-SA (build 11.0.13+8-LTS, mixed mode)
Also, same changes are working fine on VM, we are seeing this issue only on docker container. Can someone help?
I'm attempting to use spacemacs, elipse and eclimd for editing with autocompletion. However, eclimd is attempting to start the java server with the -command ping option. My version of Java does not support that option. I have been unable to find this option in the documentation. Is it deprecated? Which version of Java supports it?
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
/usr/lib/jvm/java-8-oracle/bin/java -d64 -command ping -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=#user.home/eclipse-workspace -XX:+UseG1GC -Dosgi.dataAreaRequiresExplicitInit=true -Xms256m -Xmx1024m -jar /snap/eclipse/current/plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar --launcher.suppressErrors -debug -clean -refresh -application org.eclim.application
Unrecognized option: -command
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
No version of Java supports -command.
That option is supposed to be an argument for eclim, not for java
eclim -command ping
or presumably something like
java -d64 -Dosgi.requiredJavaVersion=1.8 -Dosgi.instance.area.default=#user.home/eclipse-workspace -XX:+UseG1GC -Dosgi.dataAreaRequiresExplicitInit=true -Xms256m -Xmx1024m -jar /snap/eclipse/current/plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar --launcher.suppressErrors -debug -clean -refresh -application org.eclim.application -command ping
Either you've configured something wrong, or something has a bug.
There is no -cmd.
But maybe you are talking about a new feature in Java 11. You can now do:
java HelloWorld.java
Meaning: when you have a "single file class", with a main method, you can sidestep invoking javac first, to then run java with the class name. Instead, you can do the above and directly "run+compile" a .java file.
That feature was added with JEP 330.
I start and attach a docker container, but nothing working after setting JDK.
Docker container didn’t work after the command.
ken#namenode:~$ docker start peaceful_babbage
peaceful_babbage
ken#namenode:~$ docker attach peaceful_babbage
root#1477050f192a:/#
root#1477050f192a:/# java -version
bash: java: command not found
root#1477050f192a:/# source /etc/profile
root#1477050f192a:/# echo $PATH
/usr/lib/jdk1.7.0_60/bin:/usr/lib/jdk1.7.0_60/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root#1477050f192a:/# java -version
java version "1.7.0_60"
Java™ SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot™ 64-Bit Server VM (build 24.60-b09, mixed mode)
root#1477050f192a:/#
root#1477050f192a:/# java
Usage: java [-options] class [args…]
(to execute a class)
or java [-options] -jar jarfile [args…]
(to execute a jar file)
where options include:
-d32 use a 32-bit data model if available
-d64 use a 64-bit data model if available
-server to select the “server” VM
The default VM is server.
...
See http://www.oracle.com/technetwork/java/javase/documentation/index.html for more details.
^C
At this moment, I can’t do anything with this container. can’t kill the Java process either. Who can help me with this?
You can simply stop the container from the host :
docker stop peaceful_babbage
I’m using JBoss 7.1.3 with the following Java, Linux versions …
[dave#mymachine ~]$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
[dave#mymachine ~]$ uname -a
Linux mymachine.mydomain.org 4.1.99-99.88.amzn1.x86_64 #1 SMP Fri Feb 5 23:44:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I want to figure out how much heap is being used (not the maximum amount, but what is currently being used). Sadly, on our machine I don’t have access to jmap or jstat. So I wanted to do a heap dump using “kill -3”. So I first got the pid of my boss process
[dave#mymachine ~]$ ps -elf | grep jboss
1 S root 1251 1 0 80 0 - 28870 - 07:03 ? 00:00:00 /bin/sh /etc/rc3.d/S84jboss start
4 S root 1254 1251 0 80 0 - 34424 - 07:03 ? 00:00:00 runuser -s /bin/bash jboss -c ulimit -S -c 0 >/dev/null 2>&1 ; LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=/var/run/jboss-as/jbos-as-standalone.pid /usr/java/jboss/bin/standalone.sh -c standalone.xml
4 S jboss 1255 1254 0 80 0 - 28271 - 07:03 ? 00:00:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; LAUNCH_JBOSS_IN_BACKGROUND=1 JBOSS_PIDFILE=/var/run/jboss-as/jboss-as-standalone.pid /usr/java/jboss/bin/standalone.sh -c standalone.xml
0 S jboss 1256 1255 0 80 0 - 28272 - 07:03 ? 00:00:00 /bin/sh /usr/java/jboss/bin/standalone.sh -c standalone.xml
0 S jboss 1289 1256 3 80 0 - 603908 - 07:03 ? 00:18:26 /usr/java/default/bin/java -D[Standalone] -server -XX:+UseCompressedOops -Xms256m -Xmx1024m -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Dorg.jboss.as.logging.per-deployment=false -Dorg.jboss.boot.log.file=/usr/java/jboss/standalone/log/boot.log -Dlogging.configuration=file:/usr/java/jboss/standalone/configuration/logging.properties -jar /usr/java/jboss/jboss-modules.jar -mp /usr/java/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/java/jboss -Djbos.server.base.dir=/usr/java/jboss/standalone -c standalone.xml
0 S 602 21746 20632 0 80 0 - 27615 pipe_w 16:46 pts/1 00:00:00 grep --color=auto jboss
and then I ran “kill -3” …
[dave#mymachine ~]$ sudo kill -3 1289 1256
but nothing prints out and no file is generated in the current directory. How do I generate a heap dump with my current constraints?
I suspect that the main reason that you do not have access to the excellent serviceability tools like jmap, jstack etc. is that they are not part of the openjdk-6-jre, or openjdk-7-jre Linux packages (I am assuming debian distro here). I do not understand why that should be the case. In my opinion, these tools should be a part of the JRE and not the JDK. Here's the proof:
$> apt-file list openjdk-6-jdk | grep jmap
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/bin/jmap
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/man/ja_JP.eucJP/man1/jmap.1.gz
openjdk-6-jdk: /usr/lib/jvm/java-6-openjdk-amd64/man/man1/jmap.1.gz
$> apt-file list openjdk-6-jre | grep jmap
<nothing>
Anyway, to get these tools, you should consider using the JDK and install them using sudo apt-get install openjdk-6-jdk. This should get you not only the latest JDK 6 build (which, btw, very very old). Then, you should get the access to the tools that give us the heap dump for later analysis.
Heap dump is very different from the thread dump which shows you what the JVM threads are doing. One way to get the thread dump is by sending the QUIT signal to JVM process (using, like you said: kill -3). This will show the thread dump wherever the JVM's stdout is redirected. An (almost) equivalent tool is jstack, which is, again, a part of the JDK package, and not the JRE package.
But thread dump is not what you need. You need heap dump. So, you need jmap.
Short of this, you can estimate the actual memory being used by the JVM using the indispensable Linux top command. Make sure that you analyze the output correctly to get an estimate of how much heap is being used by the JVM.
If you are experiencing an OutOfMemoryError in your JVM, then you should consider starting it with -XX:+HeapDumpOnOutOfMemoryError so that when OOME is thrown, at least you can get a heap dump. But this way, you can get the heap dump only after you run out of heap space!
set -XX:HeapDumpPath=mpath/heapdump to a writable directory or file on java's command line for starting up JBOSS.
This path/file needs to be writable for the user JBOSS is running under and then try the kill -3 option again.
http://five.agency/java-heap-dump/
You could also problematically do a heap dump using the HotSpotDiagnosticMXBean from your application also.
see class to dump heap