We Keep Coding

java vm options and AWS::ECS::TaskDefinition

I would like to run my docker container as following:
docker run java \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseCGroupMemoryLimitForHeap
Where should I pass vm arguments when writing AWS::ECS::TaskDefinition in cloudformation stack?

I have done the same thing at work, you could pass the flags directly in Dockerfile as per below.
ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

VisualVM - Cannot take heap dump

When I click on the Heap Dump button in VisualVM 1.3.8, I get the following error:
Cannot take heap dump for user#localhost:9090
Am I missing a setting somewhere?
Edit #1
Environment is:
RHEL6
Tomcat 7.0.68
Java 1.7.0_45
Options are:
export CATALINA_OPTS="-Dcom.sun.management.jmxremote=true\
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=true \
-Djava.rmi.server.hostname=x.x.x.x \
-Dcom.sun.management.jmxremote.password.file=/file_to_pwd \
-Dcom.sun.management.jmxremote.access.file=/file_to_access \
-Xms1256m \
-Xmx1256m \
-XX:PermSize=768m \
-XX:MaxPermSize=768m \
-XX:+CMSClassUnloadingEnabled \
-Dfile.encoding=UTF-8 \
-XX:+CMSClassUnloadingEnabled \
-XX:+UseConcMarkSweepGC \
-server"
Edit #2
Here is the listener for JMX.
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="9090" rmiServerPortPlatform="9091" />

Here's what I ended up doing. We had JDK 1.8 also present on the server.
cd /path/to/java/jdk1.8.0_65/
./jmap -dump:format=b,file=/path/to/dump/tomcat_dump.bin PID
Open /path/to/dump/tomcat_dump.bin using VisualVM (I had to use Filezilla to bring it to my workstation).

One possible reason is permission issue. In Windows right click on the jvisualvm executable and run as administrator solves this issue.

Puppetserver service fails to start

I have a Vagrant CentOS VM running with ps.memory = 2048 RAM allocated.
When I try to start the puppetserver service:
$ puppet --version
4.4.0
$ sudo puppet resource service puppetserver ensure=running
Error: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
Error: /Service[puppetserver]/ensure: change from stopped to running failed: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
service { 'puppetserver':
ensure => 'stopped',
}
$ journalctl -xn
No journal files were found.
$ systemctl status puppetserver.service
puppetserver.service - puppetserver Service
Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; disabled)
Process: 4708 ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver (code=exited, status=0/SUCCESS)
Main PID: 4709 (java); : 4710 (bash)
CGroup: /system.slice/puppetserver.service
├─4709 /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/...
└─control
├─4710 /bin/bash /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh wait_for_app
└─4755 sleep 1
My JAVA_ARGS from /etc/sysconfig/puppetserver:
JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=1g"
As requested, the puppetserver.service file:
$ cat /usr/lib/systemd/system/puppetserver.service
[Unit]
Description=puppetserver Service
After=syslog.target network.target
[Service]
Type=simple
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PermissionsStartOnly=true
ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver
ExecStart=/usr/bin/java $JAVA_ARGS \
'-XX:OnOutOfMemoryError=kill -9 %%p' \
-Djava.security.egd=/dev/urandom \
-cp "${INSTALL_DIR}/puppet-server-release.jar" clojure.main \
-m puppetlabs.trapperkeeper.main \
--config "${CONFIG}" \
-b "${BOOTSTRAP_CONFIG}" $#
KillMode=process
ExecStartPost=/bin/bash "${INSTALL_DIR}/ezbake-functions.sh" wait_for_app
SuccessExitStatus=143
StandardOutput=syslog
[Install]
WantedBy=multi-user.target
An attempt at running the ExecStartPost command by hand:
$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available
So I tried again, but this time I changed some directory permissions, but still similar error (which doesn't make sense given I just changed the mode?):
$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/
$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
RuntimeError: Got 1 failure(s) while initializing: File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available
use at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1007
What could be the issue?

Are you certain that it’s an OnOutOfMemory error? I ask because I found that the latest PuppetServer includes a newer version of logback, as shown by this message in /var/log/messages:
Mar 18 01:56:21 puppetserver java: Exception in thread "main" java.lang.AbstractMethodError: ch.qos.logback.core.net.SyslogAppenderBase.createOutputStream()Lch/qos/logback/core/net/SyslogOutputStream;
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:62)
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.classic.net.SyslogAppender.start(SyslogAppender.java:48)
If you see, this replace “classic.net.Syslog” in logback.xml with "core.net.Syslog"
sed -i_old -e 's/classic.net.Syslog/core.net.Syslog/' /etc/puppetlabs/puppetserver/logback.xml
If that's not the problem, please post your logfiles.

You have provided log as
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g;
support was removed in 8.0
So, it is clear that you are using jdk 8 which removed permgen space.
The Permanent Generation (PermGen) space has completely been removed and is kind of replaced by a new space called Metaspace. The consequences of the PermGen removal is that obviously the PermSize and MaxPermSize JVM arguments are ignored and you will never get a java.lang.OutOfMemoryError: PermGen error.
So please remove the -XX:MaxPermSize=1g portion from JAVA_ARGS of location /etc/sysconfig/puppetserver
JAVA_ARGS="-Xms1g -Xmx1g"
So
And then,
your command will be look like below:
$ /usr/bin/java -Xms1g -Xmx1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
or
$ /usr/bin/java -Xms1g -Xmx1g -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
Please try this 2 commands. I hope both will run successfully. For caution purpose, I have added both.
N.B: There is no need to change the file permission. keep them as before you using.
Otherwise, if you don't want to change anything, you can downgrade java 8 to java 7
Related Link:
PermGen elimination in JDK 8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize
JAVA 8 XX:PERMSIZE AND XX:MAXPERMSIZE DISAPPEARING
JDK 8 Milestones
JEP 122: Remove the Permanent Generation

#lollercoster, you're starting your service with:
sudo puppet resource service puppetserver ensure=running
But your next command is not telling which user it is
/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
Please run whoami directly before your run the above command:
whoami
/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg
Why this commands?
I am pretty confident that your issue is a permission/usercontext issue and following commands make it more worse:
$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/
with the above cases you have to do a sudo on the right user context to have the files writable, but it depends if the other data is writable. So you can give it a try, but I would use the puppet user created for the service
User=puppet
So your puppet service runs as user puppet, but your logfiles are under vagrant user control, and not writable for puppet user.
So I would also try to switch back to:
$ sudo chown -R puppet /var/run/puppetlabs/
$ sudo chown -R puppet /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/
Java
Also I suggest you NOT to mess around with the JVM heap parameters, except you know what you do. Since you do not need to enforce a lower and a max bound on Java VM heap since JDK5. I would only limit to the upper bound
-Xmx1g
The heap and the real allocated mem will be managed well by the javaVM. As JVM needs more it will step by step increase the heap and keep the needed size (no downsizing to a lower value will occur). So you will have a better used RAM on your machine.
Also please switch to the Oracle JVM. I am facing often issues with security and Compatibility with OpenJDK. So my first test is to run it on the latest needed JDK of Oracle. In your case Oracle JDK8.
But please try first the permission things I mentioned before.
Good luck, and please keep me updated.

I have seen this behaviour before. Do not try to run the commands manually as root, because you will mess up the permissions. Instead read the logs carefully with journalctl -xe and try to understand why the service is failing. In my case I could not start the puppetserver service because there was already a private key, but not yet a public certificate.
ls /etc/puppetlabs/puppet/ssl/certs/`hostname -f`.pem
ls /etc/puppetlabs/puppet/ssl/private_keys/`hostname -f`.pem
If one of them is present without the other one, the service will fail to start up. You will see something like this in the log file:
Exception in thread "main" java.lang.IllegalStateException: Cannot initialize master with partial state; need all files or none.
If you are trying to install puppetserver for the first time, so there is no way to break an existing deployment, then you can probably just remove the exisiting private key and puppetserver will automatically generate a new pair.
rm /etc/puppetlabs/puppet/ssl/certs/`hostname -f`.pem
rm /etc/puppetlabs/puppet/ssl/private_keys/`hostname -f`.pem
And finally try again to start the puppetserver service.
service puppetserver start

/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g
Xms and Xms are less than MaxPermSize. this worked for me.

SSH command can not start remote java process

I have a start function, and I put it in a script resides on a remote site, the function's code shows below.
function start() {
cd $install_dir
mkdir -p logs
export classpath=$classpath:$target_jar
nohup java -Xms2048m -Xmx8192m -server -XX:PermSize=128m -XX:MaxPermSize=256m \
-XX:+PrintGCDetails -XX:+PrintGCDateStamps \
-XX:-OmitStackTraceInFastThrow \
-cp $target_jar $main_class >> logs/jvm.log 2>&1 &
echo "Service started, see logs"
}
And when I try to call that function use ssh ssh xxx#host "./service.sh start", I can not start the java process, I only got the response message "Service started, see logs" and there's no error, the jvm.log is also empty. It apparently to me that my script has executed, but the target java process didn't run.
If I logon to that remote site, and execute ./service.sh start, it works.

Since you were able to run the service manually, the ssh and script part is fine.
What could go wrong is the environment. For example you referred java without absolute path. Hence your may be running a different version of it. It is also possible that variable for shared library loading (LD_LIBRARY_PATH) are having different values.
Finally check for file permissins

Error Expanding Script Generated Classpath for Java in PowerShell Script

Synopsis:
I am trying to convert a shell script from UNIX to Powershell.
The script cannot "hardcode" the path, but must instead list a directory to dynamically build the path at runtime.
Also, the script must pass in 1 parameter to indicate the "type" of the run.
I am placing my "attempted" re-write of the script (after much googling/binging)
This script might be very close as the "echo" statements produced what seems to be correct.
$EXE_CLASS="com.sungard.sims.dacar.standalone.Dacar"
#$DACAR_TAG=%1
$JAVA_EXE="$ENV:JDK15\jre\bin\java"
#JAVA_EXE=\opt\java\bin\java
$JARS="dist\lib\Dacar-Engine.jar;dist\lib\Dacar-Common.jar;dist\lib\Dacar-Collector.jar;dist\lib\standalone-Dacar.jar;dist\lib\DACAR-CRG.jar;"
#dynamically append all .jar files from the jars directory.
ls jars\*.jar | % { $JARS +=";jars\"+ $_.name }
$JAVA_OPTS="-server -Xloggc:.\engine.gc.log -Xmx2048m -Xms2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=60 -XX:+PrintGCTimeStamps -XX:MaxPermSize=128m -XX:PermSize=128m -DinstName=${DACAR_TAG} -DprocType=ENGINE"
echo "exe: -> $JAVA_EXE"
echo "opts: -> $JAVA_OPTS"
echo "jars: -> $JARS"
echo "class: -> $EXE_CLASS"
& $JAVA_EXE $JAVA_OPTS "-classpath .\configuration;$JARS $EXE_CLASS 1> dacar.out 2> dacar.err"
please note that I have commented out the use of the DACAR_TAG which would normally be passed in as %1 in batch script containing "TEST|DEV|PROD|QA|STRESS|[CLIENT SPECIFIC VALUE]"
Here is the output:
PS C:\dacar> .\startDacar.ps1
exe: -> c:\Program Files\Java\jdk1.5.0_22\jre\bin\java
opts: -> -server -Xloggc:.\engine.gc.log -Xmx2048m -Xms2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTe
nuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=60 -XX:+PrintGCTimeStamps -XX:MaxPermSize=128m -XX:PermSize=128m -DinstName=TESTING -DprocType=ENGINE
jars: -> dist\lib\Dacar-Engine.jar;dist\lib\Dacar-Common.jar;dist\lib\Dacar-Collector.jar;dist\lib\standalone-Dacar.jar;dist\lib\DACAR-CRG.jar;;jars\activation.jar;jars
\commons-beanutils-1.6.jar;jars\commons-codec-1.3.jar;jars\commons-collections-3.2.jar;jars\commons-configuration-1.4.jar;jars\commons-dbcp-1.4.jar;jars\commons-digester
-1.6.jar;jars\commons-discovery-0.4.jar;jars\commons-id.jar;jars\commons-lang-2.2.jar;jars\commons-logging-1.0.3.jar;jars\commons-pool-1.5.4.jar;jars\concurrent.jar;jars
\graphics.jar;jars\ha-javamail-1.0-beta-1.jar;jars\hsqldb.jar;jars\ibatis-core-3.0.jar;jars\jcommon-1.0.6.jar;jars\jconn3.jar;jars\jdom.jar;jars\jetty-6.1.15.jar;jars\je
tty-util-6.1.15.jar;jars\jfreechart-1.0.3.jar;jars\jgroups-all.jar;jars\jmxri.jar;jars\jta.jar;jars\junit.jar;jars\log4j-1.2.9.jar;jars\mailapi.jar;jars\postgresql-8.0.3
09.jdbc2ee.jar;jars\postgresql-8.4-701.jdbc3.jar;jars\quartz-1.6.0.jar;jars\servlet-api-2.5-20081211.jar;jars\smtp.jar;jars\svnant.jar;jars\svnClientAdapter.jar;jars\svn
javahl.jar;jars\truezip-6.jar;jars\uic-optional.jar;jars\uic.jar
class: -> com.sungard.sims.dacar.standalone.Dacar
Unrecognized option: -server -Xloggc:.\engine.gc.log -Xmx2048m -Xms2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepG
-XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=60 -XX:+PrintGCTimeStamps -XX:MaxPermSize=128m -XX:PermSize=128m -DinstName=TESTING -DprocType=ENGINE
Could not create the Java virtual machine.
Adding the tag will be my next question if I can't figure this out.
JUST AS AN FYI, from with in a ps1 script I can start it the program with the following command successfully:
& "c:\Program Files\Java\jdk1.6.0_20\jre\bin\java" -server -Xloggc:.\engine.gc.log -Xmx2048m -Xms2048m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:SurvivorRatio=8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=60 -XX:+PrintGCTimeStamps -XX:MaxPermSize=128m -XX:PermSize=128m -DinstName=TESTING -DprocType=ENGINE -classpath ".\configuration;dist\lib\Dacar-Engine.jar;dist\lib\Dacar-Common.jar;dist\lib\Dacar-Collector.jar;dist\lib\standalone-Dacar.jar;dist\lib\DACAR-CRG.jar;;jars\activation.jar;jars\commons-beanutils-1.6.jar;jars\commons-codec-1.3.jar;jars\commons-collections-3.2.jar;jars\commons-configuration-1.4.jar;jars\commons-dbcp-1.4.jar;jars\commons-digester-1.6.jar;jars\commons-discovery-0.4.jar;jars\commons-id.jar;jars\commons-lang-2.2.jar;jars\commons-logging-1.0.3.jar;jars\commons-pool-1.5.4.jar;jars\concurrent.jar;jars\graphics.jar;jars\ha-javamail-1.0-beta-1.jar;jars\hsqldb.jar;jars\ibatis-core-3.0.jar;jars\jcommon-1.0.6.jar;jars\jconn3.jar;jars\jdom.jar;jars\jetty-6.1.15.jar;jars\jetty-util-6.1.15.jar;jars\jfreechart-1.0.3.jar;jars\jgroups-all.jar;jars\jmxri.jar;jars\jta.jar;jars\junit.jar;jars\log4j-1.2.9.jar;jars\mailapi.jar;jars\postgresql-8.0.309.jdbc2ee.jar;jars\postgresql-8.4-701.jdbc3.jar;jars\quartz-1.6.0.jar;jars\servlet-api-2.5-20081211.jar;jars\smtp.jar;jars\svnant.jar;jars\svnClientAdapter.jar;jars\svnjavahl.jar;jars\truezip-6.jar;jars\uic-optional.jar;jars\uic.jar" com.sungard.sims.dacar.standalone.Dacar
But that is not really an option, as That is just invoking the thing in a hard coded fashion.

One thing to note about PowerShell variable expansion is that if it's anything more than a simple $VARIABLE expression you typically need to escape it using curly braces or a sub-expression.
So this line:
$JAVA_EXE="$ENV:JDK15\jre\bin\java"
Could be represented as either:
$JAVA_EXE="${ENV:JDK15}\jre\bin\java"
or...
$JAVA_EXE="$($ENV:JDK15)\jre\bin\java"
EDIT: Apparently PowerShell does
correctly evaluate all 3 examples. I
seem to recall the first syntax not
working but perhaps that was in
PowerShell 1.0 or a CTP...
Another useful tip that I often suggest people do when working with external .exe commands is to map it to an alias using the Set-Alias command. This way it makes working with the command a lot more natural as you don't have to use the &"string" syntax.
Set-Alias java "$($ENV:JDK15)\jre\bin\java.exe"
java blah1 blah2 etc