We Keep Coding

Wildfly 17 enabling JMX remote on the same server gives logmanager errors

I’m looking to enable JMX to allow datadog to monitor our java JBoss wildfly systems but keep hitting runtime errors
I have set up the standalone.xml with
<subsystem xmlns="urn:jboss:domain:jmx:1.3">
<expose-resolved-model/>
<expose-expression-model/>
<remoting-connector use-management-endpoint="true"/>
</subsystem>
And
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
</interface>
As well as
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
Then in my startup.sh i have added
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"
But this gives me
java.lang.IllegalStateException: The LogManager was not properly
installed (you must set the "java.util.logging.manager" system
property to "org.jboss.logmanage r.LogManager")
This seems to be fairly common if I look at both here and on google but there seem to be different solutions depending on the version of wildfly.
I think I need to do something like
Set at the start of the standalone.conf
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.logmanager"
And then
JBOSS_HOME="/opt/wildfly"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.11.Final.jar -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.2.0.Final.jar"
At the end.
But I still get errors “Could not load Logmanager "org.jboss.logmanager.LogManager"”
Any advice would be appreciated.

I encountered the same issue when I upgrade Wildfly server version from 10.1 to 24.0.0 and I resolved it using the below changes.
The below JAVA_OPTS changes are required to be added at the end of the standalone.conf file
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/a:$JBOSS_HOME/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.5.4.Final.jar -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager -Xbootclasspath/a:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.18.Final.jar -Xbootclasspath/a:$JBOSS_HOME/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.2.0.Final.jar"
# Enable jmx remote management
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=4447 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dsun.util.logging.disableCallerCheck=true -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote.local.only=false"
NOTE: wildfly-common-.jar, jboss-logmanager-.jar, log4j-jboss-logmanager-.jar dependencies should be added to classpath using -Xbootclasspath/a

Use the supplied jconsole.sh script in bin, don't try and build up the classpath by hand. You also need to use the custom service url. See the docs for details

change in standalone.sh: #!/bin/sh to #!/bin/bash
then
in standalone.conf :
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.logmanager,jdk.nashorn.api,com.sun.crypto.provider,$JBOSS_MODULES_SYSTEM_PKGS"
...
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
...
BOOT_CP=()
BOOT_CP=("$(find $JBOSS_HOME/modules -type f -name "jboss-logmanager-*.jar")")
BOOT_CP+=("$(find $JBOSS_HOME/modules -type f -name "wildfly-common*.jar")")
for f in $(find $JBOSS_HOME/modules -type f -name "javax.json-*.jar"); do
BOOT_CP+=("$f")
done
x="-Xbootclasspath/a"
for p in "${BOOT_CP[#]}"; do
x+=":$p"
done
JAVA_OPTS="-Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dsun.util.logging.disableCallerCheck=true $x $JAVA_OPTS"
it is work but I have still some issues...

Mesos java Application Monitoring

I am running a java application on Mesos and Marathon. How do i monitor the heap stats of the java process? I tried jstat which complains "pid not found", but the process is running and i can check it by 'ps auwx'. Is this because the java process is running inside a cgroup? If that is the case how do we monitor mesos based applications.

The best option to monitor application is to publish it's metrics from itself. There are plenty of ways to add monitoring to your application starting from proprietarty such as NewRelic or DataDog to opensource Prometheus.
If you need to get acces via JMX you can expose this port with following configuration:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.port=$PORT1 \
-Djava.rmi.server.hostname=$MESOS_HOSTNAME
Above configuration assume you have two ports and second is dedicated for JMX

How to stop application logs from logging into catalina.out in Tomcat

The application is running in tomcat and has it own logger using org.apache.commons.logging.Log and org.apache.commons.logging.LogFactory.
The logs are getting logged at location specified in log4j.properties file, the location is as follows.
log4j.appender.logger.File=${catalina.base}/logs/applicationlogs.log
The logs are simultaneously added in following file.
/opt/apache-tomcat-8.0.26/logs/catalina.out
How to stop the application logs from getting logged in catalina.out ?

You can try to do this:
Edit "$CATALINA_BASE"/bin/catalina.sh file
Find CATALINA_OUT="$CATALINA_BASE"/logs/catalina.out
Replace with new path.
Don't forget to restart tomcat.
And as suggested in comments, to block writing to catalina.out entirely, set CATALINA_OUT=/dev/null in catalina.sh.

I believe the recommended way to "augment" catalina.sh (catalina.bat on Windows) is to:
Create or update a script in ${CATALINA_BASE}/bin called
setenv.sh (setenv.bat)
Add a line to set: CATALINA_OUT=/dev/null as noted above.
This script will be executed before executing catalina.sh/bat which will use any vars set. This is also the correct way to add to the CLASSPATH, add JAVA_OPTS, etc. without messing with catalina.sh/bat directly, which may be updated (overriden) with each tomcat upgrade.

If your application is using a console appender, then those logs will go to catalina.out. You might want to check that in your application.
To disable logging to catalina.out, you can check discussion: here

Find CATALINA_OUT="$CATALINA_BASE"/logs/catalina.out in catalina.sh
$CATALINA_BASE/bin
Set CATALINA_OUT=/dev/null
Restart Tomcat
You will see log file created as per your log configuration.

I'm using tomcat 7.0.50 and I've done following configuration.
To stop the application to log into catalina.out, you can do it by removing the handler.
This can be achieved by editting conf/logging.properties and changing:
.handlers = 1catalina.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
to
.handlers = 1catalina.org.apache.juli.FileHandler
Hope this helps. Let me know if I'm missing something.

You can try following to disable writing to catalina.out :
Locate and Edit File: {CATALINA_BASE}/bin/catalina.sh
Locate "CATALINA_OUT" and replace the path with "/dev/null":
/dev/null in Linux is a null device file. This will discard anything written to it.
CATALINA_OUT="$CATALINA_BASE"/logs/catalina.out // Original Location
CATALINA_OUT=/dev/null // replace path to /dev/null

Just changing
CATALINA_OUT="$CATALINA_BASE"/logs/catalina.out
To
CATALINA_OUT= /dev/null
In catalina.sh is not enough because the CATALINA_OUT variable will be used later in the script and that will cause an error to be thrown (aborting the startup).
What I've done instead was, I went to the command lines themselves and edited them so that the output itself will be redirected to /dev/null
(In Tomcat 9 it's in the if block in line ~447)
$_NOHUP "\"$_RUNJAVA\"" "\"$LOGGING_CONFIG\"" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS \
-D$ENDORSED_PROP="\"$JAVA_ENDORSED_DIRS\"" \
-classpath "\"$CLASSPATH\"" \
-Djava.security.manager \
-Djava.security.policy=="\"$CATALINA_BASE/conf/catalina.policy\"" \
-Dcatalina.base="\"$CATALINA_BASE\"" \
-Dcatalina.home="\"$CATALINA_HOME\"" \
-Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
org.apache.catalina.startup.Bootstrap "$#" start \
>> "$CATALINA_OUT" 2>&1 "&"
I've set the stdout stream to /dev/null
$_NOHUP "\"$_RUNJAVA\"" "\"$LOGGING_CONFIG\"" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS \
-D$ENDORSED_PROP="\"$JAVA_ENDORSED_DIRS\"" \
-classpath "\"$CLASSPATH\"" \
-Djava.security.manager \
-Djava.security.policy=="\"$CATALINA_BASE/conf/catalina.policy\"" \
-Dcatalina.base="\"$CATALINA_BASE\"" \
-Dcatalina.home="\"$CATALINA_HOME\"" \
-Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
org.apache.catalina.startup.Bootstrap "$#" start \
>> /dev/null 2>&1 "&"

Java VisualVM not able to connect to Jboss

Java VisualVM not able to connect Jboss server, when jboss server running with option -b 0.0.0.0.
jboss run script - run.sh -c web -b 0.0.0.0
When i try to run jboss with this like - run.sh -c web -b {MyIp}
then Java VisualVM is able to connect jboss server.
Can any one help me how to connect while jboss running with first option ( -b 0.0.0.0)
I am using this configuration in run.conf file in jboss
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=256m -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8077 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Add this to JAVA_OPTS
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=256m -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8077 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=0.0.0.0"
instead of 0.0.0.0 use your system IP.

Can't connect jconsole to remote server, tomcat failing to start

Trying to connect jconsole to a remote server.
I added this to my catalina.sh:
export JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9005 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Djava.rmi.server.hostname=xx.xx.xx.xx"
catalina.out shows:
Error: Exception thrown by the agent : java.net.MalformedURLException: Local host name unknown: java.net.UnknownHostException: myhostname: myhostname
Not sure why it repeats my hostname in the error message?
BTW, since I set authentication to false, in the jconsole app, do I leave username/password blank or is that for logging into the server?

You have to add the same host name in /etc/hosts file as you have defined in /etc/sysconfig/network file.
This is how I solved my problem.

I have found the solution for this issue.
Add the following in your catalina.sh file:
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=7010 -Djava.rmi.server.hostname=${IP}"
Also add the following line in your /etc/init.d/hosts file:
127.0.0.1 localhost <your_hostname>
This resolved the issue. I am able to run jconsole as well as jvisualvm on this port now.
I hope this helps !

If you want to get the IP address dynamically you can try:
IP=`ifconfig | grep 'inet addr:'| grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $1}'`
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=${IP}"

If you use '\' in your 'export' statement, remove those.
To connect to remote java process, Use IP address of the server where your java process (tomcat instance) is running. The UnknownHostException is thrown when IP address could not be determined, so another option is to add the name - IP address definition to your hosts file.

You can try adding the parameters that you have added in JAVA_OPTS to CATALINA_OPTS. It should work that way.
Also make sure u are making the settings with the same profile login from where you are running tomcat.