I have a java api service deployed in wildfly on Amazon linux, randomly the api calls are not reaching the api-service at all, but the wildfly is up and running, the memory/cpu usage seems normal at this point. I couldn't see any error on the application log.
When I restart wildfly, the application work normally.
Any pointers to debug this issue? Thank you
Wildfly Configuration
# General configuration for the init.d scripts,
# not necessarily for JBoss AS itself.
# default location: /etc/default/wildfly
## Location of JDK
JAVA_HOME="/usr/"
## Location of WildFly
JBOSS_HOME="/opt/wildfly"
## The username who should own the process.
JBOSS_USER=root
## The mode WildFly should start, standalone or domain
JBOSS_MODE=standalone
## Configuration for standalone mode
JBOSS_CONFIG=standalone.xml
## Configuration for domain mode
# JBOSS_DOMAIN_CONFIG=domain.xml
# JBOSS_HOST_CONFIG=host-master.xml
## The amount of time to wait for startup
STARTUP_WAIT=60
## The amount of time to wait for shutdown
SHUTDOWN_WAIT=60
## Location to keep the console log
JBOSS_CONSOLE_LOG="/var/log/wildfly/console.log"
## Additionals args to include in startup
# JBOSS_OPTS="--admin-only -b 127.0.0.1"
JBOSS_OPTS="-Djboss.bind.address=xxx.xxx.xxx.xxx -DConfig=/opt/app-conf/app-conf.properties -DLogDirectory=/var/log/wildfly"
Here is the output of ps -ef | grep wildfly
/usr//bin/java -D[Standalone] -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/wildfly/standalone/configuration/logging.properties -jar /opt/wildfly/jboss-modules.jar -mp /opt/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/wildfly -Djboss.server.base.dir=/opt/wildfly/standalone -c standalone.xml -Djboss.bind.address=xxx.xxx.xxx.xxx -DConfig=/opt/app-conf/app-conf.properties -DLogDirectory=/var/log/wildfly
Can you help to check if the memory and heap space are right? Or should I increase what could be the optimal value for it?
Related
I have a simple Micronaut server I am trying to launch on Heroku by building it with a heroku.yml but for some reason when I check the logs the process exits as soon as it starts. The docker image runs just fine locally and nothing else prints out in the logs so I can't find out why.
Here is my Dockerfile
FROM node as build-frontend
WORKDIR /app
COPY frontend/myfrontend/package.json .
COPY frontend/myfrontend/public ./public
COPY frontend/myfrontend/src ./src
RUN npm install .
RUN npm run build
FROM gradle:6.8.2-jre11 AS build-env
# Set the working directory to /home
WORKDIR /home
COPY --chown=gradle:gradle backend ./
COPY --from=build-frontend /app/build /home/src/main/resources/public
# Compile the application.
RUN ./gradlew assemble
FROM openjdk:11.0.10-jre-slim-buster
# Set the working directory to /home
WORKDIR /home
# Copy the compiled files over.
COPY --from=build-env /home/build/libs/myjar-0.1-all.jar /home/myjar.jar
# Starts the java app
# Using EntryPoing to pass env. variables as describe in this article
ENTRYPOINT exec java -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0 -noverify -XX:+AlwaysPreTouch -jar myjar.jar
Here is my heroku.yml
setup:
addons:
- plan: heroku-postgresql
as: DATABASE
build:
docker:
web: Dockerfile
run:
web: "exec java -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0 -noverify -XX:+AlwaysPreTouch -jar myjar.jar"
and finally my Micronaut application.yml which just sets some configs
micronaut:
application:
name: mypackage
router:
static-resources:
default:
enabled: true
paths:
- classpath:public
mapping: /**
server:
port: ${PORT:8080}
cors:
enabled: true
datasources:
default:
driverClassName: org.postgresql.Driver
dbUrl: jdbc:postgresql://mydbhost.com:port/dbname?sslmode=require
dbUsername: dbuser
dbPassword: dbpasswd
scan:
packages: mypackage
netty:
default:
allocator:
max-order: 3
When I just do docker build -t test-image:latest . and docker run -p 80:8080 test-image:latest I can connect just fine on localhost and the docker container runs the micronaut server. If it fails for some reason I see an output in the container logs detailing why. When I upload this to heroku (through a github deployment) All i see in the logs are
2023-01-09T22:25:50.145942+00:00 heroku[web.1]: Starting process with command `/bin/sh -c exec\ java\ -XX:\+UseContainerSupport\ -XX:MaxRAMPercentage\=80.0\ -noverify\ -XX:\+AlwaysPreTouch\ -jar\ myjar.jar`
2023-01-09T22:25:51.381760+00:00 heroku[web.1]: Process exited with status 0
2023-01-09T22:25:51.439962+00:00 heroku[web.1]: State changed from starting to crashed
I have tried:
To run it locally connected to the heroku addon postgres database, works just fine
Simplifying the build as much as possible
Removing the default on the port (to ensure it picks up $PORT), and running it locally with export PORT=8080 set (works just fine, docker container picks up the env port like we expect in heroku)
And I cannot figure out why it just quits immediately on Heroku. editted: originally thought it had to do with the port value but but I figured out how to give micronaut a port through the command line and it still doesn't work on Heroku (works locally)
edit:
I tried changing my application.yml to this with the DB hardcoded and still nothing. Just does not seem to work. App still crashes and there is nothing to indicate why in the logs
setup:
config:
PORT: $PORT
MICRONAUT_SERVER_PORT: $PORT
build:
docker:
web: Dockerfile
run:
web: "exec java -Dmicronaut.server.port=$PORT -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0 -noverify -XX:+AlwaysPreTouch -jar myjar.jar"
I still have no extra output from Heroku. No stacktrace or stderr or stdout at all.
Well, it isn't much of an answer. And nothing in the Heroku docs points me to why this works but removing ENTRYPOINT exec java -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0 -noverify -XX:+AlwaysPreTouch -jar myjar.jar from my Dockerfile works. With entrypoint removed I see logs again and can see my micronaut server starting. With ENTRYPOINT defined in my Dockerfile I just get what I posted about above "Process starting" followed immediately by process crashed
I have been meaning to monitor my wildfly in zabbix through JMX. The versions are listed below
wildfly: 20.0.0.1 final
zabbix: 5.0.7
java(that is present in wildfly host): openjdk version "11.0.8" 2020-07-14
The issue i am facing while i run wildlfy is as follows.
-Xbootclasspath is no longer a supported option. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit
My standalone.conf file looks like as below:
if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
JBOSS_MODULES_SYSTEM_PKGS="org.jboss.bytemani,org.joss.logmanager"
fi
# Uncomment the following line to prevent manipulation of JVM options
# by shell scripts.
#
#PRESERVE_JAVA_OPTS=true
#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms2G -Xmx2G -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=1G -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath:/opt/wildfly/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.15.Final.jar"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
else
echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi
As per my research, i have found that my java version is not compatible and i dont know the work around process.
I have a laptop: hp pav dv6 4 gb ram 32bit win 8. This is what I get when I tried to run an old project.
Error message:
Unable to start the daemon process. This problem might be caused by
incorrect configuration of the daemon. For example, an unrecognized
jvm option is used. Please refer to the User Manual chapter on the
daemon at https://docs.gradle.org/5.6.4/userguide/gradle_daemon.html
Process command line: C:\Program Files\Java\jdk1.8.0_261\bin\java.exe
-Xmx1536m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:\Gradle\gradle-4.1\wrapper\dists\gradle-5.6.4-all\ankdp27end7byghfw1q2sw75f\gradle-5.6.4\lib\gradle-launcher-5.6.4.jar
org.gradle.launcher.daemon.bootstrap.GradleDaemon 5.6.4 Please read
the following process output to find out more:
----------------------- Error occurred during initialization of VM Could not reserve enough space for 1572864KB object heap
----------------------- Check the JVM arguments defined for the gradle process in:
gradle.properties in project root directory
gradle.properties in in GRADLE_USER_HOME directory
gradle.properties:
## For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html
#
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
# Default value: -Xmx1024m -XX:MaxPermSize=256m
# org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
#
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true
#Wed Nov 18 01:37:38 PST 2020
android.enableJetifier=true
android.useAndroidX=true
org.gradle.jvmargs=-Xmx512m
systemProp.http.proxyHost=
systemProp.http.proxyPort=80
systemProp.https.proxyHost=
systemProp.https.proxyPort=80
invalidate cache restart and changing default value can make run project
I am running Weblogic 12c on CentOS6. I have enabled the JMX on Weblogic AdminServer.
But I want to enable the JMX on the cluster servers such as ms1 listed below to monitor specific Java service. What shall I do? Thank you!
Servers:AdminServer(admin)
Clusters: ms1/ms2/ms3
[root#weblogic61 bin]# ps -ef |grep java
weblogic 3618 3571 0 Jun04 ? 03:14:41 /home/jdk/jdk1.7.0_45/bin/java -server -Xms1024m -Xmx2048m -XX:MaxPermSize=1024m -Dweblogic.Name=ms1 -Djava.security.policy=/home/weblogic/Oracle/Middleware/wlserver_12.1/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/home/weblogic/Oracle/Middleware/wlserver_12.1/server/lib/cacerts -Djava.endorsed.dirs=/home/jdk/jdk1.7.0_45/jre/lib/endorsed:/home/weblogic/Oracle/Middleware/wlserver_12.1/endorsed -da -Dplatform.home=/home/weblogic/Oracle/Middleware/wlserver_12.1 -Dwls.home=/home/weblogic/Oracle/Middleware/wlserver_12.1/server -Dweblogic.home=/home/weblogic/Oracle/Middleware/wlserver_12.1/server -Dweblogic.management.discover=false -Dweblogic.management.server=http://192.168.97.61:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/home/weblogic/Oracle/Middleware/patch_wls1211/profiles/default/sysext_manifest_classpath:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server
root 27588 27541 41 09:56 pts/0 00:00:25 /home/jdk/jdk1.7.0_45/bin/java -server -Xms1024m -Xmx2048m -XX:MaxPermSize=1024m -Dweblogic.Name=AdminServer -Djava.security.policy=/home/weblogic/Oracle/Middleware/wlserver_12.1/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Djava.endorsed.dirs=/home/jdk/jdk1.7.0_45/jre/lib/endorsed:/home/weblogic/Oracle/Middleware/wlserver_12.1/endorsed -Djava.rmi.server.hostname=192.168.97.61 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -da -Dplatform.home=/home/weblogic/Oracle/Middleware/wlserver_12.1 -Dwls.home=/home/weblogic/Oracle/Middleware/wlserver_12.1/server -Dweblogic.home=/home/weblogic/Oracle/Middleware/wlserver_12.1/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/home/weblogic/Oracle/Middleware/patch_wls1211/profiles/default/sysext_manifest_classpath:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server
root 27638 27312 0 09:57 pts/1 00:00:00 grep java
zabbix 27853 1 0 Jun13 ? 00:14:16 java -server -classpath lib:lib/logback-classic-0.9.27.jar:lib/logback-core-0.9.27.jar:lib/org-json-2010-12-28.jar:lib/slf4j-api-1.6.1.jar:bin/zabbix-java-gateway-2.0.12.jar -Dzabbix.pidFile=/var/run/zabbix/zabbix_java.pid -Dzabbix.listenIP=0.0.0.0 -Dzabbix.listenPort=10052 -Dzabbix.startPollers=5 com.zabbix.gateway.JavaGateway
The first question is where are those settings coming from for your admin server. My guess is you need to edit startWebLogic.sh and find where the following are being set:
JAVA_OPTIONS="-Dcom.sun.management.jmxremote.port=12345"
JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTIONS="${JAVA_OPTIONS} -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder"
export JAVA_OPTIONS
Thos settings are probably being explicitly set for your admin server only based on an if statement or some such.
You can also set those -D values for your managed server by going to:
Admin Console -> Servers -> MS -> Server Start -> JVM Arguments
Last, in the Weblogic Domain->Configuration->General->Advanced
Enable both - Platform MBean Server Enabled and Platform MBean Server Used
Admin Console -> Servers -> Server-1 -> Server Start -> JVM Arguments
-DUseSunHttpHandler=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8501 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=192.168.20.209
Admin Console -> Servers -> Server-2 -> Server Start -> JVM Arguments
-DUseSunHttpHandler=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8502 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=192.168.20.209
along with below command line parameters ,
-Dcom.sun.management.jmxremote.port='you jmx port'
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder
Sometimes in the linux servers , imx connection doesn't get succeeded. that is because , in cloud linux host, in /etc/hosts so that the hostname resolves to the host address.
the best way to fix it is, ping the particular linux server from other machine in network and use that host IP address in the
-Djava.rmi.server.hostname=IP address that obtained when you ping that linux server.
But never rely on the ipaddress that you get from linux server using ifconfig.me. the ip that you get there is masked one which is present in the host file.
I have the following problem:
I am using Tomcat 6.0.32 and Java JDK 6.0_26. I have installed it successfully and the Tomcat start page is visible in the browser at port 8080.
I have also created $CATALINA_HOME/setenv.sh script and put some webapp-specific environment variables in it (along with the CATALINA_HOME, JAVA_HOME and CLASSPATH).
I have created a new user "tomcat", set a new home directory for him, and also passwd-ed it.
This script is being sourced from within a init script I created to start and stop Tomcat automatically on reboot. I do not use the standart startup.sh and shutdown.sh found in $CATALINA_HOME, but rather then jsvc daemon starter, so I can use port 8080 from a non-root process (Tomcat itself).
The actual problem is that, after restarting Tomcat my webapp does not receive or see the environment variable I set in setenv.sh and so it won't start.
I have tried to put the environment variable definition in various places:
.bashrc in the tomcat home directory
/etc/init.d/tomcat script
$CATALINA_HOME/bin/setenv.sh
$CATALINA_HOME/webapps/myapp/META-INF/context.xml
to no avail, after start of Tomcat my webapp does not see the required environment variables.
My question is - what the heck am I doing worng? Any suggsetions? How am I supposed to transfer env vars to an webapp if the setenv.sh does not work? What could make this mechanism faulty (allegedly this is the way to hand env vars to webapps)?
Here is the startup script I worte:
#!/bin/sh
### BEGIN INIT INFO
# Provides: allfaweb
# Required-Start: $syslog $apache $apache2 $httpd
# Should-Start:
# Required-Stop: $syslog $apache $apache2 $httpd
# Should-Stop:
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: ALLFAweb service
### END INIT INFO
ALLFAWEB_BIN=/install/apache-tomcat-6.0.32-allfaweb/bin/jsvc
test -x $ALLFAWEB_BIN || { echo "$ALLFAWEB_BIN not installed";
if [ "$1" = "stop" ]; then exit 0;
else exit 5; fi; }
# Check for existence of setenv.sh file and read it
ALLFAWEB_CONFIG=/install/apache-tomcat-6.0.32-allfaweb/bin/setenv.sh
test -r $ALLFAWEB_CONFIG || { echo "$ALLFAWEB_CONFIG not existing";
if [ "$1" = "stop" ]; then exit 0;
else exit 6; fi; }
. /etc/rc.status
rc_reset
. /install/apache-tomcat-6.0.32-allfaweb/bin/setenv.sh;
case "$1" in
start)
echo -n "Starting ALLFAweb ";
$ALLFAWEB_BIN \
-user tomcat \
-home $JAVA_HOME \
-Dcatalina.home=$CATALINA_HOME \
-pidfile $ALLFAWEB_PID \
-outfile $CATALINA_HOME/logs/catalina.out \
-errfile $CATALINA_HOME/logs/catalina.err \
-cp $CLASSPATH org.apache.catalina.startup.Bootstrap
rc_status -v
;;
stop)
echo -n "Shutting down ALLFAweb "
$ALLFAWEB_BIN \
-stop \
-pidfile $ALLFAWEB_PID \
org.apache.catalina.startup.Bootstrap
rc_status -v
;;
restart)
$0 stop
$0 start
rc_status
;;
status)
echo -n "Checking for service ALLFAweb ";
/sbin/checkproc $ALLFAWEB_BIN
rc_status -v
;;
*)
echo "Usage: $0 {start|stop|status|restart}"
exit 1
;;
esac
rc_exit
The system I am using is a SUSE SP2:
# uname -a
Linux testmachine 3.0.51-0.7.9-default #1 SMP Thu Nov 29 22:12:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Any help would be highly appreciated! Thanks in advance :)
You should be using system properties and not environment variables. Check the source for Tomcat's bin/daemon.sh script, which passes all of the standard variables to Tomcat when it is launched.
I found it. Allegedly the reason is that at first I started Tomcat 6 as root, and then changed the ownership of files to user "tomcat". The root process had written files all over the file system wiht the appropriate permissions, that is, the "tomcat" user could not read them.
The default shell of the "tomct" user was /bin/sh and not /bin/bash
I also deleted the files in $CATALINA_HOME/work and renamed the directory to which my custom env var was pointing to (and the env var accordingly).
There were also some logfiles generated from log4j in the / dicrectory, those are gone too and now everything works as expected :)
Thanks for all the feedback, you kept me going during these 3 days of frustration :)
Cheers!