We have a Windows Server 2016 vm with 8192 MB ram with six cores and running RunDeck 3.4.1 with TomCat 9 from the rundeck.war file. Lately we've been seeing a couple issues crop up. First, RunDeck keeps user login sessions open well past the 30 minute idle limit in TomCat and Second, RunDeck does not respond or is extremely sluggish when the StandBy Memory leaves less than 400 MB of 'free memory' as if it never gets access to the StandBy Cache or queue or the priority is so low it can't get access to it. When a job fails, this problem gets even worse; but it also happens on successful jobs. This is causing our server to become unresponsive multiple times a day and the only way so far to free it is to manually release sessions in TomCat and/or to reboot the server completely. In the RunDeck Profile I have set the JVM to export RDECK_JVM="$RDECK_JVM -Xmx2048m -Xms512m -XX:MaxMetaspaceSize=512m -server".
Following the official documentation, those parameters (Xmx, Xms, and MaxMetaspaceSize) need to be defined in the setenv.bat file, take a look.
I have two weblogic domains each one has one managed server, the problem is that every 3 or 4 hours may be less than the four process are killed suddenly and in domain console i found that.
./startWebLogic.sh: line 175: 53875 Killed ${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}
There is no problem in free memory in server.
free memory
Two possible explanations for this message are the Linux OOM killer and the WebLogic node manager.
You should be able to find evidence for the first in /var/log/messages (grep -i -n 'killed process' /var/log/messages). If so, add up all the Xmx parameters of the running java processes, add 35% and see if that total tops the total amount of memory in the machine. If it does, tweak the Xmx parameters downwards.
The easier way to test for the second is to kill the nodemanager process, keep it down and see if the problem persists (kill -9 `ps -ef | grep odeManager | awk '{print $2}'`). If the problem does not reoccur, check the WebLogic admin console on how the "Panic action" and "Failure action" are configured for each server and set them to "No Action". In that case also check the nodemanager and server logs to figure out why the node manager killed your managed server processes.
We have a several application deployed in Jboss server. From last few month its observed that jboss servers gets killed without logging any error or exception neither in any application log nor in Jboss server log. The crash is not periodic and happens randomly. The crash is very silent (similar to kill -9 )and its become extremely difficult to debug the root cause.From our analysis one thing sure its not out of memory issue.We analysed the syslog as well, nothing suspicious found. Is possible to force VM to generate error log before the process crash due to any reason.
Any help would be appreciated.
The JVM logs fatal errors to hs_err_<pid>.log files
You can change the location of the fatal log file using java -XX:ErrorFile=/var/log/java/java_error%p.log
I experienced similar problem using JBoss EAP 6 in Linux enviroment, when system memory is almost gone the S.O kill some process, normally the file hs_err_pidxxx.log with crash report are generated inside bin directory of JBoss. Other case that process is killed is when you are using JBoss with DC mode, when instance stay long time without communication with DC, the process is finalized, I dont know why.
I have a Grails 2.2.4 application which is packed as war and deployed to my tomcat7 server on ubuntu 12.04 LTS 64-bit, 8GB RAM.
My setenv.sh file contains the following:
CATALINA_OPTS="
-server
-Xms1G
-Xmx2G
-XX:MaxPermSize=512m";
I used htop to get the number of running processes. I figured out that there are more than 20 running java processes on my system.
Each process of the more than 20 processes looks like this:
PRI NI VIRT RES SHR S CPU% MEM* TIME+ Command
20 0 6028M 1290M 11140 S 0.0 16.2 0:01.21 /usr/lib/jvm/java-7-oracle/bin/java -Djava.util.logging.config.file
When I stat tomcat with ./bin/startup.sh my application starts without errors. When I access my application with different browsers I get more than 20 Java processes running. The only other Java process I have running is elasticsearch.
Why is tomcat starting so many processes for my application?
Do I have to limit them? If so how?
What you are probably seeing is threads not processes. According to man htop you can hide user threads interactively using the H command.
For the record, Tomcat will create a number of worker threads for processing incoming HTTP requests. If you (really) need to control the number of worker threads, there are Tomcat configuration options for doing that.
I'm using Apache and Tomcat on a Windows server and since this morning, Tomcat stops working without any logs. It doesn't hang, it just shut down.
There's no log in Tomcat, the CPU/Memory are fines, there are no System.Exit in my code.
Anybody ever had this problem?
It happens at random, after 5-10 minutes. The application responds normally and sometime, boom.. stops working.
UPDATE : Still no clue. The Admin team will install the webapp on another box...
My script to start tomcat had last line tail -f catalina.out.
Sometime I did not kill this script, the shell then timed out and killed the script with all child processes, including tomcat.
This sounds like the JVM is crashing. Have you looked for a JVM crash log? It typically has a name like hs_err_pid*.log and is created in the JVM's working directory.
If you find a file like this and upload it, then we can probably help more.
Some questions:
Have you recently changed the version of Java you are using?
What is the exact version of Tomcat you are using?
Are you using Tomcat Native (the Apache Portable Runtime)?
Faced this issue recently.
Scenario : Tomcat started successfully but automatically gets shut down after 1 hour and sometimes this happened after 1 day and nothing is there in tomcat logs.
Issue : Actual issue was high memory usage and no free SWAP memory.
How I found the solution
If tomcat don't show any logs, then there must be something in system logs so, I checked /var/log/messages but since permission denied for me I tried /var/log/dmesg and got this
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have taken successfully and the processes were not killed even after the shutdown due to some reason, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)
Recently I had this problem, If somebody faces the same issue in future I hope this will help.
Scenario: Tomcat shuts down without any logs or errors
Root Cause for my problem: synchronized method accessed from a task using TimerTask
I had a singleton class with a synchronized method accessed from various threads based on timer or user action
some times this method will take up to few minutes to complete. When TimerTask is waiting on this method for sometime (I guess timer is timed out /thread is killed or something is happening in the background) and the moment the lock on the method is released the tomcat is getting killed.
So I removed synchronized keyword and removed singleton and made some code changes for thread safety. Then the problem is gone.
How I found out: I had a log statement in the first line of synchronized method and everytime the tomcat shutdowns i found this message in the last few lines.
Regards,
Phanindra Kasturi
things to look for in debugging an issue like this:
Look at the logs directory ($TOMCAT_ROOT/logs) to make sure none of the log files have any stack traces
Look at the tomcat startup script to make check the location of the log files to see if the logs are not being written to another directory.
Another reason could be some other user/process could be issuing a kill -9 that could kill tomcat without giving it any chance to log errors.
another possibility is that some process was started this morning on the box that is binding to a port that your server requires.
Are your servlets or one of it's dependencies allowed to call System.exit()? (Not sure how locked down Tomcat VMs are in that sense)
I've had developers thinking it's ok to use exit(666); on detecting a non-invertable matrix (which isn't good, but sure as heck not fatal). Arrgh. Perhaps you have some similar culprit in your system?
I noticed CATALINA_OPTS in my path and that was set for a lower JVM size. Hence, the crash and no log trace of tomcat was found. The server automatically shutdown in less than 2 hrs.
check, CATALINA_OPTS or JAVA_OPTS -- these might have jvm settings. either increase them or comment them out and increase the swap memory.
“The Service on local computer started and then stopped ,Some services stop automatically if there are not in use by other services or programs.”
I gone through the problem i have tried so many ways to get out of the problem finally i got the solution as follows.
1) Click Run Command from start button.
2) Enter Services.msc then click OK,you will get all the services in your computer.
3) Select your service and right click on the service and select Properties
4) Goto Logon Properties and select Local System Account then click OK.
This will work.
Sometime it happens if some other program is running on the same port. For example Skype. Shut down that program before you start Tomcat.
try to clean your elipse projects because you could have tried to add another server which used port 8080 then when you try to execute the tomcat server externally that defaulty uses port 8080 the tomcat server automatically shutdowns after cleaning the project copy the new war file and paste it in bin it works fine
conclusion: when the server tries to use the port which has already been acquired you will see such type of issues.