Before redeploying the application war, I checked the xd.lck file from one of the environment path:
Private property of Exodus: 20578#localhost
jetbrains.exodus.io.LockingManager.lock(LockingManager.kt:89)
I'm testing from both Nginx Unit and Payara server to eliminate the possibility that this is an isolated case with Unit.
And process 20578 shows from htop:
20578 root 20 0 2868M 748M 7152 S 0.7 75.8 14:05.75 /usr/lib/jvm/zulu-8-amd64/bin/java -cp /
After redeployment finished successfully, accessing the web application throws:
java.lang.Thread.run(Thread.java:748)
at jetbrains.exodus.log.Log.tryLock(Log.kt:799)
at jetbrains.exodus.log.Log.<init>(Log.kt:120)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:142)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:121)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:10
And checking the same xd.lck file shows the same content. Meaning to say that "lock is not immediately released" contrary to what is described here.
My assumption is for this specific case with Payara Server (based on Glassfish) is that, the server does not kill the previous process even after redeployment has completed. Maybe perhaps for "zero-downtime" redeployment, not sure, Payara experts can correct me here.
Checking with htop the process 20578 is still running even after the redeployment.
As with Xodus, since most application servers behave this way, what would be the best solution and/or workaround so we don't need to manually delete all lock files of each environment (if can be deleted) every time we redeploy?
Solution is for the Java application to look for the process locking the file then do a kill -15 signal for example to gracefully make the Java handle the signal to be able to close environments:
// Get all PersistentEntityStore's
entityStoreMap.forEach((dir, entityStore) -> {
entityStore.getEnvironment().close();
entityStore.close();
}
I've got a Java8 application running on RHEL 6.10. This application registers a shutdown handler via the usual method:
Thread shutdownThread = new Thread(()=>{
Logger.info("Got shutdown signal");
// Do cleanup
});
Runtime.getRuntime().addShutdownHook(shutdownThread);
This application is kicked off by a Jenkins build (with the BUILD_ID env var set to dontkillme). The application initializes successfully, but then after ~30 seconds the shutdown hook is called and the application terminates. I'm trying to figure out who is shutting me down and why. I've monitored top and it doesn't appear that memory is an issue while it's running, so I don't think the OOM killer is the culprit. I've also looked at /var/log/dmesg and /var/log/messages and don't see anything relevant there either. I don't think Jenkins would be killing me, both since I set BUILD_ID and also because the application dies while the "parent" Jenkins job is still running.
What other methods / tools can I use to see what's happening? Note that my environment is very locked down, so it would be difficult to download and run something from the internet, hopefully there's something in a standard RHEL6 install I could use.
The answer turned out to be unique to the application in question. The application uses the zookeeper-3.5.5 library to connect to a Zookeeper instance. This client library has a runtime dependency on the zookeeper-jute jar, and when that jar was not present in the executing directory, this issue presented itself.
You're probably wondering why the application silently shut itself down and didn't throw a ClassNotFoundException which would have helped me debug this. Great question! I have no idea.
I created a webapplication that needs to do some cleanup on shutdown. This cleanup will take about a minute and its completely OK for it to do so.
When I deploy my webapp onto Tomcat 8 and then stop it, my ContextListener gets called and the cleanup begins. But it seems like Tomcat stops my thread the hard way and it won't complete anymore. At least on Tomcat 6 that wasn't an issue.
An ideas how to configure Tomcat 8 to stop from misbehaving?
Partial Answer:
I found out it has something to do with a performance optimization I did. I used startStopThreads="2" to start my applications in parallel, which works out well, but on shutdown this also seems to kill my threads.
If you have a task which is to be performed on shutdown, I would add this as shutdown hook. Most likely Tomcat 8 is called System.exit() which is a normal thing to do and this kills all user threads but start shutdown hooks.
A better solution is to never leave the system in a state where you really need this. i.e. you cannot assume an application will die gracefully.
if you are waiting for client to disconnect, I suggest you add a shutting down phase. During this phase you refuse new connections, move connections to another server or attempt to gracefully tell existing ones you are going away. After a short period or time out, you then shut down.
I have a web application with Servlets in the back end deployed over tomcat. The application is simple java application.
I see this error frequently in the server logs:
SEVERE: A web application appears to have started a thread named
[22] but has failed to stop it. This is very likely
to create a memory leak.
Are there any potential reasons which might be causing it?
I'd use visualvm 1.3.2 and see what threads are being created. Be sure to add all the plug-ins.
If it's not being done by your code you won't have much control over it.
You also don't know if the message is a red herring or not. Load test your code over a period of time and measure what happens.
I faced similar situation recently Resolved in below steps
I took Thread dump. ( using Kill -QUIT pid )
Found the Runnable/Thread class form dump
Then i put a debug point in run method and started application in
debug mode.
Got the code which starts My Thread and I observed it was not
stopped while stoping application.
Introduced code to stop the thread in contextDestroyed method of
AppContextListener (This is My application class which extends
ServletContextListener ) as this method will be called when i stop
tomcat.
If you set Thread as Dameon Thread it is not going to help , you can visit explanation.
Tomcat waits for all the application's threads (user threads not daemon threads) to stop before it goes down, I guess that in your case this specific thread is a user thread and therefore tomcat generated this error.
I suggest you to change this thread to daemon (assuming this one is yours)
I'm using Apache and Tomcat on a Windows server and since this morning, Tomcat stops working without any logs. It doesn't hang, it just shut down.
There's no log in Tomcat, the CPU/Memory are fines, there are no System.Exit in my code.
Anybody ever had this problem?
It happens at random, after 5-10 minutes. The application responds normally and sometime, boom.. stops working.
UPDATE : Still no clue. The Admin team will install the webapp on another box...
My script to start tomcat had last line tail -f catalina.out.
Sometime I did not kill this script, the shell then timed out and killed the script with all child processes, including tomcat.
This sounds like the JVM is crashing. Have you looked for a JVM crash log? It typically has a name like hs_err_pid*.log and is created in the JVM's working directory.
If you find a file like this and upload it, then we can probably help more.
Some questions:
Have you recently changed the version of Java you are using?
What is the exact version of Tomcat you are using?
Are you using Tomcat Native (the Apache Portable Runtime)?
Faced this issue recently.
Scenario : Tomcat started successfully but automatically gets shut down after 1 hour and sometimes this happened after 1 day and nothing is there in tomcat logs.
Issue : Actual issue was high memory usage and no free SWAP memory.
How I found the solution
If tomcat don't show any logs, then there must be something in system logs so, I checked /var/log/messages but since permission denied for me I tried /var/log/dmesg and got this
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have taken successfully and the processes were not killed even after the shutdown due to some reason, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)
Recently I had this problem, If somebody faces the same issue in future I hope this will help.
Scenario: Tomcat shuts down without any logs or errors
Root Cause for my problem: synchronized method accessed from a task using TimerTask
I had a singleton class with a synchronized method accessed from various threads based on timer or user action
some times this method will take up to few minutes to complete. When TimerTask is waiting on this method for sometime (I guess timer is timed out /thread is killed or something is happening in the background) and the moment the lock on the method is released the tomcat is getting killed.
So I removed synchronized keyword and removed singleton and made some code changes for thread safety. Then the problem is gone.
How I found out: I had a log statement in the first line of synchronized method and everytime the tomcat shutdowns i found this message in the last few lines.
Regards,
Phanindra Kasturi
things to look for in debugging an issue like this:
Look at the logs directory ($TOMCAT_ROOT/logs) to make sure none of the log files have any stack traces
Look at the tomcat startup script to make check the location of the log files to see if the logs are not being written to another directory.
Another reason could be some other user/process could be issuing a kill -9 that could kill tomcat without giving it any chance to log errors.
another possibility is that some process was started this morning on the box that is binding to a port that your server requires.
Are your servlets or one of it's dependencies allowed to call System.exit()? (Not sure how locked down Tomcat VMs are in that sense)
I've had developers thinking it's ok to use exit(666); on detecting a non-invertable matrix (which isn't good, but sure as heck not fatal). Arrgh. Perhaps you have some similar culprit in your system?
I noticed CATALINA_OPTS in my path and that was set for a lower JVM size. Hence, the crash and no log trace of tomcat was found. The server automatically shutdown in less than 2 hrs.
check, CATALINA_OPTS or JAVA_OPTS -- these might have jvm settings. either increase them or comment them out and increase the swap memory.
“The Service on local computer started and then stopped ,Some services stop automatically if there are not in use by other services or programs.”
I gone through the problem i have tried so many ways to get out of the problem finally i got the solution as follows.
1) Click Run Command from start button.
2) Enter Services.msc then click OK,you will get all the services in your computer.
3) Select your service and right click on the service and select Properties
4) Goto Logon Properties and select Local System Account then click OK.
This will work.
Sometime it happens if some other program is running on the same port. For example Skype. Shut down that program before you start Tomcat.
try to clean your elipse projects because you could have tried to add another server which used port 8080 then when you try to execute the tomcat server externally that defaulty uses port 8080 the tomcat server automatically shutdowns after cleaning the project copy the new war file and paste it in bin it works fine
conclusion: when the server tries to use the port which has already been acquired you will see such type of issues.