We Keep Coding

Tomcat restarts with errors (exit 143), runs and then fails after time

This is my first time asking a question on Stack Overflow. I recently configured an Ubuntu 16.04 virtual private server to host a web application. I run ngnix on a Tomcat server that reads and writes to a MySQL database. The application runs fine except for the fact that Tomcat restarts itself once in a while which results in a 500 error that stems from a "broken-pipe" when anyone tries to login (i.e. make a connection to the database).
I will post an image of the 500 next time it happens. I went into my vps and looked at my Tomcat restart message. This is what I see: Tomcat status message.
I also did a little diving into the Tomcat logs and this is a log file that corresponds with that restart time: Tomcat log file
I did some research to try and solve this myself, but with no success. I believe that the exit=143 is the process being terminated by another program or the system itself. I also have done some moving of the mysql-connector-java.jar. I read that it should be located in the Tomcat/lib directory and not in the WEB-INF of the web application. Perhaps I need to configure other settings.
Any help or any direction would be much appreciated. I've fought this issue for a week with having learned much, but accomplished little.
Thanks

Look at the timeline. It starts at 19:49:23.766 in the Tomcat log with this message:
A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Exit code 143 is a result of that shutdown and doesn't indicate anything.
The question you need answered is: Who send that shutdown command, and why?
On a side note: The earlier messages indicates that Tomcat lost connection to the database, and that you didn't configure a validation query. You should always configure that, since database connections in the connection pool will go stale, and that needs to be detected.
Theory: Do you have some monitoring service running that tests your application being up? Does that monitoring detect a timed-out database connection, classify that as a hung webapp and auto-restart Tomcat?

While I don't think I am able to see to the core of the problem you have with your overall setup given the small excerpt of your log files, one thing strikes the eye. In the Tomcat log, there is the line
A valid shutdown command was received via the shutdown port. Stopping the server instance.
This explains why the server was restarted. Someone (some external process, a malicious attacker, script, or whatever. Could be anything depending on the setup of your server) sent a shutdown command to Tomcat's shutdown port (8005 by default) which made the Tomcat shut down.
Refer to OWASP's recommendations for securing a Tomcat server instance for fixing this possible security whole.
Regarding the ostensible Hibernate problems you have, I don't get enough information from your logs to make a useful statement. But you can leave the MySQL jar in Tomcat/lib, since this is not the root cause of your problem.

Tomcat debugging not possible through eclipse

I've made a script called debug.sh and placed it under the bin directory (start it with ./debug.sh) to start Tomcat 8 in debugging mode:
set JPDA_ADDRESS=8000
set JPDA_TRANSPORT=dt_socket
set JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
bash catalina.sh jpda start
But if it started, there is now message which says, that Tomcat is listening on port 8000. Also if I type
netstat -nat
there is no application listening on port 8000.
What exact configuration do I have to set, to remote debug my Tomcat 8 server which is running on a specific IP or do I have just a little problem in my script?

This answer has been updated following the comments, I did not understand the problem in the first place.
I guess you have followed that doc: this is about developing Tomcat itself.
I am not sure you are using the proper way to configure the port (I don't know your specific configuration details). In a standard environment, the ports are configured in the server.xml (note that several different ports are used by Tomcat for the different services).
To remotely monitor your server, you should use a JMX client. As far as I know, Eclipse doesn't include one (or at least not one documented) - you could code one as this is a Java specification (JSR262). You have one in a standard Java environement (JConsole). By default, JMX is not enabled on Tomcat. If you need to enable it, the fine way is to follow the doc.
Tomcat JMX monitoring and JConsole are both available in most versions of Tomcat and the Java runtime.
The following exchange seems to be about your problem.

How do I debug Java servlet filters, on-the-fly, from a remote machine (I'm using Eclipse and Apache Tomcat 5.5)

I wrote a Java servlet filter on my local machine and deployed it a remote (machine) web server. Unfortunately, it's been very difficult and time-consuming trying to trace errors reported by Apache Tomcat 5.5, my JSP/servlet engine. I can't keep writing System.out.println(...), saving, deploying, testing JSP pages, and so on. It's taking too long. There has to be a better, faster way.
Is it possible to remotely debug servlet filters? I don't have a web server on my local machine, which is why I'm asking about remote debugging. Specifically, I'm looking for a way to debug, line-by-line, the servlet filter, on-the-fly, as it's happening on the remote web server. Does such a method exist?
Or, is there a better method than writing to standard output. It's taking too long and I feel that must be a more efficient means of debugging Java servlet filters.
Note: I'm using Eclipse for development.
Thank you very much for any help.
Update
Thank you all for your help. I really appreciate it.
I added the JVM argument to Tomcat, restarted Tomcat. Then, on the machine with Eclipse, I entered in the appropriate info in the Debug config, put the breakpoint in, and tested. Unfortunately, it did not work. In the config, I left it as Socket Attach, clicked apply, and that was it. I pressed the debug button and it said the connection was refused. I tried ports 8000 and 8001 and both did not work.
Let me explain what I'm trying to do, that might be better.
I have a login page called login.jsp. On that page, is a form whose action attribute is servlet/LoginServlet. When the user submits the form, it calls servlet/LoginServlet, which is mapped to a class in the web.xml file. Let's call this class com.mysite.mypkg.classA. In class A, it calls a method from another class called com.custom.mypkg.classB. I want to put a breakpoint in classB.
So, using the url with login.jsp page in the Eclipse debugger won't call it. I tried using servlet/LoginServlet and that also did not work.
What should I put in for the URL? Or, do I debug this type of setup?
Thank you.
Update 2
I found this site here, which is pretty comprehensive. I ran netstat -a and noticed that the debug port is not listed. Windows Firewall is turned off, but there could be another thing blocking the port, who knows. Anyway, I placed the VM argument here and it's not working.
Thank you.

For remote debugging you need to start the server in debug mode. There are couple of ways doing that.
1 > start the server using
catinlina.bat jpda start
2 > Add an jvm argument to the tomcat java process
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Once the server is started in debug mode , you need to change the perspective of the project in eclipse to debug.
Then go to Run - > Debug configuration.
Double click remote java application and enter the details such as
Remote IP address
Debug port . Default tomcat debug port is 8000. If you use jvm argument, use the port mentioned in the jvm argument.
Click Apply
Go to the java file you want to debug.
Put a break point in the source code and run the scenario you want to test (Eg Web application using browser)
Also , ensure that the code in the java file is in sync with code deployed on remote server.
Happy Debugging!!!
Peace.
Sanket Raut

You can attach a debugger to a running Tomcat instance, provided that you gave it the right command line options when you launched it.
The Tomcat Development Wiki explains how to do this, and as a bonus gives you instructions on how to set up to debug from the Eclipse or NetBeans IDEs.
Of course, attaching a debugger to a running Tomcat has both security and performance implications*.
* And OH&S issues - you might get badly scratched if you tried this on the wrong kind of tomcat ...

You should run your remote tomcat with the following starup parameter:
bin/catalina.bat jpda start
Then in Eclipse on your local machine go to Run -> Debug Configurations -> Remote Java Application, create new configuration here, use IP of the remote machine as a host and 8000 as a port there
Run this configuration and use the breakpoints in Eclipse for debugging

Continous Deployment for Java/JVM Web Application

My team and I would like to implement "Continuous Deployment" for our website. Continuous Deployment basically means deploying to production very frequently (multiple times a day). Supposedly Etsy does this all the time.
Our environment is Tomcat + Nginx. We already do Continuous deployment of any code change to our snapshot server (ie traditional continuous integration) using Hudson and a Hudson+Cargo plugin that hot deploys.
Surprisingly that works well (albeit over time we have to restart tomcat sometimes).
For production this is not going to work because we can't have the website down.
I have some ideas like having two web apps and redirecting while one is down.
Anybody have any ideas or has done this before in a real production environment?

From http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html:
Real-time alerting. No matter how good your deployment process, bugs can still get through. The most annoying variety are bugs that don’t manifest until hours or days after the code that caused them is deployed. To catch those nasty bugs, you need a monitoring platform that can let you know when things have gone awry, and get a human being involved in debugging them.
To effectively implement continuous deployment to production, you need good monitoring, otherwise you will not understand what is happening with your application.

I don't know WHY you think this is a good idea, but thats up to you.
I would use a balancer application with two hot systems, which can be found in tomcat itself, and then just stop the server before deployment, deploy, and restart the server. Leave a two minute window for each hot server, and you should be good.
EDIT: I wouldn't deploy EVERYTIME. We also are a small company with noch much QA (tm), but it is still one click in the build system to go live.

We use apache httpd 2.2 and mod_proxy for this
We then have 2 tomcats running, one on port 8080, and one on port 88.
The firewall prevents outside access to those ports, so only port 80 is open
Apache HTTPd is configured to listen on port 80
It is very simple to configure too.
This is a basic configuration (httpd.conf) that will work out of the box:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_http_module modules/mod_proxy_http.so
<Proxy balancer://mycluster>
BalancerMember http://localhost:8080
BalancerMember http://localhost:88 status=+H
</Proxy>
ProxyPass / balancer://mycluster/
ProxyPassReverse / balancer://mycluster/
The "+H" means that it is only used as a backup server, so when 8080 is unreachable, it will run on 88 until 8080 comes back online

Tomcat shuts down automatically

I'm using Apache and Tomcat on a Windows server and since this morning, Tomcat stops working without any logs. It doesn't hang, it just shut down.
There's no log in Tomcat, the CPU/Memory are fines, there are no System.Exit in my code.
Anybody ever had this problem?
It happens at random, after 5-10 minutes. The application responds normally and sometime, boom.. stops working.
UPDATE : Still no clue. The Admin team will install the webapp on another box...

My script to start tomcat had last line tail -f catalina.out.
Sometime I did not kill this script, the shell then timed out and killed the script with all child processes, including tomcat.

This sounds like the JVM is crashing. Have you looked for a JVM crash log? It typically has a name like hs_err_pid*.log and is created in the JVM's working directory.
If you find a file like this and upload it, then we can probably help more.
Some questions:
Have you recently changed the version of Java you are using?
What is the exact version of Tomcat you are using?
Are you using Tomcat Native (the Apache Portable Runtime)?

Faced this issue recently.
Scenario : Tomcat started successfully but automatically gets shut down after 1 hour and sometimes this happened after 1 day and nothing is there in tomcat logs.
Issue : Actual issue was high memory usage and no free SWAP memory.
How I found the solution
If tomcat don't show any logs, then there must be something in system logs so, I checked /var/log/messages but since permission denied for me I tried /var/log/dmesg and got this
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have taken successfully and the processes were not killed even after the shutdown due to some reason, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)

Recently I had this problem, If somebody faces the same issue in future I hope this will help.
Scenario: Tomcat shuts down without any logs or errors
Root Cause for my problem: synchronized method accessed from a task using TimerTask
I had a singleton class with a synchronized method accessed from various threads based on timer or user action
some times this method will take up to few minutes to complete. When TimerTask is waiting on this method for sometime (I guess timer is timed out /thread is killed or something is happening in the background) and the moment the lock on the method is released the tomcat is getting killed.
So I removed synchronized keyword and removed singleton and made some code changes for thread safety. Then the problem is gone.
How I found out: I had a log statement in the first line of synchronized method and everytime the tomcat shutdowns i found this message in the last few lines.
Regards,
Phanindra Kasturi

things to look for in debugging an issue like this:
Look at the logs directory ($TOMCAT_ROOT/logs) to make sure none of the log files have any stack traces
Look at the tomcat startup script to make check the location of the log files to see if the logs are not being written to another directory.
Another reason could be some other user/process could be issuing a kill -9 that could kill tomcat without giving it any chance to log errors.

another possibility is that some process was started this morning on the box that is binding to a port that your server requires.

Are your servlets or one of it's dependencies allowed to call System.exit()? (Not sure how locked down Tomcat VMs are in that sense)
I've had developers thinking it's ok to use exit(666); on detecting a non-invertable matrix (which isn't good, but sure as heck not fatal). Arrgh. Perhaps you have some similar culprit in your system?

I noticed CATALINA_OPTS in my path and that was set for a lower JVM size. Hence, the crash and no log trace of tomcat was found. The server automatically shutdown in less than 2 hrs.
check, CATALINA_OPTS or JAVA_OPTS -- these might have jvm settings. either increase them or comment them out and increase the swap memory.

“The Service on local computer started and then stopped ,Some services stop automatically if there are not in use by other services or programs.”
I gone through the problem i have tried so many ways to get out of the problem finally i got the solution as follows.
1) Click Run Command from start button.
2) Enter Services.msc then click OK,you will get all the services in your computer.
3) Select your service and right click on the service and select Properties
4) Goto Logon Properties and select Local System Account then click OK.
This will work.

Sometime it happens if some other program is running on the same port. For example Skype. Shut down that program before you start Tomcat.

try to clean your elipse projects because you could have tried to add another server which used port 8080 then when you try to execute the tomcat server externally that defaulty uses port 8080 the tomcat server automatically shutdowns after cleaning the project copy the new war file and paste it in bin it works fine
conclusion: when the server tries to use the port which has already been acquired you will see such type of issues.