Jboss server was throwing an Exception all of a sudden "You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null". No changes made to the datasources, prior to this. Everything got normal after a server bounce...
What are all the possibilities for this?
After digging into this issue further, we found there was a dependency which was not responding, that increased the thread count.There was also memory leak. Due to this server got restarted by itself... From the logs it was clear the server got restarted and that could be the reason this exception. As mentioned in the above comments the search pages from the google shows that this exception could be thrown when there is issue loading the datasources... Everything was fine after bouncing the server... Thanks all...
Related
i have problem with tomcat not closing after running shutdown.sh. I have profiled the server using the Jconsole and made an thread dump. Normally server runs about 120 processes, and if it doesn't close after running shutdown.sh it keeps about 30 threads. This happens randomly, mainly if Tomcat is running for longer time. We are using tomcat 8.5.28, I'm suspecting Google Firebase client for that behaviour. Maybe you have some idea what may cause the problem? Or have clues what can i do to try finding solution?
Thread dump:
http://textuploader.com/dzjc5
Follow the steps mention in the link.
http://faizakram.com/blog/kill-tomcat-service-running-on-any-port/
This is my first time asking a question on Stack Overflow. I recently configured an Ubuntu 16.04 virtual private server to host a web application. I run ngnix on a Tomcat server that reads and writes to a MySQL database. The application runs fine except for the fact that Tomcat restarts itself once in a while which results in a 500 error that stems from a "broken-pipe" when anyone tries to login (i.e. make a connection to the database).
I will post an image of the 500 next time it happens. I went into my vps and looked at my Tomcat restart message. This is what I see: Tomcat status message.
I also did a little diving into the Tomcat logs and this is a log file that corresponds with that restart time: Tomcat log file
I did some research to try and solve this myself, but with no success. I believe that the exit=143 is the process being terminated by another program or the system itself. I also have done some moving of the mysql-connector-java.jar. I read that it should be located in the Tomcat/lib directory and not in the WEB-INF of the web application. Perhaps I need to configure other settings.
Any help or any direction would be much appreciated. I've fought this issue for a week with having learned much, but accomplished little.
Thanks
Look at the timeline. It starts at 19:49:23.766 in the Tomcat log with this message:
A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Exit code 143 is a result of that shutdown and doesn't indicate anything.
The question you need answered is: Who send that shutdown command, and why?
On a side note: The earlier messages indicates that Tomcat lost connection to the database, and that you didn't configure a validation query. You should always configure that, since database connections in the connection pool will go stale, and that needs to be detected.
Theory: Do you have some monitoring service running that tests your application being up? Does that monitoring detect a timed-out database connection, classify that as a hung webapp and auto-restart Tomcat?
While I don't think I am able to see to the core of the problem you have with your overall setup given the small excerpt of your log files, one thing strikes the eye. In the Tomcat log, there is the line
A valid shutdown command was received via the shutdown port. Stopping the server instance.
This explains why the server was restarted. Someone (some external process, a malicious attacker, script, or whatever. Could be anything depending on the setup of your server) sent a shutdown command to Tomcat's shutdown port (8005 by default) which made the Tomcat shut down.
Refer to OWASP's recommendations for securing a Tomcat server instance for fixing this possible security whole.
Regarding the ostensible Hibernate problems you have, I don't get enough information from your logs to make a useful statement. But you can leave the MySQL jar in Tomcat/lib, since this is not the root cause of your problem.
So we run a Hibernate, Spring, Spring Webflow stack. From what I've read so far it might also be important to know we use c3p0-0.9.1.2.
Over the last couple of days we've noticed the server suddenly stop. Users cannot log into the website, nothing appears to happen, the browser simply sits loading the page forever. The server logs also simply halt.
When we notice this we shutdown the tomcat instance and all of a sudden quite a few of the following errors get logged;
13:05:57.492 [TP-Processor7] WARN o.h.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
13:05:57.492 [TP-Processor7] ERROR o.h.util.JDBCExceptionReporter - An SQLException was provoked by the following failure: java.lang.InterruptedException
Any ideas what these mean? Google hasn't been too helpful. Are we leaking db connections somewhere and the pool cannot gain a new session?
We have just put in a couple new Spring Webflow flows and are experiencing a slightly increased amount of website traffic but we haven't seen this behaviour before.
I suspect those InterruptExceptions come from the actual shutdown of those threads by the container, and simply indicate that those threads are existant when Tomcat shuts down.
Instead, I would grab a thread dump from Tomcat when it next freezes. I would also get a DBA to tell you what's happening in the database. From the above I'm guessing you're hung on a database resource, but a thread dump and analysis from a DBA will certainly point you in the right direction.
Here's a Thread Dump JSP as an alternative means of generating thread dumps.
We have a mysterious error in our Tomcat and webapps log. A Java web application is running on Tomcat 6, uses Oracle 11g database. All request are logged. We use Commons DBCP for database connection pooling. One feature of our application is that all connection are request-scoped. We implemented that with Spring's SmartDataSource. Releasing connections is wrapped in try/finally, so even if there's an error during the request, the connection gets released. This works well the whole time we tested it.
One day our server returned Error 503 "Service unavailable". In the logs we only found one exception: org.apache.commons.dbcp.AbandonedTrace$AbandonedObjectException. So it seems that Common DBCP abandoned connection collector found an abandoned connection and reported it. It is not error by itself, it is a suggestion that an error is somewhere else.
The trace tells us the exact time and code when the connection was acquired. We explored that time in logs and there was a request that began but never ended. This would explain why the connection was not released, but what bothers us is how try/finally could have been abrupted?
I think this is a Tomcat issue, because there was no other exception in our application logs and because the error code that the server returned was not the ususal Error 500 "Internal server error".
Does anyone have any suggestions why this could happen? It is possible that Tomcat interrupts a thread so that try/finally are ignored?
I have a web-application built with GWT (2.0.3) and run on Apache Tomcat 6.
My application uses long polling to enable client-server conversations.
When a client is unable to connect to the server it displays a disconnected message on the page and grays out the controls until it is able to resume conversation with the server.
This happens through the use of the onFailure method of the rpc services; I keep track on how many consequtive exceptions I've received and if it passes a defined threshhold the above scenario happens.
This allows notifying the user of a problem while in the background continuing to resume the server conversation.
This has been the configuration for about 6 months, and without a problem.
I compiled the application after a change and wanted to see it in stand-alone mode so I started up tomcat (not via eclipse) and everything seemed to work fine.
When I ctrl+c'd the apache (while having clients up) I saw the clients displaying a 503 error instead of my app with the disconnected message.
I then tried to reproduce the issue but was unable as the next times the app behaved as expected.
I'm not sure if it's relevant but recently I added an UncaughtExceptionHandler to my module's onModuleLoad.
Has anyone encountered such an issue?
Do you know how I can make my client immune to such an issue?
Thanks a lot,
Ittai
Probably your app tried to connect to server while it was in process of shutting down. Some of the services might have already shut so the request failed with internal server error.
I've got similar issue having an apache httpd in front of the tomcat and stopping tomcat while one of the "background" async requests were being made, due to the security redirection policy the failing request ends redirecting the browser and voilà our 503 error page.