We Keep Coding

Tomcat process not closing

i have problem with tomcat not closing after running shutdown.sh. I have profiled the server using the Jconsole and made an thread dump. Normally server runs about 120 processes, and if it doesn't close after running shutdown.sh it keeps about 30 threads. This happens randomly, mainly if Tomcat is running for longer time. We are using tomcat 8.5.28, I'm suspecting Google Firebase client for that behaviour. Maybe you have some idea what may cause the problem? Or have clues what can i do to try finding solution?
Thread dump:
http://textuploader.com/dzjc5

Follow the steps mention in the link.
http://faizakram.com/blog/kill-tomcat-service-running-on-any-port/

Tomcat8 kills my threads on shutdown

I created a webapplication that needs to do some cleanup on shutdown. This cleanup will take about a minute and its completely OK for it to do so.
When I deploy my webapp onto Tomcat 8 and then stop it, my ContextListener gets called and the cleanup begins. But it seems like Tomcat stops my thread the hard way and it won't complete anymore. At least on Tomcat 6 that wasn't an issue.
An ideas how to configure Tomcat 8 to stop from misbehaving?
Partial Answer:
I found out it has something to do with a performance optimization I did. I used startStopThreads="2" to start my applications in parallel, which works out well, but on shutdown this also seems to kill my threads.

If you have a task which is to be performed on shutdown, I would add this as shutdown hook. Most likely Tomcat 8 is called System.exit() which is a normal thing to do and this kills all user threads but start shutdown hooks.
A better solution is to never leave the system in a state where you really need this. i.e. you cannot assume an application will die gracefully.
if you are waiting for client to disconnect, I suggest you add a shutting down phase. During this phase you refuse new connections, move connections to another server or attempt to gracefully tell existing ones you are going away. After a short period or time out, you then shut down.

My application is stucked when DB is down

I'm using Hibernate and Tomcat JDBC connection pool to use my MySQL DB.
When, from any reason, the DB is down, my application got stuck.
For example, I have REST resources (with Jersey), they are not getting any requests.
I also using quartz for schedule tasks, they aren't running as well.
When I start my DB again, everything goes back to normal.
I don't even know where to start looking, anyone has an idea?
Thanks

What must be happening is your application must be receiving request but there must be some exception while establishing database connection .see the logs.
try some flow where your are not doing any database operation. It must work fine.

When the application has hung, get a thread dump of the JVM, this will tell you the state of each thread and, rather than guessing as to the cause, you'll have concrete evidence.
Having said that, and going with the guess work approach, a lot comes down to how your connection pool is configured and the action your code takes when it receives the SQLException. If the application is totally hung, then my first port of call would be to find out if the db accessing threads are in a WAIT state, or even deadlocked. The thread dump will help you to determine if that is the case.
see kill -3 to get java thread dump for how to take a thread dump

A web application appears to have started a thread named [22] but has failed to stop it. This is very likely to create a memory leak

I have a web application with Servlets in the back end deployed over tomcat. The application is simple java application.
I see this error frequently in the server logs:
SEVERE: A web application appears to have started a thread named
[22] but has failed to stop it. This is very likely
to create a memory leak.
Are there any potential reasons which might be causing it?

I'd use visualvm 1.3.2 and see what threads are being created. Be sure to add all the plug-ins.
If it's not being done by your code you won't have much control over it.
You also don't know if the message is a red herring or not. Load test your code over a period of time and measure what happens.

I faced similar situation recently Resolved in below steps
I took Thread dump. ( using Kill -QUIT pid )
Found the Runnable/Thread class form dump
Then i put a debug point in run method and started application in
debug mode.
Got the code which starts My Thread and I observed it was not
stopped while stoping application.
Introduced code to stop the thread in contextDestroyed method of
AppContextListener (This is My application class which extends
ServletContextListener ) as this method will be called when i stop
tomcat.
If you set Thread as Dameon Thread it is not going to help , you can visit explanation.

Tomcat waits for all the application's threads (user threads not daemon threads) to stop before it goes down, I guess that in your case this specific thread is a user thread and therefore tomcat generated this error.
I suggest you to change this thread to daemon (assuming this one is yours)

How to integrate memcached in a Servlet? Tomcat & memory leaks

Searching memcached java in google, the first result is Using Memcached with Java.
The guy (who calls himself Just some Random Asshole in the Internet!) proposes a Singleton based on net.spy.memcached. It basically creates 20 threads and connections by creating 20 instances of MemcachedClient. For every request it chooses one at random.
However those threads and connections are never closed and they pile up every time I hot swap the application during development (with warnings from Tomcat 7).
SEVERE: The web application [/MyAppName] appears to have started a thread named
[...] but has failed to stop it. This is very likely to create a memory leak.
By looking at MemcachedClient JavaDoc, I see a method called shutdown with the only description being "Shut down immediately." Shut down what? The client? The server? I suppose is the client, since it's in MemcachedClient and I suppose that this method would close the connection and terminate the thread. EDIT: yes, it shuts down the client.
Question 1 How to force the execution of cleanup code in Tomcat 7, before the application is hot swapped?
Question 2 Is this approach of using memcached (with cleanup code), correct or is better I start over in a different way?

I think creating 20 memcache clients is silly - that's like creating 20 separate copies of your DB connection pool. The idea with that client is that it multiplexes a variety of requests with asynch IO.
http://code.google.com/p/spymemcached/wiki/Optimizations
As far as shutting it down, simply call:
yourClient.shutdown() to shutdown immediately, or
yourClient.shutdown(3, TimeUnit.SECONDS) for example, to allow some time for a more graceful shutdown.
That could be called from your Servlet's .destroy method, or a context listener for your whole WAR.

I don't know anything about memcached, but you could probably write a custom context listener and put some kind of shutdown hook in the context listener so that when the context shutdown you could loop through the items in your singleton and shut them down.

It turned out that it was a bug of Java AWS SDK and was not related to memcached. Version 1.2.2 of Java AWS SDK has this bug fixed.