I have a problem with a Tomcat server that is unable to shutdown gracefully. I have taken a thread dump after I issued the shutdown command, and it looks like this:
http://pastebin.com/7SW4wZN9
The thread which I believe is the "suspect" that does not allow the VM to shut down is the one named "pool-4-thread-1". The rest of them are either daemon threads or internal VM threads. While trying to find out what this thread is for, I noticed that there are other java programs out there that create threads with similar names (For example, JVisualVM creates such threads).
So I'm wondering if someone else knows what this thread is and how it can be created.
These threads are probably created by an ExecutorService that you created in your code somewhere (directly or indirectly through a library) and that needs to be shutdown (for example in a ServletContextListener).
Related
The question
The software runs on one DELL server with Linux.
Language could be C++, JAVA or Python.
Both thread A and thread B assign tasks to the service thread. When receiving tasks, service thread will put the tasks on its own task queue. When thread is free, it will execute the tasks and return task results to thread A or thread B depending on who sent request.
Thread A has a higher priority than thread B.
My thoughts
It is very similar to server/client in socket programming. However, since this software runs on one same server, TCPIP does not seem a good solution to me.
Another thought is to use a common database for this, such as redis. But Redis runs also on TCPIP and I am not sure if this could be a good fit.
Someone also suggests using a service DLL and both Thread A and Thread B can invoke service DLL directly. However I do not have experiences building DLL simultaneously serving several threads. Is this possible?
My question is: how to achieve this in a suitable way?
I am trying to test an application with Jmeter. The application uses a proprietary library that creates multiple threads. In JMeter I have created a AbstractJavaSamplerClient, which does not seem to wait for all the other threads that might be generated in the application. Instead it just runs its own default method and closes leaving the other threads running in the background - since I am connecting to a server in the application, I can see through the server logs that it is still connected. Since I don't have the references to the other threads as they are instantiated through the proprietary library, I can't use the common solutions of wait() or join().How can I get the main thread to wait for all the threads (none of which I have references too)?
Put all work with the library in a separate thread in a specially created thread group. The library will create new threads in that thread group and its descendants. List all threads of that group recursively with group.enumerate(new Thread[group.activeCount()*2],true). Then you can join() them all.
You can start with
Thread.getAllStackTraces().keySet();
which will give you a set of all running threads. This will include those you are interested in and all the other threads, including those internal to the JVM.
Hopefully you'll be able to filter out the ones you are interested in by name, and then you can join() them.
I m using windows 7 OS. I have around 6 threads in my application. For the purpose of testing the alerts to check the health of the threads, i need to kill the threads manually and check if the alerts are working properly. Can we kill a thread like how we kill a process with its pid?
Dan Woods documented how to kill a thread in this blog entry...
https://web.archive.org/web/20160302023213/http://www.rhcedan.com/2010/06/22/killing-a-java-thread
The steps he performed involved using a debugger (JDB) and injecting an exception in the thread's execution. Specifically...
Ensure that your java program is started with the following parameters:
-Dcom.sun.management.jmxremote.port=50199
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Xrunjdwp:transport=dt_socket,address=50100,server=y,suspend=n
This will allow us to attach the java debugger to the running process, after
we identify which Thread is causing the problem. Also, make sure that
you have your iptables setup appropriately so as to only allow
connections on 50100 and 50199 from the hosts/workstations that you manage.
Identify the offending thread:
Kill the thread. In this example, the ThreadName is “btpool0-0?. Fire up the java debugger (also shipped with the JDK distribution), and attach to the running JVM…
[root#host ~]# jdb -attach 50100
Get a list of the running threads — this will also give us the thread id as the JVM sees it:
> threads
--snip--
(org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cb
btpool0-0 running
--snip--
The thread id that we’re going to kill is “0x25cb”. The first step of killing the thread is to jump into it, and suspend it…
thread 0x25cb
btpool0-0[1] suspend 0x25cb
btpool0-0[1] step
Step completed: <... snip ...>
btpool0-0[1] kill 0x25cb new java.lang.Exception()
killing thread: btpool0-0
btpool0-0[1] instance of
com.site.package.name(name='btpool0-0', id=9675) killed btpool0-0[1]
Exit the java debugger, and you’re done!
There is no safe way to "kill" a thread without killing the process it is in. It not something you would do deliberately. For testing purposes I would add code to your application to support this.
It's not true. You can always attach to the JVM process with GDB and do a call pthread_kill if you know the thread id. You only need to translate from the java thread dump (do a kill -3) which gives you a hex id, (native id), then look into the list of threads in GDB (info threads) and locate the real thread id.
This is proven to work.
As Peter says, you can't do this safely. Indeed on some platforms Thread.kill is not even implemented. However:
If this is just for testing, a unit test that called Thread.kill would be reasonable ... assuming it worked on the test platforms where it needed to work. (A "loud" comment in the source code would be in order to help people porting the unit test ...)
Another alternative is to add some code to the thread runnable that allows your unit tests to tell it to die. If the thread code needs to be (almost) production code for this to work, you could create a subclass that overrides something so that it "breaks" in a way that suits your purposes ... for testing. In fact, this approach allows you to cause the threads "break" in controlled ways, potentially allowing you to test different aspects of your alerting code.
You can't do it from outside (OS or debugger), you'll have to write your own Thread watchdog that can interact with the user and kill the thread you want.
Try to look here for how to handle signals with java
In java you can not kill the like unix . Either you can interrupt the tread in java or you can kill the process in unix .
Wait for some time in the thread and kill the thread in the code - simple way.
As mentioned in a previous post by buzz3791, it works by using jdb. However the change that I noticed is, you can't kill the thread, but you can interrupt or suspend the thread.
#jdb -attach 50100
threads --This will show all threads running on the jvm under Groups and Reference Handler section.
Groups
Reference Handler
:(com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary)0x36c1:
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)
thread 0x36c1-- This will be the thread id that can be picked from one of the threads in the thread that you wish to kill/interrupt and run this interrupt 0x36c1
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)[1] interrupt 0x36c1
you can try multiple times the same interrupt command and if it is already interrupted, it will show that the thread id is invalid. Thus you know that the thread is killed, this can be verified by looking at the stack trace and confirmed.
Tested this on the OrientDB database server with jdk 8 and it works.
is there a way to determing if a jvm is shutting down normally? Shutdown hook can only spawn a thread, is there a way to determine if the JVM is existing normally or abnormally at that time?
You could write a file on startup and delete it again on graceful exit. If the JVM is gone but the file is still there you know that it crashed or has otherwise exited in a unintended manner.
I remembered a similar question being asked time ago. One possible course of action is the use of SignalHandler.
You can read the full article here. It appears to be related to IBM JVM but I think it is equally valid for Java Hotspot.
A little-known feature of Java is the
ability of an application to install
its own signal handler, which is
supported through the sun.misc.Signal
class. However, use caution when using
classes from the sun.misc package
because it contains undocumented
support classes that may change
between releases of Java. You can
install a Java handler for any signal
that is not used by the JVM. These
signal handlers are similar to native
handlers because they're invoked when
a native system signal is raised, but
they will always run as a separate
Java thread. Essentially, when a
signal is raised for which a Java
signal handler is available, the JVM's
"signal dispatcher thread" is woken up
and informed of the signal. The signal
dispatcher thread then invokes a Java
method to create and start a new
thread for the installed Java signal
handler. To write a Java signal
handler, define a class that
implements the sun.misc.SignalHandler
interface and register the handler by
using the sun.misc.Signal.handle()
method.
Check return sttaus using command $?
When the JVM is shutting down normally, this means that the main thread has ended. If the JVM is shutting down for some other resaon (e.g. the user pressed Strg+C), the main thread is still running. So you can store a reference to the main thread in your shutdown hook and check whether this thread is still alive. Of course this assumes that the main thread will normally be the last running thread in your application. I don't know how the situation is if one of the threads called System.exit(), but you could easily find this out.
i saw comment like this
one place i have seen this problem is if you keep creating threads, and instead of calling start(), call run() directly on the thread object.
This will result in the thread object not getting dereferenced...
So after sometime the message unable to create new native thread comes up
on the Sun Java Forums
In my application, initialy we plan to use thread, but later, we decided no need anymore, so we just call run() instead of start(). Do we need to do manual GC for new threadClass(..) ?
my tomcat startup setting
-Xms1024m -Xmx1024m -XX:MaxPermSize=450m
Why do you create a Thread in the first place?
Your code should implement the Runnable interface instead.
Then, when you decide that you want to run it in a thread, simple instantiate a Thread with the Runnable as the argument and call start() on the Thread object.
If, instead, you just want to run it in your current thread, simply call run() on your Runnable object.
This has several advantages:
you don't involve any Thread objects as long as you don't care about separate threads
your code is wrapped in a Runnable which fits closer conceptually: you're not writing some special kind of Thread, do you? You simply write some code that can be executed/run.
you can easily switch to using an Executor which further abstract away the decision
And last but not least you avoid any potential confusion on whether or not a native thread resource is created.
When you call run() method no new thread should be created. And your objects will be collected by Garbage collector when they are not referenced.
Your other part of code may be creating lot of Threads.
Try using ThreadPoolExecutor (thread pooling) in your code to limit threads in your application, And tune your threadpool size accordingly for better performance.
You can also check following to debug your issue: (referenced from link)
There are a few things to do if you encounter this exception.
Use the lsof -p PID command (Unix
platforms) to see how many threads
are active for this process.
Determine if there is a maximum
number of threads per process defined
by the operating system. If the limit
is too low for the application, try
raising the per-process thread limit.
Examine the application code to
determine if there is code that is
creating threads or connections (such
as LDAP connections) and not
destroying them. You could dump the
Java threads to see if there are an
excessive number has been created.
If you find that too many connections
are opened by the application, make
sure that any thread that the
application creates is destroyed. An
enterprise application (.ear) or Web
application (.war) runs under a
long-running JVM. Just because the
application is finished does not mean
that the JVM process ends. It is
imperative that an application free
any resources that it allocates.
Another solution would be for the
application to use a thread pool to
manage the threads needed.
This link describes quite nicely how this error is thrown by the JVM:
http://javaeesupportpatterns.blogspot.ro/2012/09/outofmemoryerror-unable-to-create-new.html
Basically it's very dependent on the OS. On RedHat Linux 6.5 (most likely other distros/version and kernel versions) the max_threads=max_process x 2.
The max number of threads is very dependent on the number of allowed processes. Which the max number of processes is dependent on the max physical memory you have installed.
If you have a look in the limits.conf file (on my RHL 6.5 it's in /etc/security/limits.d/90-nproc.conf). Exert form the file:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc **1024**
root soft nproc unlimited
You'll see that for non root users it's 1024 (which means 2048 max threads).
To see the max number of threads that your user is allowed to create run this command "cat /proc/sys/kernel/threads-max" or "sysctl kernel.threads-max".
To solve an issue like this (at least it worked for me) as root you'll need to ncrease the max allowed threads:
echo 10000 > /proc/sys/kernel/threads-max
This affects all users and the root. The user needs to log out and then log in again for the settings to take affect.