How can I suspend the execution of a JVM for a configurable amount of time, similar to want happens on a full, serial, Garbage Collection? I want to test some edge cases but it's difficult to create the exact pauses I need with manually generating garbage + System.gc().
I'm trying to 'freeze' all the threads in the application, to simulate a situation where an application becomes unresponsive, and then resumes execution. The application is part of a cluster, and I'm trying to debug some leave / join / re-join problems.
JVM suspend can be done from command line with jdb:
jdb -attach 8787
Initializing jdb ...
> suspend
All threads suspended.
> resume
All threads resumed.
> exit
But this requires it to be started with -Xdebug ...
There is also universal way to suspend and resume any process on Linux/Unix:
kill -SIGSTOP PID
kill -SIGCONT PID
See also How to suspend/resume a process in Windows?
When debugging from Eclipse you can supsend all threads, I guess other debuggers allow to do that too. So you'd need to start your server with JVM debug options and remote connect to it.
In my case (running JBoss) I modify the startup script by adding this line:
set JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n %JAVA_OPTS%
Then in Eclipse, Run -> Debug Configurations -> Remove Java Application -> New
Normally you just need to provide the hostname and the port.
Use a Virtual Machine and suspend its execution outright.
I think you probably want something that is less invasive, but one approach is to have a controlling thread obtain a lock and then have all the other threads also try to obtain that lock. Then have the controlling thread sleep for the time you need before giving it up. A slightly less clumsy version of this would be to use a semaphore with one permit per thread and then have the controlling thread obtain them all, before having all the other threads try to acquire a permit.
As I said that's pretty invasive you'd have to hack that into your code for each thread.
Another approach would be to run your code under a debugger and manually suspend each thread.
Related
we've got a slightly grown spring webapp (on tomcat 7) that is very slow in shutdown. (which has negative impacts on the performance of our continous delivery)
My suspicion is, that there must be some bean that is blocking (or taking very long) in it's #PreDestroy method.
So far I've ensured that it's not related to a thread(pool) that is not shut down correctly by giving distinct names to every pool, thread and timer and ensuring that they are either daemon threads or being shut down correctly.
Has anybody every solved a situation like this and can give me a hint on how to cope with this?
BTW: killing the tomcat process is not an option - we really need a clean shutdown for our production system.
Profiling would be the nuclear option. It's probably easy to get a picture of what's happening (especially if it is just blocked threads since that state will be long lived) just using thread dumps. If you take 2 dumps separated by a few seconds and they show the same or similar output for one or more threads then that is probably the bottleneck. You can get a thread dump using jstack or "kill -3" (on a sensible operating system).
and if you're on Windows, then selecting the java console window, and hitting ctrl + pause will dump to that window - just hit 'enter' to resume execution
I m using windows 7 OS. I have around 6 threads in my application. For the purpose of testing the alerts to check the health of the threads, i need to kill the threads manually and check if the alerts are working properly. Can we kill a thread like how we kill a process with its pid?
Dan Woods documented how to kill a thread in this blog entry...
https://web.archive.org/web/20160302023213/http://www.rhcedan.com/2010/06/22/killing-a-java-thread
The steps he performed involved using a debugger (JDB) and injecting an exception in the thread's execution. Specifically...
Ensure that your java program is started with the following parameters:
-Dcom.sun.management.jmxremote.port=50199
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Xrunjdwp:transport=dt_socket,address=50100,server=y,suspend=n
This will allow us to attach the java debugger to the running process, after
we identify which Thread is causing the problem. Also, make sure that
you have your iptables setup appropriately so as to only allow
connections on 50100 and 50199 from the hosts/workstations that you manage.
Identify the offending thread:
Kill the thread. In this example, the ThreadName is “btpool0-0?. Fire up the java debugger (also shipped with the JDK distribution), and attach to the running JVM…
[root#host ~]# jdb -attach 50100
Get a list of the running threads — this will also give us the thread id as the JVM sees it:
> threads
--snip--
(org.mortbay.thread.BoundedThreadPool$PoolThread)0x25cb
btpool0-0 running
--snip--
The thread id that we’re going to kill is “0x25cb”. The first step of killing the thread is to jump into it, and suspend it…
thread 0x25cb
btpool0-0[1] suspend 0x25cb
btpool0-0[1] step
Step completed: <... snip ...>
btpool0-0[1] kill 0x25cb new java.lang.Exception()
killing thread: btpool0-0
btpool0-0[1] instance of
com.site.package.name(name='btpool0-0', id=9675) killed btpool0-0[1]
Exit the java debugger, and you’re done!
There is no safe way to "kill" a thread without killing the process it is in. It not something you would do deliberately. For testing purposes I would add code to your application to support this.
It's not true. You can always attach to the JVM process with GDB and do a call pthread_kill if you know the thread id. You only need to translate from the java thread dump (do a kill -3) which gives you a hex id, (native id), then look into the list of threads in GDB (info threads) and locate the real thread id.
This is proven to work.
As Peter says, you can't do this safely. Indeed on some platforms Thread.kill is not even implemented. However:
If this is just for testing, a unit test that called Thread.kill would be reasonable ... assuming it worked on the test platforms where it needed to work. (A "loud" comment in the source code would be in order to help people porting the unit test ...)
Another alternative is to add some code to the thread runnable that allows your unit tests to tell it to die. If the thread code needs to be (almost) production code for this to work, you could create a subclass that overrides something so that it "breaks" in a way that suits your purposes ... for testing. In fact, this approach allows you to cause the threads "break" in controlled ways, potentially allowing you to test different aspects of your alerting code.
You can't do it from outside (OS or debugger), you'll have to write your own Thread watchdog that can interact with the user and kill the thread you want.
Try to look here for how to handle signals with java
In java you can not kill the like unix . Either you can interrupt the tread in java or you can kill the process in unix .
Wait for some time in the thread and kill the thread in the code - simple way.
As mentioned in a previous post by buzz3791, it works by using jdb. However the change that I noticed is, you can't kill the thread, but you can interrupt or suspend the thread.
#jdb -attach 50100
threads --This will show all threads running on the jvm under Groups and Reference Handler section.
Groups
Reference Handler
:(com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary)0x36c1:
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)
thread 0x36c1-- This will be the thread id that can be picked from one of the threads in the thread that you wish to kill/interrupt and run this interrupt 0x36c1
OrientDB (/xxx.xxx.xxx.xxx:123) <- BinaryClient (/xxx.xxx.xxx.xxx:678)[1] interrupt 0x36c1
you can try multiple times the same interrupt command and if it is already interrupted, it will show that the thread id is invalid. Thus you know that the thread is killed, this can be verified by looking at the stack trace and confirmed.
Tested this on the OrientDB database server with jdk 8 and it works.
It is possible to kill a thread that is in state RUNNING in a non programatically way?
I know that top command in *nix can show threads. Can I kill the thread in the OS?
I'd like to know if there is a way to link a thread to a process so I can kill only that specific thread and not the application.
We had a bug in our code that kept a thread in state RUNNING in a synchronized method. The thread kept the lock on the object "hanging" the application.
The bug is fixed. But I wonder if is possible.
The short answer is "maybe, but you should not and most of the time it won't work either".
The long answer is:
"Maybe..."
Some JVM implementation map java threads to OS threads and some do not. If the JVM does a mapping to a native OS thread, you might be able to kill that thread with some process tool that the OS provides (like kill on *nix). If the JVM does green threads, meaning it doesn't map a Java thread to an OS level thread, then you are basically out of luck using OS level tools. Luckily only very few JVM implementations do this. An approach that can be used regardless in which way the JVM organizes it's threads, is using the java debugger. This article describes the procedure of doing it: http://www.rhcedan.com/2010/06/22/killing-a-java-thread/.
"but you should not do it"
Killing a thread on the OS level will almost certainly leave the JVM in an undefined state (read "jvm might crash or delete all files on your disk or do whatever it fricking pleases to do"). Even when going the debugger way, only a very small amount of java applications (read "no application made on this planet") will properly handle the event that an outside application is killing one of it's threads. As a result these applications will be put in an undefined state (read "application might crash or delete all files on your disk or do whatever it fricking pleases to do").
"and most of the time it won't work either"
If the thread is really stuck with some blocked IO etc, then killing the thread won't work, it will just not respond. If a program is stuck it's probably better to kill the whole program, find the issue with the program and fix it instead of killing a single thread.
For all your doubts on killing a thread, refer this:
http://download.oracle.com/javase/1.4.2/docs/guide/misc/threadPrimitiveDeprecation.html
On linux, there is a tkill(int tid, int sig) command, similar to kill.
On windows, ProcessExplorer can do it from gui, don't know if there is anything with cli.
I want to exit a java process and free all the resources before it finishes its normal running, if a certain condition is meet. I dont however want to quit JVM, as I have other java programs running at the same time. Does return; do the above, or is there a better way to do it?
Thanks.
There is one JVM process per running Java application. If you exit that application, the process's JVM gets shut down. However, this does not affect other Java processes.
You need to understand the JVM mechanism and clarify the terminology.
Let's use the following as datum for the terminology.
Threads are divisions of concurrently processed flows within a process.
A process is an OS level thread. The OS manages the processes. A process is terminated by sending a termination signal to the OS management. The signal may be sent by the process itself or by another process that has the applicable privilege.
Within a process, you can create process level threads. Process level threads are normally facilitated by the process management of the OS, but they are initiated by the process and terminated by the process. Therefore, process level threads are not the same as processes.
An application is a collection of systems, programs and/or threads that cooperate in various forms. A program or process within an application may terminate without terminating the whole application.
Within the context of JVM terminology, program may be one of the following.
A program is run per JVM process. Each program consumes one JVM process and is invoked by supplying the classpath of java bytecode and specifying the main entry point found in the classpath. When you terminate a java program, the whole jvm process that ran that program also terminates.
A program is run per process level thread. For example, an application run within a tomcat or JEE server is run as a thread within the JEE process. The JEE process is itself a program consuming one JVM process. When you terminate an application program, the JEE process does not terminate.
You may initiate process level threads within a java program. You may write code that terminates a thread but that would not terminate the process (unless it is the last and only running thread in the process). The JVM garbage collection would take care of freeing of resources and you do not need to free resources yourself after a process level thread is terminated.
The above response is simplified for comprehension. Please read up on OS design and threading to facilitate a better understanding of processes and the JVM mechanism.
If the other threads running concurrently are not daemon threads, leaving main will not terminate the VM. The other threads will continue running.
I completely missed the point though.
If you start each program in a separate JVM, calling System.exit() in one of them will not influence the others, they're entirely different processes.
If you're starting them through a single script or something, depending on how it is written, something else could be killing the other processes. Without precise information about how you start these apps, there's really no telling what is going on.
#aix's answer is probably apropos to your question. Each time you run the java command (or the equivalent) you get a different JVM instance. Calling System.exit() in one JVM instance won't cause other JVM instances to exit. (Try it and see!)
It is possible to create a framework in which you do run multiple programs within the same JVM. Indeed this is effectively what you do when you run a "bean shell". The same sort of thing happens when your "programs" are services (or webapps, or whatever you call them) running in some application server framework.
The bad news is that if you do this kind of thing, there is no entirely reliable way make an individual "program" go away. In particular, if the program is not designed to be cooperative (e.g. if it doesn't check for interrupts), you will have to resort to the DEPRECATED Thread.stop() method and friends. And those methods can have nasty consequences for the JVM and the other programs running in it.
In theory, the solution to that problem is to use Isolates. Unfortunately, I don't think that any mainstream JVMs support Isolates.
Some common usecases leading these kind of requirements can be solved through tools like Nailgun, or Drip.
Nailgun allows you to run what appears to be multiple independent executions of a commandline program, but they all happen in the same JVM. Therefore repeated JVM start-up time does not have to be endured. If these execution interact with global state, then the JVM will get polluted in time and things start to break up.
Drip will use a new JVM for each execution, but it always keeps a precreated JVM with the correct classpath and options ready. This is less performant, but it can guarantee correctness through isolation.
i saw comment like this
one place i have seen this problem is if you keep creating threads, and instead of calling start(), call run() directly on the thread object.
This will result in the thread object not getting dereferenced...
So after sometime the message unable to create new native thread comes up
on the Sun Java Forums
In my application, initialy we plan to use thread, but later, we decided no need anymore, so we just call run() instead of start(). Do we need to do manual GC for new threadClass(..) ?
my tomcat startup setting
-Xms1024m -Xmx1024m -XX:MaxPermSize=450m
Why do you create a Thread in the first place?
Your code should implement the Runnable interface instead.
Then, when you decide that you want to run it in a thread, simple instantiate a Thread with the Runnable as the argument and call start() on the Thread object.
If, instead, you just want to run it in your current thread, simply call run() on your Runnable object.
This has several advantages:
you don't involve any Thread objects as long as you don't care about separate threads
your code is wrapped in a Runnable which fits closer conceptually: you're not writing some special kind of Thread, do you? You simply write some code that can be executed/run.
you can easily switch to using an Executor which further abstract away the decision
And last but not least you avoid any potential confusion on whether or not a native thread resource is created.
When you call run() method no new thread should be created. And your objects will be collected by Garbage collector when they are not referenced.
Your other part of code may be creating lot of Threads.
Try using ThreadPoolExecutor (thread pooling) in your code to limit threads in your application, And tune your threadpool size accordingly for better performance.
You can also check following to debug your issue: (referenced from link)
There are a few things to do if you encounter this exception.
Use the lsof -p PID command (Unix
platforms) to see how many threads
are active for this process.
Determine if there is a maximum
number of threads per process defined
by the operating system. If the limit
is too low for the application, try
raising the per-process thread limit.
Examine the application code to
determine if there is code that is
creating threads or connections (such
as LDAP connections) and not
destroying them. You could dump the
Java threads to see if there are an
excessive number has been created.
If you find that too many connections
are opened by the application, make
sure that any thread that the
application creates is destroyed. An
enterprise application (.ear) or Web
application (.war) runs under a
long-running JVM. Just because the
application is finished does not mean
that the JVM process ends. It is
imperative that an application free
any resources that it allocates.
Another solution would be for the
application to use a thread pool to
manage the threads needed.
This link describes quite nicely how this error is thrown by the JVM:
http://javaeesupportpatterns.blogspot.ro/2012/09/outofmemoryerror-unable-to-create-new.html
Basically it's very dependent on the OS. On RedHat Linux 6.5 (most likely other distros/version and kernel versions) the max_threads=max_process x 2.
The max number of threads is very dependent on the number of allowed processes. Which the max number of processes is dependent on the max physical memory you have installed.
If you have a look in the limits.conf file (on my RHL 6.5 it's in /etc/security/limits.d/90-nproc.conf). Exert form the file:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc **1024**
root soft nproc unlimited
You'll see that for non root users it's 1024 (which means 2048 max threads).
To see the max number of threads that your user is allowed to create run this command "cat /proc/sys/kernel/threads-max" or "sysctl kernel.threads-max".
To solve an issue like this (at least it worked for me) as root you'll need to ncrease the max allowed threads:
echo 10000 > /proc/sys/kernel/threads-max
This affects all users and the root. The user needs to log out and then log in again for the settings to take affect.