Problem scenario : The problem is noticed in sonic MF container (The jvm).The container has hosted some java services responsible for db operations and message transformations.Once started, the container runs fine for 2-3 weeks and terminates by its own without throwing any exceptions.
After much research, we are unable to find out why or what has triggered the jvm (MF Container) to shutdown.
Is there a way by which I can get the thread dumps when the jvm goes down automatically ? I'm using java 1.6. Is there any other approach to this problem I should follow ?
Thanks in advance.
You could try java.lang.Runtime.addShutdownHook(), and have the hook iterate over all threads and dump their stack traces with Thread.getAllStackTraces(). However, if the JVM was shutdown by Runtime.halt() then the hooks won't be called. More complicated would be to use instrumentation to hook into the calls to Runtime.exit() and Runtime.halt() (or Shutdown.sequence(), see edit #2), so you can see exactly what's happening at the time that either is called.
EDIT: Another way of doing it would be to install a SecurityManager which doesn't enforce any security, but which dumps the list of threads whenever SecurityManager.checkExit() is invoked, since both halt() and exit() call that security manager method. This would be a lot easier than using instrumentation, and you could even decide to throw an exception in addition to logging what the threads are doing.
EDIT 2: The system the JVM is running on can tell the JVM to terminate, in which case using a security manager won't work. Nor will using instrumentation on Runtime.exit() or Runtime.halt(), since the method that gets invoked is java.lang.Shutdown.exit(). And if the JVM is shutting down because the last daemon thread finished then Shutdown.shutdown() is invoked. But shutdown hooks will work in either of those situations. So you should always use the shutdown hooks, even if you're also going to use the security manager or instrumentation.
See also https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/hangloop.html "Troubleshooting Hanging or Looping Processes"
However, at least in my case, Eclipse is hung, and does not respond to any of these.
Related
TL;DR:
Is there a way to find out, that JVM-shutdown is only prevented from the threads started by my code? Is it for example possible to automatically trigger AutoCloseable.close() on Shutdown?
Context
I am building a library, that should be used by several customers. This means, besides providing a documentation, I can't enforce certain things.
Architecture
(I try to describe it as abstract as possible and avoid unnecessary details)
I have a "Manager" object (which is kind of a Factory), that is used to create a "Service" object, that in turn needs some data to work accordingly. Since that data is loaded from some "slow" backend service (which also might change from time to time), I use a separate (Daemon)-Thread that checks for updates and injects new data into that service as soon as available. (This also means that unless the first update, that service is simply in "noop mode". But that's ok.)
Now the "Updater" (which runs in my daemon thread) uses a library that again starts a thread when opening a connection and it's necessary to call "close" to ensure that this secondary thread is stopped - otherwise it is not possible, to shutdown the JVM properly.
As a safety-net I call the close() method inside the finalize() method of my "Manager" (which keeps a reference to all Updater instances). This is not 100% safe, since it's not predictable when GC runs (even more during shutdown!), but it's my only option.
Update: Here is some abstract example code that illustrates the architecture and the according problem
Problem
This architecture causes two possible pitfalls:
If the implementation does not keep a reference to the instance of the manager, it will be garbage collected at some point and trough the finalize method the necessary background updates will be stopped.
If the implementation keeps an instances of the manager, it must call the close method during the shutdown of the according system, otherwise the JVM can't terminate properly.
So my actual problem is the "potential unreliability" of the developers, which are using that library.
Does anyone have an idea how to build a solution, that could handle both pitfalls?
It would be nice to have some Auto-AutoCloseable ;) that is called during Shutdown (e.g by the DestroyJavaVM Thread or similar).
Solutions I tried unsuccessfully
Inside the Updater I am closing the "problematic" connection inside a "try-finally" block, but that daemon thread is not interrupted / stopped automatically as well.
I registered a Runtime.getRuntime().addShutdownHook(...) that would close all connections, but this shutdown hook is never called since a Shutdown is only initiated when all user-threads are stopped.
Update: Solved at my implementation, but not the problem
I solved my problem as I found that the third party library (RabbitMQ Client) offers a setThreadFactory method that I can use to ensure the spawned Threads are Daemon-Threads.
Good luck for me with my 3rd party library, but the described problem is still possible.
You want the AutoCloseable resources to be closed so the shutdown is orderly, I guess.
AutoCloseable objects should be used (by your library clients) in a manner that ensures they are closed when they are no longer needed. In almost all cases, they should be using a try-with-resources block, so they are closed even if an exception is thrown.
You should take advantage of that by requiring your library clients perform a controlled shutdown of each thread when they receive a request to shutdown the program. A thread performs a controlled shutdown by returning from each Runnable.run method, or throwing an exception from each Runnable.run method. I believe this is the only reliable means of closing resources, because it ensures nested resource allocations are reallocated in the correct order. More generally, as a library writer you can not know what other operations your library clients might want to do on shutdown, so you should give them complete control over the shutdown.
You can help them do that by having your library code properly handle InterruptedException and the Thread.interrupted flag.
Ok, I guess there is no good solution for that problem, but since I found that my 3rd party library gives the possibility to create Daemon-Threads, this is exactly what was necessary to fix it.
Maybe this is also a problem that should be solved on the "human level" by providing a good documentation and ensure a proper usage of AutoCloseable. A good developer should know how to deal with that.
From a technical point of view I found these possible solutions, which are just "safety nets" no one should rely on.
call "close()" inside the finalize method
I implemented a service, that can be used to register AutoClosable resources. It runs inside a Daemon-Thread and checks every few seconds, if it finds the JavaDestroyVM Thread and in such a case, closes all registered AutoClosables and stops itself.
Disclosure: If a system runs "outside of the main method" (the JavaDestroyVM Thread will run all the time), this is solution won't work.
Update: The RegisterAutoClosable-Service was a very ugly/hacky solution - I deleted it and plan to change the design, to avoid such situations, but finally it's the responsibility of the implementing developer to close opened resources properly.
We have a Java App that connects via RMI to another Java app.
There are multiple instances of this app running at the same time, and after a few days an instance just stops processing... the CPU is in 0 and I have an extra thread listening to an specific port that helps to shutdown the App.
I can connect to the specific port but the app doesn't do anything.
We're using Log4j to keep a log and nothing is written, so there aren't any exceptions thrown.
We also use c3p0 for the DB connections.
Anyone have ideas?
Thanks,
I would suggest starting with a thread dump of the affected application.
You need to see what is going on on a thread by thread basis. It could be that you have a long running thread, or other process which is blocking other work from being done.
Since you are running linux, you can get your thread dump with the following command
kill -3 <pid>
If you need help reading the output, please post it in your original question.
If nothing is shown from the thread dump, other alternatives can be looked at.
Hum... I would suggest using JMetter to stress the Application and take note of anything weird that might be happening (such as Memory Leaks, Deadlocks and such). Also review the code for any Exceptions that might interrupt the program (or System.exit() calls). Finally, if other people have access to the computer, makes sense to check if the process wasn't killed manually somehow.
I want to exit a java process and free all the resources before it finishes its normal running, if a certain condition is meet. I dont however want to quit JVM, as I have other java programs running at the same time. Does return; do the above, or is there a better way to do it?
Thanks.
There is one JVM process per running Java application. If you exit that application, the process's JVM gets shut down. However, this does not affect other Java processes.
You need to understand the JVM mechanism and clarify the terminology.
Let's use the following as datum for the terminology.
Threads are divisions of concurrently processed flows within a process.
A process is an OS level thread. The OS manages the processes. A process is terminated by sending a termination signal to the OS management. The signal may be sent by the process itself or by another process that has the applicable privilege.
Within a process, you can create process level threads. Process level threads are normally facilitated by the process management of the OS, but they are initiated by the process and terminated by the process. Therefore, process level threads are not the same as processes.
An application is a collection of systems, programs and/or threads that cooperate in various forms. A program or process within an application may terminate without terminating the whole application.
Within the context of JVM terminology, program may be one of the following.
A program is run per JVM process. Each program consumes one JVM process and is invoked by supplying the classpath of java bytecode and specifying the main entry point found in the classpath. When you terminate a java program, the whole jvm process that ran that program also terminates.
A program is run per process level thread. For example, an application run within a tomcat or JEE server is run as a thread within the JEE process. The JEE process is itself a program consuming one JVM process. When you terminate an application program, the JEE process does not terminate.
You may initiate process level threads within a java program. You may write code that terminates a thread but that would not terminate the process (unless it is the last and only running thread in the process). The JVM garbage collection would take care of freeing of resources and you do not need to free resources yourself after a process level thread is terminated.
The above response is simplified for comprehension. Please read up on OS design and threading to facilitate a better understanding of processes and the JVM mechanism.
If the other threads running concurrently are not daemon threads, leaving main will not terminate the VM. The other threads will continue running.
I completely missed the point though.
If you start each program in a separate JVM, calling System.exit() in one of them will not influence the others, they're entirely different processes.
If you're starting them through a single script or something, depending on how it is written, something else could be killing the other processes. Without precise information about how you start these apps, there's really no telling what is going on.
#aix's answer is probably apropos to your question. Each time you run the java command (or the equivalent) you get a different JVM instance. Calling System.exit() in one JVM instance won't cause other JVM instances to exit. (Try it and see!)
It is possible to create a framework in which you do run multiple programs within the same JVM. Indeed this is effectively what you do when you run a "bean shell". The same sort of thing happens when your "programs" are services (or webapps, or whatever you call them) running in some application server framework.
The bad news is that if you do this kind of thing, there is no entirely reliable way make an individual "program" go away. In particular, if the program is not designed to be cooperative (e.g. if it doesn't check for interrupts), you will have to resort to the DEPRECATED Thread.stop() method and friends. And those methods can have nasty consequences for the JVM and the other programs running in it.
In theory, the solution to that problem is to use Isolates. Unfortunately, I don't think that any mainstream JVMs support Isolates.
Some common usecases leading these kind of requirements can be solved through tools like Nailgun, or Drip.
Nailgun allows you to run what appears to be multiple independent executions of a commandline program, but they all happen in the same JVM. Therefore repeated JVM start-up time does not have to be endured. If these execution interact with global state, then the JVM will get polluted in time and things start to break up.
Drip will use a new JVM for each execution, but it always keeps a precreated JVM with the correct classpath and options ready. This is less performant, but it can guarantee correctness through isolation.
I have J2SE application running in 1.5 java VM in RHEL OS. One of the task of the application is to create 3 infinitely running user threads, during startup. The purpose is to check for a request of a particular type in backend DB table and do corresponding operations.
As we observed, the long running threads suddenly stops running, but still the application is alive and JVM process can be seen, in ps -ef|grep java
Can someone throw light on why threads which are created to run in infinite loop, stops suddenly? Any ideas on how to detect this issue and possible resolution will be of great help
With Regards,
Krishna
I would suggest sending a Ctrl+Break to your app, dumping the threads and analysing the output. Perhaps your threads are waiting for some input (IO). Perhaps they're deadlocked. Perhaps they've exited with an uncaught exception. The thread dump will tell you what's going on (and it helps if you name your threads in advance so you can identify them in the dump).
Perhaps you are having unhandled exceptions.
First of all, you should log all your thread activity (you can use log4j to achieve this).
You can also override the uncaughtException method of the ThreadGroup class and create alerts for when a thread dies due to an exception that has not been caught.
If a thread sets a shutdown hook using
Runtime.getRuntime().addShutdownHook();
calls via jna the method:
kernel32.CreateToolhelp32Snapshot (0x00000002, 0)
it crashes the VM.
If I call the same method in the
WindowListener.windowClosing()
hook, the call does not crashes the VM.
Any idea why?
I can post part of the VM crash error report if it could be of any use.
edit: see the VM crash report on pastebin
Posting the VM crash report should help.
Post part of the crash report on pastebin or the like maybe some can get some info out of that.
Well I don't know for sure but if you read the java doc for addShutdownHook()
Shutdown hooks run at a delicate time
in the life cycle of a virtual machine
and should therefore be coded defensively.
They should, in
particular, be written to be
thread-safe and to avoid deadlocks
insofar as possible. They should also
not rely blindly upon services that
may have registered their own shutdown
hooks and therefore may themselves in
the process of shutting down.
....
Shutdown hooks should also finish
their work quickly.
...
these leads me to the conclusion that maybe calling such a method either needs services from the JVM that aren't available anymore or that this call takes too long.