I have a Java 1.6 application that accesses a third party native module, through a JNI class provided as the interface. Recently we noticed that a SEGFAULT is occurring in the native module, and is causing our application to crash. Is it possible to catch and handle this event, at least to log it properly before dieing?
I tried both Java techniques in the article from kjp's answer. Neither worked. Attempting to install a signal handler on 'SEGV' results in the exception
Signal already used by VM: SEGV
The shutdown handler I installed simply failed to fire, presumably because of what the IBM article states:
Shutdown hooks will not be run if
Runtime.halt() method is called to terminate the JVM. Runtime.halt() is provided to allow a quick shutdown of the JVM.
The -Xrs JVM option is specified.
The JVM exits abnormally, such as an exception condition or forced abort generated by the JVM software.
If all you want to do is log and notify you can write a script which runs your application. When the application dies, the script can detect whether the application terminated normally and from the hs_errXXXX file which has all the crash/SEGV information and mail it to someone (and restart the application if you want)
What you need to do is to run the faulty JNI code in another JVM and communicate with that JVM using RMI or JMS or Sockets. This way when the library dies, it won't bring down your main application and you can restart it.
Based on several weeks of research at the time, as well as conversations with JVM engineers at a conference this is not possible. The system will not let you install a SignalHandler on the SEGV signal.
Related
Problem scenario : The problem is noticed in sonic MF container (The jvm).The container has hosted some java services responsible for db operations and message transformations.Once started, the container runs fine for 2-3 weeks and terminates by its own without throwing any exceptions.
After much research, we are unable to find out why or what has triggered the jvm (MF Container) to shutdown.
Is there a way by which I can get the thread dumps when the jvm goes down automatically ? I'm using java 1.6. Is there any other approach to this problem I should follow ?
Thanks in advance.
You could try java.lang.Runtime.addShutdownHook(), and have the hook iterate over all threads and dump their stack traces with Thread.getAllStackTraces(). However, if the JVM was shutdown by Runtime.halt() then the hooks won't be called. More complicated would be to use instrumentation to hook into the calls to Runtime.exit() and Runtime.halt() (or Shutdown.sequence(), see edit #2), so you can see exactly what's happening at the time that either is called.
EDIT: Another way of doing it would be to install a SecurityManager which doesn't enforce any security, but which dumps the list of threads whenever SecurityManager.checkExit() is invoked, since both halt() and exit() call that security manager method. This would be a lot easier than using instrumentation, and you could even decide to throw an exception in addition to logging what the threads are doing.
EDIT 2: The system the JVM is running on can tell the JVM to terminate, in which case using a security manager won't work. Nor will using instrumentation on Runtime.exit() or Runtime.halt(), since the method that gets invoked is java.lang.Shutdown.exit(). And if the JVM is shutting down because the last daemon thread finished then Shutdown.shutdown() is invoked. But shutdown hooks will work in either of those situations. So you should always use the shutdown hooks, even if you're also going to use the security manager or instrumentation.
See also https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/hangloop.html "Troubleshooting Hanging or Looping Processes"
However, at least in my case, Eclipse is hung, and does not respond to any of these.
I'm developing an application that will run as a service. I have added a shutdown hook to the program to do some file cleanup. When I install the program as a service (on Linux as a daemon or as a Windows service) the shutdown hook executes correctly. However, when running in the IDE the shutdown hook is never executed. Is there a way to stop the running process in the IDE and have the shutdown hook execute?
Thanks,
Pablo
Unfortunately there is no way to do this within the current Netbeans system,since the kill methods for both types of app testing (Debug and Normal) both, well kill the process, giving it no chance to cleanup.
Depending on the architecture of your app you may be able to add a System.exit(0) call, but other than that you are stuck with opening it up in the console.
I disagree. A sigkill in unix or terminate process in windows by definition will kill the running process without giving any ability for the application to catch these events. This is necessary to terminate an unresponsive process. These signals should only be used when a process does not respond to a siginit (ctrl c) or a sigterm or end task in windows. NetBeans seems to be sending a sigkill rather than siginit or sigterm. As far as I'm concerned this is bad practice.
There should be a secondary option to kill the process however the primary end process button should be to send siginit or sigterm. Applications should anticipate the user hitting ctrl c or end task on their application and as a best effort cleanly close files/sockets and save persistent/state data. An application should not anticipate a sigkill or terminate process but rather be developed in such a way that a sigkill or terminate process are not necessary.
By NetBeans using these methods as the only means to terminate the application from the IDE is erroneous.
Nope. Don't cleanup your program using shutdown hooks. From the docs...
In rare circumstances the virtual machine may abort, that is, stop
running without shutting down cleanly. This occurs when the virtual
machine is terminated externally, for example with the SIGKILL signal
on Unix or the TerminateProcess call on Microsoft Windows. The virtual
machine may also abort if a native method goes awry by, for example,
corrupting internal data structures or attempting to access
nonexistent memory. If the virtual machine aborts then no guarantee
can be made about whether or not any shutdown hooks will be run.
When the contract says that there is no guarantee that your shutdown hooks will always be called, it is not advisable to use them for critical operations.
I have a Java program which depends on a native library that crashes randomly (and therefore bring down the whole JVM).
This library is about I/O, therefore can be easily isolated from the rest of the program with a second JVM.
I want to split the program in two so that the main program keeps running, but the native library can be restarted after crash. The question is: what is the best way to let these 2 JVM communicate? One of them will crash and be restarted but this should not affect the other JVM (besides some timeout with the I/O because of the crash and restart).
Is RMI crash resistant? Should I use a socket? A memory-mapped file?
I would use a persistent JMS Queue between the processes. This will be the most crash resistent. (Fixing the library is the best option)
I am trying to debug a program running on a remote solaris machine.
I wanted to know path taken by the program during the execution just like stake trace
like
Class A.method1 called method2 in class B
Class B.method2 called method3 in class C
..
...
Control returned from method2 in class B to
method1 in Class A etc
We cannot run the program from eclipse since the environment cannot be reproduced since it is a large enterprise level system having many upstream and downstream systems.
I suspect somewhere in the program there is an exception being thrown and that is not handled properly like a empty catch block/not logging or rethrowing the exception.
What is the best way to debug such programs. Please help me with your solutions.
You can use Eclipse to debug remote applications, all you need to do is add a few startup parameters: http://eclipse.dzone.com/articles/how-debug-remote-java-applicat
You can debug remotely with eclipse, see this
If you want to trace the execution of your Java code you can use a tool called InTrace.
InTrace supports outputting the trace from a Java program to disk or over the network to the InTrace UI. This should be ideal for debugging your Java program on a remote solaris machine.
NOTE: InTrace is a free and open source tool which I have written.
Since, your remote machine runs under Solaris you could use DTrace to dynamically add probes to your running JVMs and the complete system.
You could remote debug, which Eclipse supports, but may not be permitted by the system admin. The only other thing I can think of is lots and lots of logging
in such cases I have used Btrace https://dzone.com/articles/introduction-btrace , it doesn't require you to setup a remote debugger, nor to have the source code of your Java application.
Another way is to run jdb https://docs.oracle.com/javase/7/docs/technotes/tools/windows/jdb.html and attach to the process - but you must enable the process JVM to use jdwp -agentlib:jdwp=transport=dt_shmem,server=y,suspend=n
In general I agree that the JDK is lacking a professional built-in tracing/debugging tool
If a thread sets a shutdown hook using
Runtime.getRuntime().addShutdownHook();
calls via jna the method:
kernel32.CreateToolhelp32Snapshot (0x00000002, 0)
it crashes the VM.
If I call the same method in the
WindowListener.windowClosing()
hook, the call does not crashes the VM.
Any idea why?
I can post part of the VM crash error report if it could be of any use.
edit: see the VM crash report on pastebin
Posting the VM crash report should help.
Post part of the crash report on pastebin or the like maybe some can get some info out of that.
Well I don't know for sure but if you read the java doc for addShutdownHook()
Shutdown hooks run at a delicate time
in the life cycle of a virtual machine
and should therefore be coded defensively.
They should, in
particular, be written to be
thread-safe and to avoid deadlocks
insofar as possible. They should also
not rely blindly upon services that
may have registered their own shutdown
hooks and therefore may themselves in
the process of shutting down.
....
Shutdown hooks should also finish
their work quickly.
...
these leads me to the conclusion that maybe calling such a method either needs services from the JVM that aren't available anymore or that this call takes too long.