I am running a java app as daemon on a linux machine using a customized shell script.
Since I am new to java and linux, I want to know is it possible that the app itself resurrects itself(just like restart) and recovers from cases like app crashing, unhandled exceptions or out of memory etc.
thanks in advance
Ashish Sharma
The JVM is designed to die when there is an unrecoverable error. The ones you described fall in this category.
You could, however, easily write a shell script or a Python script that checks if the process is alive, and if it is dead, waits a few seconds and revive it. As a hint to doing this, the Unix command "pgrep" is your friend, as you can check for the exact command line used to fire a JVM (and thus including the starting class file). This way, you can determine if that specific JVM instance is running, and restart it.
All that being said, you may want to add some reporting or logging capability and check if often, because it is too easy to assume that things are ok when in fact the daemon is dying every few minutes. Make sure you've done what you could to prevent it from dying before resurrecting it.
There are Wrappers that can handle that, like Java Service Wrapper (Be aware, that the Community Edition ist under GPL) and some alternatives, mentioned here
To be honest, relaunching the daemon without any question after a crash is probably not a good idea; well it depends greatly on the type of processing achieved by your daemon, but if for example it processes files from a given directory, or requests coming from a queue manager, and the file / message contains some unexpected data causing the crash, relaunching the daemon would make it crash again immediately (excepting when the file / message is removed no matter it has been correctly processed or not, but as well it seems not to be a good idea).
In short, it's probably better to track down the possible crash reasons and fix them when possible (or at least log the the problem and go ahead, provided that the log message would ever be scanned to warn at last a human being, so some action can be engaged upon such "failures").
Anyway if you have very good reasons to do such, a solution even simpler than "checking that the process is alive" (as it would probably in some way involve some "ps -blahblah" stuff), you could just put the java program launching in a shell "while true" loop as follows :
while true
do
# launch the java program here, no background
# when crashing, the shell will be given hand back
java -classpath blahblah...
echo "program crashed, relaunching it..."
done
On unix based systems, you may use "inittab" to specify the program. If process dies, it is re-started by OS.(respawn)
I am not sure if the app itself can handle such crashes. You could write a shell script in linux which could be running as a cron job itself to manage the app, checking if the java app is running on scheduled intervals and if not, it will restart it automatically.
Related
I have a java program that should run on a Windows machine. It should run "forever", i.e. when the JVM or the program crashes, it should be restarted. When the computer is restarted it should also be restarted.
I saw advice to wrap the program as a "Windows service", but the tools I found seem to be either costly, complicated or outdated.
Can somebody describe me a straightforward way to achieve the desired behaviour?
For the part where you want to start the program after restart you can create a simple batch (.Bat) file and u can put that file in the startup folder.
Also you can use the same file for running the program when it crashes. you can use tasklist command and check if your java program is running and if it is not .just start the program.
Just check our windows batch this is one of the best things you can get everything for doing anything on windows without anything expensive
Yet Another Java Service Wrapper is a tool that easily wraps your Java program into a Windows service. Just start the program, note down the PID and enter it into the wrapper. Two things, which are probably universal to services, should be noted:
For connection to the network, you need to specify an account with the necessary rights.
Connected network drives are not available.
To be clear, this is a upstart/linux debugging problem not a java problem per-se.
I have a java application installing a shutdownhook. It shows some funny behaviour in ubuntu-GNOME, namely that the shutdown hook never run if a restart or power off were scheduled. At first i thought it was a problem with my shutdown hook, so i simplified it until it was only writing a line to file (yes i know about the log4j2 logger problem with shutdownhooks so i disabled theirs too). Then when that didn't work i started hacking /etc/init/sendsigs
I added this to the beginning of the do_stop function:
app="$(jps | grep appname.jar | cut -d' ' -f1)"
echo "$app" >> "/home/i30817/output.txt"
sure enough, that showed that it was no longer running, so the shutdown hooks were never activated by sendsigs
Then i used lastcomm from here replacing my edit of sendsigs by:
echo `/home/i30817/lastcomm` > "/home/i30817/output.txt"
And it told me that the java process exited with 1 and was not signaled:
java X i30817 ?? 10.26 secs Sun Mar 2 12:44 E 1
but this still didn't help me find what actually killed it and why. This problem is not reproducible with a smaller example, so it's probably something in the larger application (but not the shutdown hook, since it was minimized) that doesn't like the shutdown process and manages to kill the process, but i can't figure it out... redirecting the process output to a file doesn't say anything either eg:
java -jar /home/i30817/Documents/projects/app/dist/app.jar > allout.text 2>&1
is empty of everything but normal app output
Can you help me figure out this? There are a lot of duplicate questions about the same thing too (but they think it's the shutdownhooks malfunctioning).
edit: more detail, now that i understand the problem a bit better. I think now that processes not being there on sendsigs is normal. Java applications, and maybe others use a protocol from the window manager where SIGHUP, SIGHUP and SIGCONT is sent on shutdown/logout. The JVM hooks SIGHUP to launch the shutdown hooks. I tested this with a very small example that only adds a shutdownhook and has a infinite cycle on main, and ran it with a system tap script in the background:
java -jar app.jar
and in another shell
sudo stap -o process.txt sigkill.stp
However when i tried that with my application i think i found the culprit:
PROCESS: SIGSEGV java.signal_generate sent to java 2280
but don't really know what to do about it considering there is no thread dump or anything and this is strange to reproduce (only my app, only during shutdown).
edit2: now i suspect the reason for the 'abrupt' termination without core dump is the ulimit during shutdown. So i'm trying to solve that in preparation for a bug report. I edited /etc/security/limits.conf to add this and rebooted
* soft core unlimited
root hard core unlimited
* hard rss 10000
(fs.suid_dumpable=2 was set by ubuntu, no apparmor i think)
but during shutdown i edited /etc/init.d/sendsigs again to print ulimit -a and sleep for 30 seconds before killing the processes, and it seems that during reboot the ulimit gets reset again? And moreover, it had a different output like it was using another executable version, for instance instead of saying 'core file size' it had 'core(dump)' or something like that).
edit3: ah, i need to have fs.suid_dumpable=1 instead - gonna try now.
Maybe the init ulimit doesn't matter for shutdown core dump triggering. After all the jvm was executed from the user env so it should be using the user ulimit.
edit4: eh. After much commenting of code i reached the following conclusion that i could have reached from RTFM:
the sigsegvs are harmless.
the non-zero exit code is not.
If the AWT is still up, the signal is always non zero and the shutdown hooks never run. Even a small example still prevents execution of shutdown hooks in linux reboot if a JFrame is up (unlike windows, where they will start). Looking at the source, the application shutdownhooks are run on a slot by themselves, slot 1. I bet slot '0' is the AWT and that is halting the system somehow.
I guess it's time to check the package private signal handling libs to see if i can get SIGHUP before the JVM decides to terminate everything without even giving the opportunity for cleanup code to run.
According to docs
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook(java.lang.Thread)
In rare circumstances the virtual machine may abort, that is, stop running without shutting down cleanly....
If the virtual machine aborts then no guarantee can be made about whether or not any shutdown hooks will be run.
What I would like to accomplish is to start a Java program and have it keep running until the user kills it with a control-C. I realize that it is possible to do this by creating a BufferedReader and having it endlessly loop while reading the BufferedReader, but what I am doing involves me backgrounding the Java program (e.g., java -jar app.jar &) which kills standard in so that method would not work. I've read a bit on Java's daemon threads, but I also do not think that is the correct solution in this instance because I want the JVM to stay alive.
Any help would be much appreciated.
Thanks,
Chris
I'm guessing you're launching the java program from a different process perhaps? A potential option may be integrating the Java Service Wrapper ? Depending on what you are doing, licensing may be an issue, and you probably wouldn't kill the process with a ctrl-c, but it's a thought (in addition to the others given above).
guys, I encounter a confusing issue! I want to invoke a executable JAR to compute PI from my main java applicaion using runtime.exec(), which create a new JVM for running the executable JAR. My issue is that when the PI computation is done, the JVM is still alive. my main java application has no idea whether PI computation is finished or not. And I also want when the PI computation is done, the JVM could be shutdown! How can I implement that! ThankS!!
When you call Runtime.exec() you will get a Process object back. You need to call waitFor() on this.
You will also need to capture the stdout/stderr streams (in separate threads to prevent blocking - see this answer for more info).
This all leaves aside why you're doing this in a separate JVM, and why you can't load the relevant classes into your current app and run the library locally.
To answer the second part of your question, the JVM always exits when no non-daemon threads are still running. Or in plainer speech, when your application is "done" and the main method exits without leaving any threads running in the background, the JVM will finish.
This is true regardless of whether you launch the Java process yourself from your desktop/command line, or fire it off via Runtime.exec() (which is broadly equivalent). So when your Pi calculation terminates it will shut down the JVM you spawned, and when your original program finishes then its JVM will also exit.
Though I agree completely with Brian here, I can't see the benefit in running a Java app as a separate process when you should be able to just run it in the original JVM (barring some really unusual environmental stuff such as setting niceness or processor affinity of the various processes).
check what the jar does when you run it standalone, maybe its waiting for input and therefor it will never exit
I have an application that has a license for a set number of cpus and I want to be able to set the number of cpus that java runs in to 1 before the check is done. I am running Solaris and have looked at pbind but thought that if I started the application and then used pbind it would have checked the license before it had set the number of CPUs that java could use.
Does anyone know a way of starting an application with a set number of CPUs on Solaris?
It is a workaround, but using Solaris 10 you could set up a zone with a single CPU available and then run the application inside that zone.
If you want to do testing without running the full application, this bit of Java is most likely what they are using to get the number of CPU's:
Runtime runtime = Runtime.getRuntime();
int nrOfProcessors = runtime.availableProcessors();
A full example here.
This isn't a complete solution, but might be enough to develop into one. There's definitely a point at which the java process exists (and thus can be controlled by pbind) and at which point it hasn't yet run the code to do the processor check. If you could pause the launch of the application itself until pbind had done its work, this should be OK (assuming that the pbind idea will work from the CPU-checking point of view).
One way to do this that should definitely pause the JVM at an appropriate place is the socket attach for remote debuggers and starting with suspend mode. If you pass the following arguments to the java invocation:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,suspend=y,server=y
then the JVM will pause after starting the java process but before executing the main class, until a debugger/agent is attached to port 8000.
So perhaps it would be possible to use a wrapper script to start the program in the background with these parameters, sleep for a second or so, use pbind to set the number of processors to one for the java process, then attach and detach some agent to port 8000 (which will be enough to get Java to proceed with execution).
Flaws or potential hiccoughs in this idea would be whether running in debug mode would notably affect performance of your app (it doesn't seem to have a big impact in general), whether you can control some kind of no-op JDWP agent from the command line, and whether you're able to open ports on the machine. It's not something I've tried to automate before (though I've used something broadly similar in a manual way to increase the niceness of a Java process before letting it loose), so there might be other issues I've overlooked.
I think the most direct answer to your question is to use pbind to bind the running shell process, and then start Java from that shell. According to the man page the effects of pbind are inherited by processes that are created from a bound process. Try this:
% pbind -b 0 $$
% java ...
Googling over, I found that you are right, pbind binds processes to processors.
More info and examples at: http://docs.sun.com/app/docs/doc/816-5166/pbind-1m?a=view