We Keep Coding

What happens for in-progress job when do restart java

When the java app in the middle of the service process restarts, waits for the process and after finishing that continued or does abort the process.

Depends on which process we are talking about, in-flight http requests are killed, but this behaviour can be configured, see https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#web.graceful-shutdown.

Gracefully shutdown JBoss application

I am looking for an approach to gracefully handle shutdown of JBoss (Wildfly AS 8.2). This would mean that all current requests are served and the webapp stops receiving further requests. I found that this is possible via command line in version 9 of the application server -
./jboss-cli.sh --controller=remoting://<host>:<port>
--connect --command=:shutdown(timeout=t)
Using this JBoss gracefully handles all requests for t seconds and gracefully shuts down (this would require an upgrade from version 8 to 9).
One possible approach would be to handle this in the Java application by maintaining a count of active requests and waiting for this number to go to 0 till a timeout and then exitting, basically replicating the above mentioned functionality.
I need to shutdown the webapp/JBoss remotely, so we are looking for a JMX (Java Management Extension) based solution. Does JBoss expose any such operation to gracefully shut down possibly via JMX or any other technology?
PS- Ctrl-C or kill commands donot shutdown JBoss gracefully.

JBoss EAP 6/7 and above allows graceful shutdown via CLI and even:
The signals SIGHUP, SIGINT and SIGTERM all trigger a graceful shutdown of the JBoss 7/6 application server. Those are signals are respectively the commands kill -1 $PID, kill -2 $PID(or control+c), and kill $PID.
https://access.redhat.com/solutions/18601

You can use the Jboss cli remotely as the URL you pass in the controller param shows. You could even use the "REST" API to execute it.
The shutdown should be available from JMX.

wait with systemd until a service socket becomes available and then start a depended service

Currently I have slow starting java service in systemd which takes about 60 seconds until it opens its HTTP port and serves other clients.
Another client service expects this service to be available (is a client of the this service), otherwise it dies after a certain retry. It also started with systemd. This is to be clear also a service. But uses the former like database.
Can I configure systemd to wait until the first service has made his socket available? (something like if the socket is actually listens , then the second client service should start).

Initialization Process Requires Forking
systemd waits for a daemon to initialize itself if the daemon forks. In your situation, that's pretty much the only way you have to do this.
The daemon offering the HTTP service must do all of its initialization in the main thread, once that initialization is done and the socket is listening for connections, it will fork(). The main process then exits. At that point systemd knows that your process was successfully initialized (exit 0) or not (exit 1).
Such a service receives the Type=... value of forking as follow:
[Service]
Type=forking
...
Note: If you are writing new code, consider not using fork. systemd already creates a new process for you so you do not have to fork. That was an old System V boot requirement for services.
"Requires" will make sure the process waits
The other services have to wait so they have to require the first to be started. Say your first service is called A, you would have a Requires like this:
[Unit]
...
Requires=A
...
Program with Patience in Mind
Of course, there is always another way which is for the other services to know to be patient. That means try to connect to the HTTP port, if it fails, sleep for a bit (in your case, 1 or 2 seconds would be just fine) then try again, until it works.
I have developed both methods and they both work very well.
Note: A powerful aspect to this method, if service A gets restarted, you'd get a new socket. This server can then auto-reconnect to the new socket when it detects that the old one goes down. This means you don't have to restart the other services when restarting service A. I like this method, but it's a bit more work to make sure it's all properly implemented.
Use the systemd Auto-Restart Feature?
Another way, maybe, would be to use the restart on failure. So if the child attempts to connect to that HTTP service and fails, it should fail, right? systemd can automatically restart your process over and over again until it succeeds. It's sucky, but if you have no control over the code of those daemons, it's probably the easiest way.
[Service]
...
Restart=on-failure
RestartSec=10
#SuccessExitStatus=3 7 # if success is not always just 0
...
This example waits 10 seconds after a failure before attempting to restart.
Hack (last resort, not recommended)
You could attempt a hack, although I do not ever recommend such things because something could happen that breaks such... in the services, change the files so that they have a sleep 60 then start the main process. For that, just write a script like so:
#!/bin/sh
sleep 60
"$#"
Then in the .service files, call that script as in:
ExecStart=/path/to/script /path/to/service args to service
This will run the script instead of directly your code. The script will first sleep for 60 seconds and then try to run your service. So if for some reason this time the HTTP service takes 90 seconds... it will still fail.
Still, this can be useful to know since that script could do all sorts of things, such as use the nc tool to probe the port before actually starting the service process. You could even write your own probing tool.
#!/bin/sh
while true
do
sleep 1
if probe
then
break
fi
done
"$#"
However, notice that such a loop is blocking until probe returns with exit code 0.

You have several options here.
Use a socket unit
The most elegant solution is to let systemd manage the socket for you. If you control the source code of the Java service, change it to use System.inheritedChannel() instead of allocating its own socket, and then use systemd units like this:
# example.socket
[Socket]
ListenStream=%t/example
[Install]
WantedBy=sockets.target
# example.service
[Service]
ExecStart=/usr/bin/java ...
StandardInput=socket
StandardOutput=socket
StandardError=journal
systemd will create the socket immediately (%t is the runtime directory, so in a system unit, the socket will be /run/example), and start the service as soon as the first connection attempt is made. (If you want the service to be started unconditionally, add an Install section to it as well, with WantedBy=multi-user.target.) When your client program connects to the socket, it will be queued by the kernel and block until the server is ready to accept connections on the socket. One additional benefit from this is that you can restart the service without any downtime on the socket – connection attempts will be queued until the restarted service is ready to accept connections again.
Make the service signal readiness to systemd
Alternatively, you can set up the service so that it signals to systemd when it is ready, and order the client after it. (Note that this requires After=example.service, not just Requires=example.service! Dependencies and ordering are orthogonal – without After=, both will be started in parallel.) There are two main service types that might make this possible:
Type=forking: systemd will consider the service to be ready as soon as the main program exits. Since you can’t fork in Java, I think you would have to write a small shell script which starts the server in the background and then waits until the socket is available (while ! test -S /run/example; do sleep 1s; done). Once the script exits, the service is considered ready.
Type=notify: systemd will wait for a message from the service before it is considered ready. Ideally, the message should be sent from the service PID itself: check if you can call the sd_notify function from libsystemd via JNI/JNA/whatever (specifically, sd_notify(0, "READY=1")). If that’s not possible, you can use the systemd-notify command-line tool (--ready option), but then you need to set NotifyAccess=all in the service unit (by default, only the main process may send notifications), and even then it likely will not work (systemd needs to process the message before systemd-notify exits, otherwise it will not be able to verify which cgroup the message came from).

spring rabbitmq java application correct shutdown

I have a standalone java application based on spring and spring-rabbit libraries. I start it like this:
nohup java -jar myapp.jar &
But sometimes I have to restart the application for upgrading it. Now I use killall -9 java, but it's not the best way. How to stop it correctly and make sure that all requests which comes to rabbit listeners of this app at that period of time will not be partially processed and will just be rejected and go to other rabbit consumer?

First of all - do not use killall -9 - it sends a SIGKILL signal to the JVM which cannot be intercepted and will not allow for an orderly shutdown. Instead, use killall or killall -15 (15 is the default signal) that sends SIGTERM which is intercepted by the JVM and allows for an orderly shutdown.
Second of all - do not ACK the RabbitMQ message prematurely - only do so once the message has actually been processed. Until you ACK the message, RabbitMQ will keep it in the "unacked" state. If the consumer dies without ACKing, the message will be put back on the queue for another consumer to pick it up.
Depending on the framework you are using, you may need to register a shutdown hook to close your application in a clean way. For example, if you are using standalone Spring, you should call ConfigurableApplicationContext#registerShutdownHook on the ApplicationContext that you create to make sure all the beans (including RabbitMQ consumers) are closed correctly.

First of all, don't use autoack if you afraid of data loss. Set autoAck = false, then in your consumer, when handling a message, ack it or nack when you really finished message processing. So, you will not loss data if you somehow shutdown your java process (even if you shutdown your machine). RabbitMq will store it until your client will ack it.
read here https://www.rabbitmq.com/tutorials/tutorial-two-java.html
in 'message acknoledgement' part.
In order to properly run your java process, write your bash script, that will stop/start/restart your programm.

JMX - Monitor process existence

I want to be able to monitor a process's existence continuously and restart it if it had crashed or killed for any reason using JMX. Stopping and starting a process is not a probelm as the agent executes a script for it. I can monitor process's existence by implementing heartbeats between the agent and the monitored process but I am looking for something using JMX itself, if something exists?

You can expose the component as a JMX managed resource.
Try to do the heartbeat.
If you get instanceNotFound from the managed bean server then it has crashed.