Related to this question: Remove lock on environment at every transaction end and How does Jetbrains YouTrack server scales over load?
What can be done to implement a way for Xodus Lock Mechanism to work even with non-graceful application shutdown? For instance, if the application process (which opened write access to a Xodus environment) is killed, the lock remains and the new application process cannot write to the database anymore due to the .lck file, so a manual find . -name "xd.lck" -type f -delete needs to be executed to make it work again.
Additionally, this is true also with multi-process servlet containers/servers which spawns multiple processes of the same application. So the question, how can Xodus Locking mechanism play well with these scenarios?
The lock is immediately released after "the application process (which opened write access to a Xodus environment) is killed".
Related
Before redeploying the application war, I checked the xd.lck file from one of the environment path:
Private property of Exodus: 20578#localhost
jetbrains.exodus.io.LockingManager.lock(LockingManager.kt:89)
I'm testing from both Nginx Unit and Payara server to eliminate the possibility that this is an isolated case with Unit.
And process 20578 shows from htop:
20578 root 20 0 2868M 748M 7152 S 0.7 75.8 14:05.75 /usr/lib/jvm/zulu-8-amd64/bin/java -cp /
After redeployment finished successfully, accessing the web application throws:
java.lang.Thread.run(Thread.java:748)
at jetbrains.exodus.log.Log.tryLock(Log.kt:799)
at jetbrains.exodus.log.Log.<init>(Log.kt:120)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:142)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:121)
at jetbrains.exodus.env.Environments.newLogInstance(Environments.java:10
And checking the same xd.lck file shows the same content. Meaning to say that "lock is not immediately released" contrary to what is described here.
My assumption is for this specific case with Payara Server (based on Glassfish) is that, the server does not kill the previous process even after redeployment has completed. Maybe perhaps for "zero-downtime" redeployment, not sure, Payara experts can correct me here.
Checking with htop the process 20578 is still running even after the redeployment.
As with Xodus, since most application servers behave this way, what would be the best solution and/or workaround so we don't need to manually delete all lock files of each environment (if can be deleted) every time we redeploy?
Solution is for the Java application to look for the process locking the file then do a kill -15 signal for example to gracefully make the Java handle the signal to be able to close environments:
// Get all PersistentEntityStore's
entityStoreMap.forEach((dir, entityStore) -> {
entityStore.getEnvironment().close();
entityStore.close();
}
I have number of java process running on my machine. I need to track how many times each process is getting restarted.
For Example:
Let us consider two java process
Process 1 - Its restarted for 5 times.
Process 2 - Its restarted for 2 times.
I'm able to get the PID, java command of the running processes. But I could not able to differentiate once the process got restarted. Because the PID changed after the restart. Also I can't consider the java command because two instance of same application which has same command.
So what are the other ways to track the java process restart ?
You want your processes to keep the same identity after a restart. Ideally, you would have a parameter, system property or environment variable telling the process its identity.
As you state in the question, this identity cannot be passed on the command line. Thus, the process has to "find" its identity by acquiring an exclusive resource.
This resource could be a shared system implementing locks but it is probably to complex.
As exclusive resources, we have network sockets. So you could make your processes artificially opening a socket in the sole objective to make it acquire an identity.
You can use the code from https://stackoverflow.com/a/116113/2242270 in order to open a socket in a range. The identity of the process is then the port that could be opened.
I have java web application with pool of selenium ChromeDrivers. This pool is exists because of quicker response from REST api (without creating WebDrivers every time).
The problem is, that the chromedrivers are still alive after Tomcat restart.
Is it possible to set some expiration time of each process? Because I cant kill processes manualy every time when I restart the Tomcat.
Or maybe start the chromedrivers with some daemon configuration same as java threads?
Thanks for any answer :)
You can kill chromedriver.exe processes from your java app as described here:
Killing a process using Java
In our projects that's what we do kill the chrome driver processes completely to prevent it from affecting the new runs.
I have a java app that has multiple instances over a local network. It uses Redis Redlock to manage integrity of a shared database. Issue here is this java app is still highly unstable so that it crash lot of times. When one instance crashed and it held the lock at the time of crash all other instance get stuck. My question is can I release a lock from a Redis CLI when an instance of Java app which hold the lock crashed.
With the CLI I could remove lock from Redis server with command
DEL <lock name>
When doing so the waiting thread could acquire the lock. I don't know this is the right way. But it works.
We need to implement a graceful shutdown mechanism into our Servlet application.
EDIT: We want to make it simple as possible, which would be handling a kill signal sent via operating system's function. This would allow system admins to use built in shell utilities (kill or taskkill on Windows), otherwise they would have to install another utility just to "talk" with server.
This mechanism works in two phases:
upon shutdown request, deny certain critical activities
block until previously initiated critical actions are completed; these may take several hours
Phase #1 is implemented in our DAO layer.
Phase #2 is implemented in our ServletContextListener#contextDestroyed method
Our problem is that once contextDestroyed is called the Servlet container stops servicing further HTTP requests.
EDIT: contextDestroyed is called when someone is calling the operating system's kill function on server's process.
We would like to let the application alive during Phase #2, notifying the users that some activities are unavailable.
Use a filter to keep a list of all critical requests.
When the "prepare shutdown" request is received, the filter should start denying some requests.
Write a servlet that tells you how many critical jobs are still left in the queue.
In the shutdown tool, send the "prepare shutdown". The poll the servlet for the number of critical jobs. When this reaches 0, send the actual shutdown command.
To make this happen, create a service in the business layer which orchestrates this. Note that everything must happen before contextDestroyed() is being called! Your special application shutdown doesn't fit into the J2EE view of the world, so you have to manage it yourself.
The service should be able to tell interested parties when a shutdown is in progress, how many critical jobs are still running, etc. Servlets and filters can then use this service to deny requests or tell how many jobs are left.
When all jobs are done, deny all requests except access to the "shutdown info" servlet which should then tell that the app is now ready for death.
Write a tool which gives the administrators a nice UI to initiate shutdown of your app.
[EDIT] You may feel tempted to prevent the OS from shutting down your application. Don't do that.
What you should do is write a special tool to shut down your application using the two phase process that I described above. This should be the standard way to shutdown.
Yes, administrators will complain about it. On Unix, you can hide this tool by putting it into the init script, so no one will notice. There might be a similar solution on Windows.
Killing the server should always be possible to be able to stop it in case of (un)expected circumstances like: bugs in your shutdown code, emergency shutdown during power failure, bugs in your application code, or when Murphy happens.