We Keep Coding

weblogic freeze and block all processing thread

We deploy our web application onto weblogic server 10.3.6. We try to run a stress test on our server. But sometime there a thread which do something we do not know yet, making the whole weblogic server freeze and does not process any request nor running thread, except only that thread.
We try and find out one of the reason is jxl library which call GC when read, create new file. After we disable it, our application run much more smoother. But it still freeze for some other reason.
So i want to ask is there a way to find out what our server doing at freezing time? Or is there any possible reason for the whole server to freeze like when they call GC?

Tomcat restarts with errors (exit 143), runs and then fails after time

This is my first time asking a question on Stack Overflow. I recently configured an Ubuntu 16.04 virtual private server to host a web application. I run ngnix on a Tomcat server that reads and writes to a MySQL database. The application runs fine except for the fact that Tomcat restarts itself once in a while which results in a 500 error that stems from a "broken-pipe" when anyone tries to login (i.e. make a connection to the database).
I will post an image of the 500 next time it happens. I went into my vps and looked at my Tomcat restart message. This is what I see: Tomcat status message.
I also did a little diving into the Tomcat logs and this is a log file that corresponds with that restart time: Tomcat log file
I did some research to try and solve this myself, but with no success. I believe that the exit=143 is the process being terminated by another program or the system itself. I also have done some moving of the mysql-connector-java.jar. I read that it should be located in the Tomcat/lib directory and not in the WEB-INF of the web application. Perhaps I need to configure other settings.
Any help or any direction would be much appreciated. I've fought this issue for a week with having learned much, but accomplished little.
Thanks

Look at the timeline. It starts at 19:49:23.766 in the Tomcat log with this message:
A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Exit code 143 is a result of that shutdown and doesn't indicate anything.
The question you need answered is: Who send that shutdown command, and why?
On a side note: The earlier messages indicates that Tomcat lost connection to the database, and that you didn't configure a validation query. You should always configure that, since database connections in the connection pool will go stale, and that needs to be detected.
Theory: Do you have some monitoring service running that tests your application being up? Does that monitoring detect a timed-out database connection, classify that as a hung webapp and auto-restart Tomcat?

While I don't think I am able to see to the core of the problem you have with your overall setup given the small excerpt of your log files, one thing strikes the eye. In the Tomcat log, there is the line
A valid shutdown command was received via the shutdown port. Stopping the server instance.
This explains why the server was restarted. Someone (some external process, a malicious attacker, script, or whatever. Could be anything depending on the setup of your server) sent a shutdown command to Tomcat's shutdown port (8005 by default) which made the Tomcat shut down.
Refer to OWASP's recommendations for securing a Tomcat server instance for fixing this possible security whole.
Regarding the ostensible Hibernate problems you have, I don't get enough information from your logs to make a useful statement. But you can leave the MySQL jar in Tomcat/lib, since this is not the root cause of your problem.

How to process servlet requests during long shutdown

We need to implement a graceful shutdown mechanism into our Servlet application.
EDIT: We want to make it simple as possible, which would be handling a kill signal sent via operating system's function. This would allow system admins to use built in shell utilities (kill or taskkill on Windows), otherwise they would have to install another utility just to "talk" with server.
This mechanism works in two phases:
upon shutdown request, deny certain critical activities
block until previously initiated critical actions are completed; these may take several hours
Phase #1 is implemented in our DAO layer.
Phase #2 is implemented in our ServletContextListener#contextDestroyed method
Our problem is that once contextDestroyed is called the Servlet container stops servicing further HTTP requests.
EDIT: contextDestroyed is called when someone is calling the operating system's kill function on server's process.
We would like to let the application alive during Phase #2, notifying the users that some activities are unavailable.

Use a filter to keep a list of all critical requests.
When the "prepare shutdown" request is received, the filter should start denying some requests.
Write a servlet that tells you how many critical jobs are still left in the queue.
In the shutdown tool, send the "prepare shutdown". The poll the servlet for the number of critical jobs. When this reaches 0, send the actual shutdown command.
To make this happen, create a service in the business layer which orchestrates this. Note that everything must happen before contextDestroyed() is being called! Your special application shutdown doesn't fit into the J2EE view of the world, so you have to manage it yourself.
The service should be able to tell interested parties when a shutdown is in progress, how many critical jobs are still running, etc. Servlets and filters can then use this service to deny requests or tell how many jobs are left.
When all jobs are done, deny all requests except access to the "shutdown info" servlet which should then tell that the app is now ready for death.
Write a tool which gives the administrators a nice UI to initiate shutdown of your app.
[EDIT] You may feel tempted to prevent the OS from shutting down your application. Don't do that.
What you should do is write a special tool to shut down your application using the two phase process that I described above. This should be the standard way to shutdown.
Yes, administrators will complain about it. On Unix, you can hide this tool by putting it into the init script, so no one will notice. There might be a similar solution on Windows.
Killing the server should always be possible to be able to stop it in case of (un)expected circumstances like: bugs in your shutdown code, emergency shutdown during power failure, bugs in your application code, or when Murphy happens.

JMX - Monitor process existence

I want to be able to monitor a process's existence continuously and restart it if it had crashed or killed for any reason using JMX. Stopping and starting a process is not a probelm as the agent executes a script for it. I can monitor process's existence by implementing heartbeats between the agent and the monitored process but I am looking for something using JMX itself, if something exists?

You can expose the component as a JMX managed resource.
Try to do the heartbeat.
If you get instanceNotFound from the managed bean server then it has crashed.

Starting and reconnecting to a spring bean from console

I like to write a little server application being controlled by a little console app (start, pause, stop, etc). For the server spring should be used (part of it already exist). So what I do to start the server is something like this:
start a server thread and exit main method
and then, in the thread:
load application context from xml
start some worker threads connecting to beans doing stuff
Now I want the server to be stopped by another command. How can I connect to the already running context and shut it down?
Thanks for your help,
Alexander

The classical way to manage running Java code is through JMX.
Spring lets you export beans as MBeans.
And for accessing JMX via the command line, look at the answers to this recent question:
Calling JMX MBean method from a shell script

You could create a pid file, when the server starts, it should log the pid to a file, server.pid. When you do a stop, you can read the process and kill it. Or even simpler, have a ever running thread in your main class which keeps looking for a file in some folder. As soon as the file becomes available or gets modified, it will stop the server.