We Keep Coding

java process finishes after some time with no particular reason

I have a java .jar file that i launch on an AWS instance in detached mode. So when i exit the ssh session, it still runs.
The app does some network stuff, and is expected to run for days until it finishes it task.
I have made logs all over the app, made log in the end of main method. I also made a global try/catch and added logging to the catch section.
Still, after some days i enter into ssh and see that the app just stopped running. No exceptions, main method did not complete because the log in the end did not trigger. It seems that the process was just killed in the middle of working. Sometimes it works for 5 hours, sometimes for 3-4 days without stopping.
I have no idea what could be the cause of this. I expect the java process to run until it finished, or until it crashes. Am i missing something?
upd:
it is an aws t2.micro, i think, the free tier one. It runs ubuntu 18.04.3 LTS

You need to monitor the server and application. The first thing to look at is your instance cloudwatch statistics for any CPU or memory spikes. If you find one, you know what you need to fix if you want to run your application on micro instance. For further reading
Monitoring Your Instances Using CloudWatch
Alternatively, you can collect and dump the java process statistics regularly when you are running the application. This can give insight of how heap,stack and cpu usage. Check this SO post for further details :
How do I monitor the computer's CPU, memory, and disk usage in Java?

It it possible to take a snapshot for Java Profilers with Java?

I would like to periodically take a snapshot of my runtime (the .snapshot files used on profilers for Java, like YourKit, JProfiler, VisualVM, etc), is possible to take a snapshot by calling a method or something else? With Java, running on the same jvm?

I would suggest periodically running a Java Flight Recorder.
What is it? It is a profiling tool similar to the ones you already mentioned, it captures all the usual data.
How to use it? Here is a short description on how to start it up. Be careful that the application you want to profile needs these extra JVM params:
java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder
And finally for a periodic scheduling you need a command that you can use from crontab. For that you can use the example command from the previous link:
jcmd 5368 JFR.start duration=60s filename=myrecording.jfr
Where "5368" is the PID of your profiled application and the rest is self-explanatory.

Each profiler will have a different mechanism for periodically saving snapshots.
For JProfiler, use a "Timer" trigger with a "Save snapshot" action. If you want to record data for a limited time just before saving the snapshot, add the following sequence of trigger actions:
"Start recording" (with the desired recording types selected)
"Sleep" (for the desired amount of time)
"Stop recording"
"Save snapshot" (with "add unique number to file name" selected)
If you would rather like to control recording and saving snapshots programmatically from the same JVM, use the Controller API like this:
Controller.startCPURecording(true);
Thread.sleep(10000);
Controller.saveSnapshot(new File("snapshot.jps"));

Can I suspend Java runtime on method entry to have time to attach by debugger or profiler?

Is there a way to suspend all business-logic execution threads of Java process when execution hits some method?
If it is not possible via external configuration is there any method call to embedd into application to achieve desired result?

According to how to profile application startup with visualvm I can start application with
-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=y
to stop execution on app startup.
May be it would be possible to set breakpoint and move to interesting app state and then attach with profiler and start app again.
Another option is to start with Java agent: https://visualvm.github.io/startupprofiler.html

One solution is the place this code right before the method you want to debug:
System.out.println("Press Any Key To Continue...");
new java.util.Scanner(System.in).nextLine();
This waits until you press enter, so you have time to attach your debugger.
Another method is to use the IDE Netbeans, which contains a debugger and allows you to set breakpoints anywhere.

Multi-threaded time-based call hierarchy

I am using eclipse to write java code. If I'm debugging some code I can set a breakpoint and follow along as the code goes through each of the functions or I can backtrack. I can also look at the call hierarchy or the references to get an idea. But that's not enough.
I would like to have a some sort of time-based visualization of what each thread is doing along the process from ... let's say "point A" (pressing a button on the interface) to "point B" (getting the result). I want to see which classes/methods were called in what order. I want a good way to visualize what kind of output is coming from one method and going into another method which fires off a new process ...etc.
Is a profiler the only thing available for this type of visualization? Basically I want an action diagram or flow diagram created. Is there some plugin or app which can generate something like this?
Edit: Here is an example of what I'm thinking ... at least visually:
essmodel.sourceforge.net/index.html
It has some flow of where the code is leading. But I think this is just a static map of what classes lead to other classes and what inputs/output options are available. I would want to map the flow based on a specific case.

JProfiler offers such a view, it's called the "Call tracer":
It's important to restrict your filters very carefully in order not to record to much data.
Disclaimer: My company develops JProfiler.

I believe using a profiler is going to be your best option. Are you familiar with VisualVM? It comes with the JDK (look for "jvisualvm.exe" inside your JDK's bin directory) and is capable of profiling local virtual machines automatically as well as remote machines when configured properly. And it does give a pretty slick overview of what threads are running and the code they are spending time in, so I think you could easily do what you need from it. And best of all, it's free :)
As I said, local profiling is a breeze. You just run JVisualVM.exe standalone, and it will find any and all java processes running on the local machine automatically (you can just pick them out of a menu that VisualVM gives you upfront). If you want to profile remotely, set the following VM arguments for whatever it is that you're running:
-Dcom.sun.management.jmxremote.port=[0-65535]
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
Then within VisualVM, use the hostname of the machine your remote JVM is running on and the port you configured in the first VM argument above.

Restoring/Restarting a java daemon from crash

I am running a java app as daemon on a linux machine using a customized shell script.
Since I am new to java and linux, I want to know is it possible that the app itself resurrects itself(just like restart) and recovers from cases like app crashing, unhandled exceptions or out of memory etc.
thanks in advance
Ashish Sharma

The JVM is designed to die when there is an unrecoverable error. The ones you described fall in this category.
You could, however, easily write a shell script or a Python script that checks if the process is alive, and if it is dead, waits a few seconds and revive it. As a hint to doing this, the Unix command "pgrep" is your friend, as you can check for the exact command line used to fire a JVM (and thus including the starting class file). This way, you can determine if that specific JVM instance is running, and restart it.
All that being said, you may want to add some reporting or logging capability and check if often, because it is too easy to assume that things are ok when in fact the daemon is dying every few minutes. Make sure you've done what you could to prevent it from dying before resurrecting it.

There are Wrappers that can handle that, like Java Service Wrapper (Be aware, that the Community Edition ist under GPL) and some alternatives, mentioned here

To be honest, relaunching the daemon without any question after a crash is probably not a good idea; well it depends greatly on the type of processing achieved by your daemon, but if for example it processes files from a given directory, or requests coming from a queue manager, and the file / message contains some unexpected data causing the crash, relaunching the daemon would make it crash again immediately (excepting when the file / message is removed no matter it has been correctly processed or not, but as well it seems not to be a good idea).
In short, it's probably better to track down the possible crash reasons and fix them when possible (or at least log the the problem and go ahead, provided that the log message would ever be scanned to warn at last a human being, so some action can be engaged upon such "failures").
Anyway if you have very good reasons to do such, a solution even simpler than "checking that the process is alive" (as it would probably in some way involve some "ps -blahblah" stuff), you could just put the java program launching in a shell "while true" loop as follows :
while true
do
# launch the java program here, no background
# when crashing, the shell will be given hand back
java -classpath blahblah...
echo "program crashed, relaunching it..."
done

On unix based systems, you may use "inittab" to specify the program. If process dies, it is re-started by OS.(respawn)

I am not sure if the app itself can handle such crashes. You could write a shell script in linux which could be running as a cron job itself to manage the app, checking if the java app is running on scheduled intervals and if not, it will restart it automatically.