I have a java .jar file that i launch on an AWS instance in detached mode. So when i exit the ssh session, it still runs.
The app does some network stuff, and is expected to run for days until it finishes it task.
I have made logs all over the app, made log in the end of main method. I also made a global try/catch and added logging to the catch section.
Still, after some days i enter into ssh and see that the app just stopped running. No exceptions, main method did not complete because the log in the end did not trigger. It seems that the process was just killed in the middle of working. Sometimes it works for 5 hours, sometimes for 3-4 days without stopping.
I have no idea what could be the cause of this. I expect the java process to run until it finished, or until it crashes. Am i missing something?
upd:
it is an aws t2.micro, i think, the free tier one. It runs ubuntu 18.04.3 LTS
You need to monitor the server and application. The first thing to look at is your instance cloudwatch statistics for any CPU or memory spikes. If you find one, you know what you need to fix if you want to run your application on micro instance. For further reading
Monitoring Your Instances Using CloudWatch
Alternatively, you can collect and dump the java process statistics regularly when you are running the application. This can give insight of how heap,stack and cpu usage. Check this SO post for further details :
How do I monitor the computer's CPU, memory, and disk usage in Java?
Related
I have a browser game built on a java web server using jsp.
I added a new module that uses some http session object and keep data in it. However, after it runs 3-4 hours, it suddenly stops working and freezes. When I check the error log, I dont see any exception thrown.
The server has 50-60 online in a moment.
I monitored the server using visualVM and here is the result after 4 hours until it stops :
I set the max memory as 1024Mb. As you can see its problem is not about the memory.
The thing that I notice is when the server stops, the thread amount increased.
According to the screenshot, should I doubt the httpsession object ? Why does the server stop responding ??
It looks like a system limitation or a deadlock.
Your thread graph looks like problematic : the number of living thread is important and never decreases. A web application should be stateless. The living tread count should rises when the requests arrive but also drops when the requests are finished.
I have not the impression it is the case in your application.
MGorgon is right.
You should also check "Deadlock detection" in jconsole.
If you use a JDK 6+ version, you could use ThreadMXBean. It has a findDeadlockedThreads() methods and other interesting methods to address your need.
Anyway, if it is not a deadlock, to get more information about the cause of the problem, I advise you to look in the system log whatever you OS is. You would have maybe interesting things.
TL;DR: Is there a foolproof (!) way I can detect from my master JVM that my slave JVM spawned via 2 intermediate scripts has experienced an OutOfMemory error on Linux?
Long version:
I'm running some sort of application launcher. Basically it receives some input and reacts by spawning a slave Java application to process said input. This happens via a python script (to correctly handle remote kill commands for it) which in turn calls a bash script (generated by Gradle and sets up the classpath) to actually spawn the slave.
The slave contains a worker thread and a monitor thread to make callbacks to a remote host for status updates. If status updates fail to occur for a set amount of time, the slave gets killed by the launcher. The reason for it not responding CAN be an OutOfMemoryError, however it can also be other reasons. I need to differentiate an OutOfMemoryError of the slave from some other error which caused it to stop working.
I don't just want to monitor memory usage and say once it reaches like 90% "ok that's enough". It may very well be that the GC succeeds in cleaning up sufficiently for the workload to finish. I only want to know if it failed to clean up and the JVM died because not enough memory could be freed.
What I have tried:
Use the -XX:OnOutOfMemory flag as a JVM option for the slave which calls a script which in turn creates an empty flag file. I then check with the launcher for the existence of the flag file if the slave died. Worked like a charm on Windows, did not work at all on Unix because there is a funky bug which causes the execution of the flag call to require the exact same amount of Xmx the slave has used. See https://bugs.openjdk.java.net/browse/JDK-8027434 for the bug. => Solution discarded because the slave needs the entire memory of the machine.
try{ longWork(); } catch (OutOfMemoryError e) { createOomFlagFile(); System.exit(100); } This does work in some cases. However there are also cases where this does not happen and the monitor thread simply stops sending status updates. No exception occurs, no OOM flag file gets created. I know from SSHing onto the machine though that Java is eating all the memory available on the system and the whole system is slow.
Is there some (elegant) foolproof way to detect this which I am missing?
You shouldn't wait for the OutOfMemory. My suggestion is, that you track memory consumption from the master application via Java Management Beans and issue warnings when memory consumption gets critical. I never did that on my own before, so I cannot get more precisely on how to do that, but maybe you find out or some others here can provide a solution.
Edit: this is the respective MXBean http://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryMXBean.html
I have a java project that runs for a long time since it has a lot of things to churn through (controlling other subprocesses and all).
I would like to display the progress of this executable on a webpage that I can access for the period of execution of the Java executable.
How do I spin up a Node.js server from my java excutable so that if the java executable exits, the server knows to also exit and save the report information so far somewhere. Also, am I doing something that others have done before?
You could set up a heartbeat (a signal from your java application), and have the node app look for that (as part of the data it processes from the java application, or as a separate message). When there is no data, you still need to send the heartbeat. But as long as it is a small message, it will hopefully not impact your performance significantly.
Then set the node server up to shutdown (or restart), and write out your exiting data, if the heartbeat is absent for a fixed amount of time.
I have an application that will run fine the first time it is ran. Now if I leave the application and force stop it in manage applications and come back in it will crash with OutOfMemory. Why would it work the first time and run out of memory on subsequent calls?
There is really no way to know with the information you provide. However that is totally possible, it is likely a process that runs when the app loads is leaking memory massively.
We have a Java App that connects via RMI to another Java app.
There are multiple instances of this app running at the same time, and after a few days an instance just stops processing... the CPU is in 0 and I have an extra thread listening to an specific port that helps to shutdown the App.
I can connect to the specific port but the app doesn't do anything.
We're using Log4j to keep a log and nothing is written, so there aren't any exceptions thrown.
We also use c3p0 for the DB connections.
Anyone have ideas?
Thanks,
I would suggest starting with a thread dump of the affected application.
You need to see what is going on on a thread by thread basis. It could be that you have a long running thread, or other process which is blocking other work from being done.
Since you are running linux, you can get your thread dump with the following command
kill -3 <pid>
If you need help reading the output, please post it in your original question.
If nothing is shown from the thread dump, other alternatives can be looked at.
Hum... I would suggest using JMetter to stress the Application and take note of anything weird that might be happening (such as Memory Leaks, Deadlocks and such). Also review the code for any Exceptions that might interrupt the program (or System.exit() calls). Finally, if other people have access to the computer, makes sense to check if the process wasn't killed manually somehow.