OOM using CRON but not using SHELL - java

When I start a java program with high memory usage ("-Xmx52g") by shell, everything is working well. However if I start the same program with the same command and same user by CRON, I get a java.lang.OutOfMemoryError just after a few seconds.
Additionally CRON isn't able to do anything as long as I don't kill the blocked java program. No matter which cronjob should be started, it always ends up with "(CRON) error (can't fork)" in syslog. After killing the java program all new cronjobs are working fine again.
The problem only occurs with Ubuntu 16.04, all older versions worked very well. Is this a bug or a new security feature? I didn't find any information about this issue, so I hope anyone may help.

You need more RAM or size to launch it
http://javarevisited.blogspot.com/2011/09/javalangoutofmemoryerror-permgen-space.html

Related

Let java program run indefinitely - restart when it crashes

I have a java program that should run on a Windows machine. It should run "forever", i.e. when the JVM or the program crashes, it should be restarted. When the computer is restarted it should also be restarted.
I saw advice to wrap the program as a "Windows service", but the tools I found seem to be either costly, complicated or outdated.
Can somebody describe me a straightforward way to achieve the desired behaviour?
For the part where you want to start the program after restart you can create a simple batch (.Bat) file and u can put that file in the startup folder.
Also you can use the same file for running the program when it crashes. you can use tasklist command and check if your java program is running and if it is not .just start the program.
Just check our windows batch this is one of the best things you can get everything for doing anything on windows without anything expensive
Yet Another Java Service Wrapper is a tool that easily wraps your Java program into a Windows service. Just start the program, note down the PID and enter it into the wrapper. Two things, which are probably universal to services, should be noted:
For connection to the network, you need to specify an account with the necessary rights.
Connected network drives are not available.

How do I add continuous entropy on a linux server

Environment:
Headless Linux server
Red Hat Enterprise Linux Server release 6.7 (Santiago)
2.6.32-573.12.1.e16.x86_64
I have a java(7) program that I run from command line that spawns multiple threads and hits a oracle database simultaneously using ojdbc7.
Intermittently I see connection reset error:
Could not get JDBC Connection; nested exception is java.sql.SQLRecoverableException: IO Error: Connection reset
exactly as reported here and resolved here.
I tried the following to mitigate the issue:
Add these variations to my command line
-Djava.security.egd=file:///dev/urandom
-Djava.security.egd=file:/dev/../dev/urandom
-Djava.security.egd=file:/dev/./urandom
Tried adding to java.security file
securerandom.source=file:///dev/urandom
securerandom.source=file:/dev/../dev/urandom
securerandom.source=file:/dev/./urandom
Also tried ojdbc6
But I still see the issue. Which means that even with urandom in use, still there is not enough realtime entropy. Which is understandable, because when I run my java program, everything else is stopped (this is a application server with multiple running jvms, usually. )
I am wondering if there is anything I can do to 'cause' entropy on the server. I have limited access to this server so I am limited in what I can do. This java job is expected to run for a couple of hours, so I can not keep typing on the keyboard etc. May be run a simple program in the background that 'does' something?
Any ideas? I tried rngd command but apparently I do not have permission to use it. Any help greatly appreciated, been stuck with this issue for a while now.
Edit:
I tried to run another java program (using java.awt.Robot) that simulates Keyboard presses continuously during my original program run, but due to limited permissions on the server I wasn't able to get it working. But in general, would that be a good way to ensure some entropy?

Solved: JVM hangs forever with no CPU usage or I/O

I'm experiencing this weird problem that the JVM hangs forever very frequently.
I first observed the problem when my Java IDEs frequently hang the entire system GUI. IntelliJ IDEA hangs on indexing almost every single time upon start. Sometimes it proceeds to resolving dependency but always hangs in the end. When this happens, I can type in gnome-terminal, but the commands can't seem to be executed. I can't launch new applications with Alt-F2 or anything alike.
I had to switch to a text console and "killall -9 java" to kill the IDEA process and get control back. "kill -3 java" won't work. The log file contains nothing related, the thread dump is empty. Once the IDE hung, jstack cannot be attached to the process. "jstack -l pid" also hangs. "jstack -F pid" can't attach to the process. Visualvm hangs as well.
The CPU usage by the Java process is 0% and there is no I/O going on.
I've observed the same behavior when using Eclipse. Sometimes it hangs on start up, sometimes upon saving and sometimes upon running a Java application.
Maven / sbt builds executed within text-only ttys cause the same kind of hang, so I guess it's not a window manager / desktop environment / display driver problem.
I highly suspect it's a file system or I/O issue but I have no clue how to debug that. I've tried fsck with no luck, and my system works perfectly fine when not running java programs.
Things I've ruled out:
Permission issues: running IntelliJ with sudo doesn't help, hangs 100% of the time.
Display driver: I've tried both the Nvidia proprietary driver and nouveau, the open source one. Doesn't help.
Window manager / desktop environment: I use Cinnamon, but I've tried running IntelliJ under Unity. Doesn't help.
Java version: I've tried both Oracle Java 7 and Oracle Java 8. I'll probably try OpenJDK but I doubt it would work.
IntelliJ version: I've tried IntelliJ 13 through 14.1. All exhibited the same behavior.
Limited memory: I have 16G RAM with 16G swap space, so memory should not be a limiting factor.
Kernel log doesn't look suspicious. I can't get any kind of log remotely indicating what went wrong.
Any idea?
UPDATE (2015/04/29): The problem seems to have fixed itself after I accidentally kicked the power cable and cold restarted the computer... Still a mystery but IntelliJ is usable as of now.
Some things to check
- The Java IDEs run best with a lot of ram. I usually ask for at least
8G of memory for my dev workstation.
- Make sure you have a stable version of everything, look for known working versions/configurations on Ubuntu
- You have to manually allocate memory in IntelliJ IDEA versions < 14. For example: How to increase IDE memory limit in IntelliJ IDEA on Mac?
- Besides system logs, run tools like top and see what's happening in terms for cpu and ram when running the IDE
I had similar problems a while ago but with Eclipse. The problem was that there was no swap place at all ;) - obviously it should not be a problem with 16GB of RAM.
Could You post JVM arguments for Intellij? And also I have an idea to create another Intellij installation (eg. go back to 14 version) and see if there is similar problem (also compare JVMs settings between these two).
Edit
Ok so try:
use different JRE/JDK. If the problem disappears it will tell us more.
You are on linux so it makes it easy to monitor several things - you said that there is no CPU utilization or hard I/O. But how do you know that? Maybe it will be informative to have some statistics gathered - eg.jstat for JVM itself or for system information (You think that is I/O problem) try:
iostat -hm -p sda 1
Which will print I/O statistics for sda (if you have different discs change device parameter) in 1sec loops (can be also changed). Start this with system and dump output to the file - maybe there is some kind of 'disaster' happening before JVM hangs. Note: iostat sometimes is not available on system itself (on my Linux Mint is not) - install then package sysstat and the command will be available.
Seem to have fixed after a cold restart from an accidental power loss.. Weirdest problem I have ever seen.

MATLAB hangs when I try to use the java package jdde, but only for the first time after a system reboot

I'm using the external java package jdde in MATLAB. Please note that for the following example, the DLL file that comes with the package needs to be on the MATLAB librarypath. The method to do this is different depending on your MATLAB Version.
Using jdde in MATLAB works fine, except for the first time after I reboot the computer or I logoff/logon in Windows. When I run the following code for the first time after a computer reboot, MATLAB will stay in busy mode forever (with 0% CPU). When this happens, I kill the MATLAB process in the task manager and restart MATLAB. When I run the same code again, it will execute instantly (not staying busy forever).
javaaddpath('C:\pretty-tools-JDDE-1.0.2.jar')
a = com.pretty_tools.dde.client.DDEClientConversation;
a.connect('','');
To sum it up, the above code will cause MATLAB to stay busy forever the first time I run it after a system reboot or user logoff/logon. When I run it again after killing the MATLAB process, it will work perfectly fine (not hanging up MATLAB).
I have seen this behavior on different computers, and in different Versions of MATLAB (2010 and 2012). I'm using Windows 7 x64.
In the code example, the a.connect command is the one that causes MATLAB to stay busy forever. Putting this command in a try/catch block would not help, because the a.connect doesn't cause an error, it just never does continue.
I'm not sure if this problem is caused by MATLAB or by the java package.
Any ideas how to get rid of this behavior would be much appreciated.
Note: The input argument of a.connect does not matter, it will always hang, so I just gave '' as input in this example.
Code hangs without any know reason in DdeInitialize() method. New build JDDE-2.0.3 contains workaround for this problem.
Try running the add path command on its own so that there is a second or two before it tries to execute code dependant on the jar. I have found this to often be the problem with intermittent issues having to do with jars in Matlab
Switch over to classic mode initially so that u will get rid of that.

Tomcat stopped without any log or any stack

We have trouble with Tomcat 5.5 which stops at night on our production servers (Linux CentOS 4.8) and we have no idea why it stops...
There is no Tomcat's log in catalina.out or any application's log.
We tried different things to find why the server stops:
configure Tomcat to be able to generate a core dump
instrument System.exit() method with javassist to find if the method was called
add a shutdown hook to the JVM (with Runtime.getRuntime().addShutdownHook())
None of them worked, we have no core dump, the Exit method and the shutdown hook are not called.
My conclusions are:
The VM is not terminated properly but crash without any log.
Any idea or log to read to find why Tomcat stops?
1) Make sure you know where stderr is redirected and check if anything got printed there.
2) Check the memory limits on Tomcat and how much free memory does the system have. Review the Linux system logs under /var/log to see if anything suspicious happened during the time. For example, kernel can randomly kill a process (almost) without a trace if the system is running low on memory.
We've ran 5.5 in production for years and never had any unexplained shutdowns, FWIW.
This worked for me.
As suggested here in other answers checked system logs in /var/log/messages but permission denied for me. So, I used dmesg command instead and got this in the logs
"Out of memory: Kill process 14606 (java) score 106 or sacrifice child".
In the output I also noticed Swap Memory free 0 K. Ran top command to confirm the same. So, somehow there was a high memory usage which caused the OS to kill my tomcat process.
After spending hours finally got the reason.
ps -ef | grep tomcat showed that there were several tomcat processes running for the same application. It seems that, earlier tomcat shutdowns might not have been completed successfully and due to some reason the processes were not killed even after the shutdown, which was causing the high memory usage.
So, killed all running tomcat processes using kill. SWAP memory got freed.
Started tomcat again, worked fine. :)
Tomcat 7 has an option inside catalina to prevent the System.exit class call or something similar: http://ci.apache.org/projects/tomcat/tomcat7/docs/security-manager-howto.html .
Maybe there's a similar option for the 5.5 version. Try the documentation.
There are options to redirect the output to the same console that you use to start Tomcat. This information is redirected to logs when you execute on Unix based systems, on Windows, it remains with the console if not redirected.
Most probably there is a stack-overflow exception. This is typical behavior of Tomcat when it happens. For example, you're trying to serialize to JSON or XML beans with cyclic dependencies (but without handling of the cycles).
Everytime I had this issue (several times) it always has been this one. All other stops are usually logged properly (like OutOfMemory etc).
This type of stops leaves no trace anywhere.

Categories