Environment:
Headless Linux server
Red Hat Enterprise Linux Server release 6.7 (Santiago)
2.6.32-573.12.1.e16.x86_64
I have a java(7) program that I run from command line that spawns multiple threads and hits a oracle database simultaneously using ojdbc7.
Intermittently I see connection reset error:
Could not get JDBC Connection; nested exception is java.sql.SQLRecoverableException: IO Error: Connection reset
exactly as reported here and resolved here.
I tried the following to mitigate the issue:
Add these variations to my command line
-Djava.security.egd=file:///dev/urandom
-Djava.security.egd=file:/dev/../dev/urandom
-Djava.security.egd=file:/dev/./urandom
Tried adding to java.security file
securerandom.source=file:///dev/urandom
securerandom.source=file:/dev/../dev/urandom
securerandom.source=file:/dev/./urandom
Also tried ojdbc6
But I still see the issue. Which means that even with urandom in use, still there is not enough realtime entropy. Which is understandable, because when I run my java program, everything else is stopped (this is a application server with multiple running jvms, usually. )
I am wondering if there is anything I can do to 'cause' entropy on the server. I have limited access to this server so I am limited in what I can do. This java job is expected to run for a couple of hours, so I can not keep typing on the keyboard etc. May be run a simple program in the background that 'does' something?
Any ideas? I tried rngd command but apparently I do not have permission to use it. Any help greatly appreciated, been stuck with this issue for a while now.
Edit:
I tried to run another java program (using java.awt.Robot) that simulates Keyboard presses continuously during my original program run, but due to limited permissions on the server I wasn't able to get it working. But in general, would that be a good way to ensure some entropy?
Related
I have a java program that should run on a Windows machine. It should run "forever", i.e. when the JVM or the program crashes, it should be restarted. When the computer is restarted it should also be restarted.
I saw advice to wrap the program as a "Windows service", but the tools I found seem to be either costly, complicated or outdated.
Can somebody describe me a straightforward way to achieve the desired behaviour?
For the part where you want to start the program after restart you can create a simple batch (.Bat) file and u can put that file in the startup folder.
Also you can use the same file for running the program when it crashes. you can use tasklist command and check if your java program is running and if it is not .just start the program.
Just check our windows batch this is one of the best things you can get everything for doing anything on windows without anything expensive
Yet Another Java Service Wrapper is a tool that easily wraps your Java program into a Windows service. Just start the program, note down the PID and enter it into the wrapper. Two things, which are probably universal to services, should be noted:
For connection to the network, you need to specify an account with the necessary rights.
Connected network drives are not available.
When I start a java program with high memory usage ("-Xmx52g") by shell, everything is working well. However if I start the same program with the same command and same user by CRON, I get a java.lang.OutOfMemoryError just after a few seconds.
Additionally CRON isn't able to do anything as long as I don't kill the blocked java program. No matter which cronjob should be started, it always ends up with "(CRON) error (can't fork)" in syslog. After killing the java program all new cronjobs are working fine again.
The problem only occurs with Ubuntu 16.04, all older versions worked very well. Is this a bug or a new security feature? I didn't find any information about this issue, so I hope anyone may help.
You need more RAM or size to launch it
http://javarevisited.blogspot.com/2011/09/javalangoutofmemoryerror-permgen-space.html
I'm experiencing this weird problem that the JVM hangs forever very frequently.
I first observed the problem when my Java IDEs frequently hang the entire system GUI. IntelliJ IDEA hangs on indexing almost every single time upon start. Sometimes it proceeds to resolving dependency but always hangs in the end. When this happens, I can type in gnome-terminal, but the commands can't seem to be executed. I can't launch new applications with Alt-F2 or anything alike.
I had to switch to a text console and "killall -9 java" to kill the IDEA process and get control back. "kill -3 java" won't work. The log file contains nothing related, the thread dump is empty. Once the IDE hung, jstack cannot be attached to the process. "jstack -l pid" also hangs. "jstack -F pid" can't attach to the process. Visualvm hangs as well.
The CPU usage by the Java process is 0% and there is no I/O going on.
I've observed the same behavior when using Eclipse. Sometimes it hangs on start up, sometimes upon saving and sometimes upon running a Java application.
Maven / sbt builds executed within text-only ttys cause the same kind of hang, so I guess it's not a window manager / desktop environment / display driver problem.
I highly suspect it's a file system or I/O issue but I have no clue how to debug that. I've tried fsck with no luck, and my system works perfectly fine when not running java programs.
Things I've ruled out:
Permission issues: running IntelliJ with sudo doesn't help, hangs 100% of the time.
Display driver: I've tried both the Nvidia proprietary driver and nouveau, the open source one. Doesn't help.
Window manager / desktop environment: I use Cinnamon, but I've tried running IntelliJ under Unity. Doesn't help.
Java version: I've tried both Oracle Java 7 and Oracle Java 8. I'll probably try OpenJDK but I doubt it would work.
IntelliJ version: I've tried IntelliJ 13 through 14.1. All exhibited the same behavior.
Limited memory: I have 16G RAM with 16G swap space, so memory should not be a limiting factor.
Kernel log doesn't look suspicious. I can't get any kind of log remotely indicating what went wrong.
Any idea?
UPDATE (2015/04/29): The problem seems to have fixed itself after I accidentally kicked the power cable and cold restarted the computer... Still a mystery but IntelliJ is usable as of now.
Some things to check
- The Java IDEs run best with a lot of ram. I usually ask for at least
8G of memory for my dev workstation.
- Make sure you have a stable version of everything, look for known working versions/configurations on Ubuntu
- You have to manually allocate memory in IntelliJ IDEA versions < 14. For example: How to increase IDE memory limit in IntelliJ IDEA on Mac?
- Besides system logs, run tools like top and see what's happening in terms for cpu and ram when running the IDE
I had similar problems a while ago but with Eclipse. The problem was that there was no swap place at all ;) - obviously it should not be a problem with 16GB of RAM.
Could You post JVM arguments for Intellij? And also I have an idea to create another Intellij installation (eg. go back to 14 version) and see if there is similar problem (also compare JVMs settings between these two).
Edit
Ok so try:
use different JRE/JDK. If the problem disappears it will tell us more.
You are on linux so it makes it easy to monitor several things - you said that there is no CPU utilization or hard I/O. But how do you know that? Maybe it will be informative to have some statistics gathered - eg.jstat for JVM itself or for system information (You think that is I/O problem) try:
iostat -hm -p sda 1
Which will print I/O statistics for sda (if you have different discs change device parameter) in 1sec loops (can be also changed). Start this with system and dump output to the file - maybe there is some kind of 'disaster' happening before JVM hangs. Note: iostat sometimes is not available on system itself (on my Linux Mint is not) - install then package sysstat and the command will be available.
Seem to have fixed after a cold restart from an accidental power loss.. Weirdest problem I have ever seen.
I'm using the external java package jdde in MATLAB. Please note that for the following example, the DLL file that comes with the package needs to be on the MATLAB librarypath. The method to do this is different depending on your MATLAB Version.
Using jdde in MATLAB works fine, except for the first time after I reboot the computer or I logoff/logon in Windows. When I run the following code for the first time after a computer reboot, MATLAB will stay in busy mode forever (with 0% CPU). When this happens, I kill the MATLAB process in the task manager and restart MATLAB. When I run the same code again, it will execute instantly (not staying busy forever).
javaaddpath('C:\pretty-tools-JDDE-1.0.2.jar')
a = com.pretty_tools.dde.client.DDEClientConversation;
a.connect('','');
To sum it up, the above code will cause MATLAB to stay busy forever the first time I run it after a system reboot or user logoff/logon. When I run it again after killing the MATLAB process, it will work perfectly fine (not hanging up MATLAB).
I have seen this behavior on different computers, and in different Versions of MATLAB (2010 and 2012). I'm using Windows 7 x64.
In the code example, the a.connect command is the one that causes MATLAB to stay busy forever. Putting this command in a try/catch block would not help, because the a.connect doesn't cause an error, it just never does continue.
I'm not sure if this problem is caused by MATLAB or by the java package.
Any ideas how to get rid of this behavior would be much appreciated.
Note: The input argument of a.connect does not matter, it will always hang, so I just gave '' as input in this example.
Code hangs without any know reason in DdeInitialize() method. New build JDDE-2.0.3 contains workaround for this problem.
Try running the add path command on its own so that there is a second or two before it tries to execute code dependant on the jar. I have found this to often be the problem with intermittent issues having to do with jars in Matlab
Switch over to classic mode initially so that u will get rid of that.
We have a problem where our java process is hanging forever,
Unless a Kill -9 is issued against it.
The same Process is running successfully in the Other Solaris Envs,
Java process consist single thread which starts and end after doing some processing On the Data ,Though from the logs and data we can see that the code is completely executed and all the data is processed.
but if we do JPS we will always see that process is running.
we are Using EHcache with spring for caching purpose and UCP for the connection pool.
On The dB side we Have ORACLE RAC Structure.
took several Jstacks and can never See the Process sticking in the my code.
though from thread dump can see there are lot of UCP threads hanging there.
Also Adding a Shutdown hook and removing It in the end,but some reason seems the shutdownhook is never called.
Due to project restrictions ,cant paste the code.
can Anyone Please help
my customer is facing same problem with our installer hanging on Solaris. when installer ran in debug mode, we realized that java which is embedded with installer is hanging. Please post in case any of you found answer for it.