I have a Java application running on Window Server 2008 R2 SP1 (Under Copyright 2009) with JVM 6u16 x64, my program will write log in every 5 seconds to show it is still alive, however, the program itself has stopped writing log for 10 to 20 seconds with no reason.
I have used a VisualVM to monitor my Java application, but it stopped as well, as the VisualVm keep flooding new threads (for monitoring) during the halt period
The halt happens only between 9am to 4pm (3 to 4 times a day) but not in the night time.
This Java application works fine on another Window Server 2008 R2 SP1 (Under Copyright 2007), so I am wondering whether the new Win Server 2008 (2009 ver) has some compatibility issue with the JVM, and whether there are any KB which has fixed that
Tested 2 more things, but still no luck
1. Wrote a C++ program to write a date time log in every second, but it didn't stop writing when the JVM halt, which indicates no issues related to the OS
2. Upgrade to JVM 6u45, but the java program still paused for 1 time and last for 10 seconds, which is better than 6u16, as it happened around 3 to 4 times in day time.
Thanks for help
It sounds like you may have an issue with occasional "full GC" pauses.
I suggest you turn on GC logging, and then see if there is a correlation between these unwanted outages and the events in the GC log messages; read about the -Xloggc:file option in the Java command manual entry.
Depending on what that tells you, the solution may be to adjust your JVM's GC options.
Related
One java standalone application taking more time for execution in Oracle Solaris 11.4 OS which takes 3 times less time in Red Hat Linux server. What can be the issue? The code is exactly the same.
Ubuntu 12.04 LTS
java -version
java version "1.6.0_38"
Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)
4 core CPU - some Dell server hardware
10 threads from time to time run a "heavy" job over several minutes. At other periods they are doing nothing.
1 thread is supposed to wake up every 5 (or so) secs and send a quick ping over the network to another process. This works nicely as long as the other 10 threads do nothing, but when the other 10 threads are running a "heavy" job it never (or very rarely) get to run and send its ping.
I could understand this if this "heavy" job was CPU intensive. But during such a "heavy" job top says something like 50-100% IO-wait but around 1% CPU usage. Profiling shows that by far most of the time spent by the 10 threads are spent in (waiting I guess) in some NIO call. It all adds up, and is kinda expected because a lot of the heaviness about the job is to read files from disk.
What I do not understand is that during such a "heavy" job, the 1 thread doing pings do not get to run. How can that be explained when top shows 1% CPU usage and it seems (profiling and top) that the 10 threads are spending most of their time waiting for IO. Isnt the 1 ping-thread supposed to get execution-time when the other threads are waiting for IO?
Java thread priority is equal on all 11 threads.
Spreading a few yields here and there in the 10 threads seem to solve (or lower) the problem, but I simply do not understand why the ping thread does not get to run without the yields, when the other threads do not do much but wait for IO.
ADDITIONAL INFO 05.03.2014
I have reproduced the problem in a simpler setup - even though not very simple yet (you will have to find out how to install a Apache Zookeeper server, but it is fairly simple - I can provide info later)
Find Eclipse Kepler project here (maven - build by "mvn package"): https://dl.dropboxusercontent.com/u/25718039/io-test.zip
Find binary here: https://dl.dropboxusercontent.com/u/25718039/io-test-1.0-SNAPSHOT-jar-with-dependencies.jar
Start a Apache ZooKeeper 3.4.5 (on port 2181) server on a machine. On another separate machine (this is where I have Ubuntu 12.04 LTS etc. as described above) run the binary as follows (first create a folder io-test-files - 50GB space needed)
nohup java -cp io-test-1.0-SNAPSHOT-jar-with-dependencies.jar dk.designware.io_test.ZKIOTest ./io-test-files 10 1024 5000000 IP-of-ZooKeeper-server:2181 > ./io-test-files/stdouterr.txt 2>&1 &
First it creates 10 5GB files (50GB is way more than machine RAM so not much help by OS file-cache), then starts a ZooKeeper client (which is supposed to keep its connection with the ZooKeeper server up by sending pings/heartbeats regularly), then makes 10 threads doing random-access into the 10 files creating a lot of disk IO, but really no real usage of the CPU. I see that the ZooKeeper client eventually loses its connection ("Zk state"-prints stop saying "CONNECTED" - in stdouterr.txt), and that is basically what I do not understand. The ZooKeeper client-thread only wants to send a tiny heartbeat with several secs apart, and only if it is not able to do that within a period of 20 secs is will lose its connection. I would expect it to have easy access to the CPU, because all the other threads are basically only waiting for disk IO.
During the test I see the following using "top"
Very high "Load average". Above 10 which I do not understand, because there are basically only 10 threads doing something. I also thought that "Load average" only counted threads that actually wanted to do real stuff on the CPU (not including waiting of IO), but according to http://en.wikipedia.org/wiki/Load_%28computing%29 Linux also counts "uninterruptible sleeps" including threads waiting of IO. But I really do not hope/think that it will prevent other threads that have real stuff to do, from getting their hands on the CPU
Very high %wa, but almost no %sy and %us on the CPU(s)
Here is the output from one of my runs: https://dl.dropboxusercontent.com/u/25718039/io-test-output.txt
I wrote a Java program, which analyses other programs. The execution may take very long (= days). Now (after three days), I have the problem, that my program / process is sleeping (S). It still has allocated 50% of the memory and sometimes it prints new output, but top shows must of the time 0% CPU.
I used jstack to be sure, that there are still runnable threads. Hence, it seems not to be a deadlock problem. I do not know, why the process does not get more cpu time. I chanced the niceness of the java process from 0 to -10, but nothing happends.
More details:
The process runs on a linux server: Ubuntu 10.04.4 LTS.
I started my process with screen. So, I do not have to be logged in all the time.
screen -S analyse ant myParameters
The server has almost nothing to do.
Thanks for your help.
Start your program in debug mode. Then you can attach to it with any Java debugger and inspect what it is doing.
I have a java application which uses a few MulticastSocket instances to listen
to a few UDP multicast feeds. Each such socket is handled by a dedicated thread.
The thread reads each Datagram, parses it's content and writes to log (log4j) the packet's sequence id (long) and the timestamp the Datagram was received.
When I try to run 2 instances of the same application on a Windows Server 2008 R2,
with 2 * 6 cores and compare the 2 logs created by the 2 applications,
I notice that quite frequently the timing of the packets isn't the same.
Most packets are received by the 2 apps at the same time (milis), but frequently
there's a difference of about 1-7ms diff between the reception time of the same packet
by the 2 apps.
I tried allocating more buffers in the NIC, and also made the socket read buffer bigger.
In addition I tried minimizing GC runs and I also use -verbose:gc and can see
that GC times and the problematic timing diff does not occur at the same time.
This allows me to assume that my problem isn't GC related.
No drop packets problem was observed, and a bandwidth problem is not likely.
Ideas / Opinions are welcome.
Thanks.
By default Windows timer interrupt frequency is 100 Hz (1 tick per 10ms). It means that OS cannot guarantee that Java threads will get woken up at higher precision.
Here's an excerpt from a prominent James Holmes article about timing in Java - it could be your case:
for Windows users, particularly on dual-core or multi-processor systems (and it seems most commonly on x64 AMD systems) if you see erratic timing behaviour either in Java, or other applications (games, multi-media presentations) on your system, then try adding the /usepmtimer switch in your boot.ini file.
PS: by no means I'm credible in the field of Windows performance optimization, also starting from Windows 2008 HPET is supported, but how it is related to timer interrupt frequency is a mystery to me.
7ms is a very good result for a 6 core machine, and drift in java will be much higher than that if garbage collector kicks in.
Dont forget that java runtime has it's own overhead as well.
I'm running a java application that is supposed to process a batch of files through some decisioning module. I need to process the batch for about 50 hrs. The problem I'm facing is that the process runs fine for about an hour and then starts to idle. So, I did this - I run the JVM for one hour and then shut it down, restart the JVM after 30 mins, but still for some reason the second run is taking almost 4-5 hrs. to do what the first run does in 1 hr. Any help or insights would be greatly appreciated.
I am running this on a 64-bit windows r2 server, 2 intel quad core processors(2.53 GHz), 24 GB RAM. Java version is 1.6.0_22(64-bit), memory allotted to the application is - heap(16 GB) and PermGen(2GB).
the external module is also running on a jvm and i am shutting that down too, but i have a feeling that it is holding on to memory even after shutdown. before i start the jvm RAM usage is 1 GB, after I end it it tends to stay at about 3 GB. Is there any way i can ask JAVA to forcibly release that memory?
Are you sure the JVM you are trying to close is indeed closed?
Once a process ends all of the RAM it had allocated is no longer allocated. There's no way for a process to hang on to it once it closes, which also means there's no way for you to tell it to do so. This is done by the Operating System.
Frankly, this sounds like the JVM is still running, or something else is eating the RAM. Also, it sounds like you're trying to workaround a vicious bug instead of hunting it down and killing it?
I suspect the JVM isn't exiting at all. I see this symptom regularly with a JBoss instance that grinds to a halt with OutOfMemoryExceptions. When I kill the JVM (via a SIGTERM/SIGQUIT), the JVM doesn't exit in a timely fashion since it's consuming all its resources throwing/handling OOM exceptions.
Check your logs and process table carefully to ensure the JVM quits properly. At that point the OS will clear all resources related to that process (including the consumed memory)
i've noticed something in the process.
After I shut down the JVM, if i delete all the files that i wrote to the file system, then the RAM usage comes back to the original 1 GB.
Does this lead to anything and can i possibly do something about it.
Out of interest: Have you tried splitting up the process so that it can run in parallel?
A 50hr job is a big job! You have a decent machine there (8 cores and 24GB Ram) and you should be able to parallelise some parts of it.