In my program, closing a java.util.RandomAccessFile sometimes takes exactly 45 seconds (well, almost exactly: between 44.998 and 45.003 seconds). The program creates and closes lots of small files. Usually closing the file is very quick (between 0 and 0.1 seconds). If I debug the program, it's stuck in the native method RandomAccessFile.close0.
The same problem also occurs when using FileOutputStream instead of RandomAccessFile (in which case the program is blocked in the native method FileOutputStream.close0).
Has somebody an idea what that could be? Can you reproduce the problem on your system (I can reproduce it only on a Mac, not on Windows XP; I didn't test yet on Linux)?
Update 2:
This only seems to happend on Mac OS X. I use JDK 1.6.0_22-b04. It happens on both 32-bit and 64-bit. On Windows XP it doesn't seem to occur.
My test case is:
import java.io.File;
import java.io.RandomAccessFile;
public class TestFileClose {
public static void main(String... args) throws Exception {
for (int i = 0; i < 100000; i++) {
String name = "test" + i;
RandomAccessFile r = new RandomAccessFile(name, "rw");
r.write(0);
long t = System.currentTimeMillis();
r.close();
long close = System.currentTimeMillis() - t;
if (close > 200) {
System.out.println("closing " + name +
" took " + close + " ms!");
}
if (i % 2000 == 0) {
System.out.println("test " + i + "/100000");
}
new File(name).delete();
}
}
}
Example output on my machine:
test 0/100000
test 2000/100000
test 4000/100000
test 6000/100000
test 8000/100000
test 10000/100000
closing test10030 took 44998 ms!
test 12000/100000
test 14000/100000
test 16000/100000
closing test16930 took 44998 ms!
test 18000/100000
test 20000/100000
In my case, it turned out to be McAfee antivirus installed on my machine. I had to install it (company policy)...
The problem also showed up if I disabled the on-access scan.
It could be garbage collection activity, triggered by opening/closing a large number of RandomAccessFile objects; there may be nothing magic about 45 seconds - it could just be the time it takes the JVM on your machine to traverse the heap scavenging for objects to free. Having said that, 45 seconds is an awfully long GC pause; One application I worked on recently always suffered full GC's of about 11 seconds.
Try monitoring your program using JConsole or JVisualVM, or when you start the program try adding the following options:
-verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Then have a look in the gc.log file produced to see what the application stop times are; if you instrument your code to print timestamps you may be able to tie the close() behaviour to specific GC activity:
...
if (close > 200) {
System.out.println(new Date());
System.out.println("closing " + name +
" took " + close + " ms!");
}
...
If it is GC related, in the gc.log file, you'd be looking for Full garbage collections and/or application stop times at around the timestamps your program outputs files.
Tinkering with the heap settings (-Xmx=... and XX:MaxPermSize=...) may then give you a completely different profile.
On a side note, if it is a temporary file, try using File file = File.createTempFile(prefix, suffix) and pass that into the RandomAccessFile - this may create files in /var/tmp (or whatever its called) on OS X, thus using an in-memory file system instead of a disk-based file system.
Related
When I build my project with gradle, the test outputs are huge and I would like to keep them. Therefore, I activated showing the output streams:
test {
testLogging.showStandardStreams = true
}
Unfortunately, gradle does not seem to be able to handle big standard outputs. For demonstration, I created a minimal example: https://github.com/DaGeRe/stdout-test It is a project creating big standard output in a test by
#Test
public void test() {
long start = System.currentTimeMillis();
for (int i = 0; i < 200; i++) {
for (int j = 0; j < 10000; j++) {
long current = System.currentTimeMillis() - start;
System.out.println("This is a simple logging output: " + i + " " + j + " " + current);
}
}
}
If I run this in a standard maven project, it finishes in about 2 minutes:
reichelt#reichelt-desktop:~/workspaces/stdout-test$ time mvn test &> mvn.txt
real 1m34,130s
user 0m31,333s
sys 1m12,296s
If I run it in gradle by time ./gradlew test &> gradle.txt, it does not finish at all (in reasonable time) and the output contains many Expiring Daemon because JVM heap space is exhausted. A way to solve this temporarily would be increasing heap memory (like suggested here: JVM space exhausted when building a project through gradle ), but -Xmx4g does not change anything according to my experiments, and this obviously will not scale for bigger outputs. Also, running ./gradlew -i test does not change the behavior.
The project also contains example files for outputs from maven (https://github.com/DaGeRe/stdout-test/blob/master/mvn.txt) and gradle (https://github.com/DaGeRe/stdout-test/blob/master/gradle.txt - aborted process after ~10 minutes) and one of the heap dumps (https://github.com/DaGeRe/stdout-test/blob/master/java_pid10812.hprof.tar) gradle created. There are only minor increases in the current time in the gradle-log (third output number of every line). Therefore, I assume that gradle mainly has problems printing to stdout and not executing the program.
This shows that, while gradle has some problems printing to stdout, it seems to not block the test execution. Is there any switch or parameter I could give gradle, which forces gradle to directly print to stdout instead of doing its memory-intense processing? Unfortunately, I did not find any in the documentation (https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html).
EDIT Just finished a test run on a server:
reichelt#r147:~/workspaces/dissworkspace/stdout-test$ time ./gradlew test &> gradle.txt
real 28m17,959s
user 216m37,351s
sys 0m12,410s
Ends with an exception:
This is a simple logging output: 89 7842 1416
This is a simple logging output: 89 7843 1416
This is a simple logging output: 89 7844 1416
This is a simple logging output: 89 7845 1416
This is a simple logging output: 89 7846 1416
FAILURE: Build failed with an exception.
* What went wrong:
GC overhead limit exceeded
I don't really have a solution for you, but I want to share a few observations that are too long to put into a comment.
First of all, I can reproduce your OutOfMemory problem from your Github repository. I googled it a bit, and while there are other reports on OOM on this out there, none had a solution. I think it is just a limitation in Gradle when enabling showStandardStreams. I tried fiddling around with the console output type and a few other parameters, but none had an effect.
However, with disabling showStandardStreams, I would not get an OOM. Not even after bumping the number of iterations from 200*10000, that you specified, to 1000*10000. It worked fine and the output got saved to both a .bin, .xml and a .html file for later inspection.
What's more, Gradle ran it more than twice as fast as Maven on my machine:
λ time ./gradlew test &> gradle.txt
real 1m23.113s
user 0m0.015s
sys 0m0.031s
λ time mvn test &> mvn.txt
real 3m6.671s
user 0m0.183s
sys 0m0.566s
Not sure why there is such a big difference between the two.
While I completely agree that it would be nice to use showStandardStreams for large outputs, just like Maven defaults to, it appears it is just not possible unless you can afford to raise the maximum heap size accordingly. On the other hand, having the output saved in the report is also rather nice, which is something you don't get from the Surefire plugin in Maven.
I spent days looking for a way of reading the output of a console app in realtime. Every way I came across doing it was almost identical to all others, but somehow didn't seem to work. Whenever I ran the code, my console command would not output any text for several minutes, leading me to assume that it was actually outputting data at the end of it's execution rather than during. However, now I know that there is some kind of bottleneck and I don't believe it's with my code.
It's difficult to hand over a more minimal reproducible example, because I believe it would make it more difficult to comprehend, so please excuse the verbosity. I'm not even sure if this issue can be reproduced at all, since it appears to be a problem with the environment.
I'm running Windows 7 (I know, I'm uncool).
I'm using IntelliJ IDEA IDE (which almost out-weighs the uncoolness of running Windows 7).
Here's the output of java --version:
Java 11.0.3 2019-04-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.3+12-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.3+12-LTS, mixed mode)
To illustrate the problem, I have the following code:
public class Main {
private static Timer timer;
public static void main(String[] args) throws IOException {
java.lang.String[] commands = {"cmd", "/c", "sfc", "/scannow"};
ProcessBuilder pBuilder = new ProcessBuilder(commands);
pBuilder.redirectErrorStream();
timer = new Timer();
try {
dbg("Starting process...");
Process p = pBuilder.start();
dbg("Process started.");
InputStream in = p.getInputStream();
final Scanner scanner = new Scanner(in);
new Thread(() -> {
dbg("Thread started.");
int i = 1;
while (scanner.hasNextLine()) {
String nextLine = scanner.nextLine();
dbg("Line " + i + " - " + nextLine);
i++;
}
scanner.close();
dbg("Closing scanner.");
}).start();
dbg("Waiting for exit status result...");
int exitStatus = p.waitFor();
dbg("exit status: " + exitStatus);
p.destroy();
} catch (NullPointerException | InterruptedException | IOException e) {
e.printStackTrace();
}
}
private static void dbg(String messageToAppend) {
System.out.println(timer.getTimePassedAsStr(messageToAppend));
}
}
I've made a simple timer class to help me debug the code, since it's taking a lot longer to start running than I'd expect. It works fine, so I won't post the code. However, here is the output of running that I get:
[000:00.000] Starting process...
[000:00.149] Process started.
[000:00.150] Input stream created.
[000:00.204] Scanner created.
[000:00.205] Waiting for exit status result...
[000:00.205] Thread started.
[005:51.762] Line 1 -
[005:51.763] Line 2 -
[005:51.763] Line 3 - Beginning system scan. This process will take some time.
[005:51.763] Line 4 -
[005:51.763] Line 5 -
[005:51.764] Line 6 -
[005:51.764] Line 7 - Beginning verification phase of system scan.
[005:51.764] Line 8 -
[012:42.484] Line 9 - Verification 100% complete.
[012:42.485] Line 10 - Windows Resource Protection found corrupt files but was unable to fix some of them.
[012:42.486] Line 11 -
[012:42.486] Line 12 - Details are included in the CBS.Log windir\Logs\CBS\CBS.log. For example
[012:42.487] Line 13 -
[012:42.487] Line 14 - C:\Windows\Logs\CBS\CBS.log
[012:42.487] Line 15 -
[012:42.488] Closing scanner.
[012:42.491] exit status: 0
Usually, when I run the sfc /scannow directly from the console, it starts up almost instantly, but when I run it from within my Java app, there's almost a 6 minute delay before I get any output.
I have also tried executing it through the CMD interpreter like so:
java.lang.String[] commands = {"cmd", "/c", "sfc", "/scannow"};
But I get the same result. I've also tried running it as a single thread, like so:
public static void runSfc() throws IOException {
timer = new Timer();
Runtime rt = Runtime.getRuntime();
String[] commands = {"sfc", "/scannow"};
dbg("Running command...");
Process proc = rt.exec(commands);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(proc.getInputStream()));
BufferedReader stdError = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
String s;
int i = 1;
dbg("Here is the standard output of the command:");
while ((s = stdInput.readLine()) != null) {
dbg("Line " + i + " - " + s);
i++;
}
i = 1;
dbg("Here is the standard error of the command (if any):");
while ((s = stdError.readLine()) != null) {
dbg("Line " + i + " - " + s);
i++;
}
}
It appears that each subsequent run, be it single-threaded or multi-threaded, takes either roughly the same amount of time or even longer and inexplicably subsequently shorter again. The latest run stands at 15 minutes before I saw the first line of output in the console window.
The app needs to be run as Administrator, if that makes any difference?
It makes no difference whether I run the code from my IDE, or compile it and run it from the command line.
I have tried reinstalling Java - no difference.
I tried using a version ? instead - no difference.
Curiously, when the scan is unable to run because it's already running elsewhere, it started it's initial output after about 25 seconds (down by over 5 minutes). It's the same situation when run within a thread too.
[000:00.000] Running command...
[000:00.203] Here is the standard output of the command:
[000:25.508] Line 1 -
[000:25.509] Line 2 -
[000:25.509] Line 3 - Beginning system scan. This process will take some time.
[000:25.509] Line 4 -
[000:25.509] Line 5 -
[000:25.509] Line 6 - Another servicing or repair operation is currently running.
[000:25.509] Line 7 -
[000:25.509] Line 8 - Wait for this to finish and run sfc again.
[000:25.509] Line 9 -
[000:25.510] Here is the standard error of the command (if any):
[000:25.510] End of execution
Question 1: Why is it taking more than 5 minutes for the process to output anything, when running it from the command line yields output almost instantly? Does anyone else on a Window machine experience this?
Question 2: I wanted to receive progress updates as the scan progressed. I've not realised, that since the progress updates are preceded with a \r and don't end with a \n until the process gets to 100%, that I won't see any of update messages until the scan has finished, since it never actually registers as having received a line of output. Obviously, I need to read bytes instead of lines. Does anyone know a way of reading this kind of text (that starts with a \r and doesn't end with a \n until the end of the process) without my having to reinvent the wheel?
EDIT:
Mean average execution time when executing directly via CLI
13m 53s (5 runs)
Mean average execution time when executing via the Java app
12m 32s (5 runs)
So it looks like execution time within the Java app is roughly the same as executing it directly in the console, and that the execution time isn't slow at all. It's just that it takes a good while before the first line of output is displayed.
I can only conclude that this must indeed be a buffer-flushing issue as #Andreas suggested. As it stands, I see no way around the problem.
On my university’s high end computing cluster, I use the following script to run a Java program (repls.class):
#$ -S /bin/bash
#$ -q serial
#$ -l h_vmem=10G
source /etc/profile
module add java
export CLASSPATH=$CLASSPATH:`pwd`
java -Xmx3000M -Xms128M -Djava.awt.headless=true -classpath ".:./netlogo/app/netlogo-6.1.0.jar" repls 1 1 1 5 "E,16070608MAT,16070608MAT,L160706-08-MATHS-R3.txt,L160706-08-MATHS.csv"
(The cluster operating system is CentOS Linux, with job submission handled by Son of Grid Engine.)
‘repls.class’ starts NetLogo, running my program ‘VizSim19Calib.nlogo’ headlessly. It also sets several global variables for the run.
‘VizSim19Calib.nlogo’ runs many simulations (replications – but testing with 5).
The problem is that each simulation is taking approx. 3 s to run, whereas on my own desktop each simulation takes approx. 1.5 s!
It doesn’t matter what settings I use for virtual memory, heap or stack – even doubling these makes no difference, viz.: #$ -l h_vmem=20G and -Xmx6000M -Xms256M
Why does the simulation run so slowly?
Could the location of the NetLogo class and jar files be responsible?
They are in directories under my home folder.
My Java program ‘repls.java’ is basically:
import org.nlogo.headless.HeadlessWorkspace;
public class repls {
public static void main(String[] args) {
try { …
HeadlessWorkspace workspace =
HeadlessWorkspace.newInstance() ;
try {
workspace.open("VizSim19Calib.nlogo",false);
workspace.command("startup");
workspace.command("set Test? false");
workspace.command("set SIMULATION-RUN-ID " + args[0]);
…
workspace.command(
"RunOneLessonParamReps SelectedLessonData #Replications"
);
workspace.dispose();
}
catch(Exception ex) {
…
}
}
catch (NumberFormatException e) {
…
} } }
I realized that NetLogo was using all 4 cores on my desktop, but only 1 on one of the university cluster's single compute nodes, by default. I increased this to 4 then 8 cores, and the speed improvement is as desired! I'll try 16 tonight. I think the matter is closed.
I have a cross-platform (Linux and Windows) program, currently running on Java 7. The program will launch one or more workers as Processes using ProcessBuilder (using instructions from here):
String javaHome = System.getProperty("java.home");
String javaBin = javaHome + File.separator + "bin" + File.separator + "java";
String classpath = System.getProperty("java.class.path");
String className = MyClass.class.getCanonicalName();
ProcessBuilder pb = new ProcessBuilder(javaBin, "-cp", classpath, className);
pb.directory(null);
pb.inheritIO();
Process p = pb.start();
The reason for this approach was designed to solve an issue with the older system, which ran the workers in threads. However, due to the nature of the code we're running, it's possible for the workers to enter a loop from which they won't exit. Since Java does not support the idea of terminating a thread, we moved to a completely separate process under the assumption that Java could actually kill a process.
However, it appears that isn't true in Java 7 - Process.destroy() sends a SIGTERM, not a SIGKILL, and our stalled workers aren't being killed as we'd expect. Java 8 implemented Process.destroyForcibly(), which would solve the problem, but upgrading the core Java version is likely to introduce a number of bugs, and upgrading is something we'd like to avoid if at all possible.
Most other solutions involve reflecting into the UNIXProcess class, getting the pid, and piping a kill command to the shell. However, this won't work for us as we run on Windows, where Process does not have any grasp of a pid, and further forces us to include OS-specific code paths which is extremely undesirable.
Is there some reliable way to get Java to terminate something?
You can try this API SIGAR.
Sample code:
final Sigar sigar = new Sigar();
final long[] processes = sigar.getProcList();
for (final long processId : processes) {
System.out.println(processId + " = " + ProcUtil.getDescription(sigar, processId));
final String processDescription = ProcUtil.getDescription(sigar, processId);
if(processDescription.contains("notepad.exe")){
System.out.println("Found notepad.exe with id [" + processId + "] - KILLING IT!");
sigar.kill(processId, -9);
}
}
Code source:Link
The solution in my case, since I was writing both the parent and the child processes, was to add a ShutdownHook which halted the JVM (a simple System.exit(1) was not sufficient):
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
Runtime.getRuntime().halt(5);
}
});
After this, when a SIGTERM was received the process would terminate, as though it had been sent SIGKILL by the parent process.
The attached simple Java code should load all available cpu core when starting it with the right parameters. So for instance, you start it with
java VMTest 8 int 0
and it will start 8 threads that do nothing else than looping and adding 2 to an integer. Something that runs in registers and not even allocates new memory.
The problem we are facing now is, that we do not get a 24 core machine loaded (AMD 2 sockets with 12 cores each), when running this simple program (with 24 threads of course). Similar things happen with 2 programs each 12 threads or smaller machines.
So our suspicion is that the JVM (Sun JDK 6u20 on Linux x64) does not scale well.
Did anyone see similar things or has the ability to run it and report whether or not it runs well on his/her machine (>= 8 cores only please)? Ideas?
I tried that on Amazon EC2 with 8 cores too, but the virtual machine seems to run different from a real box, so the loading behaves totally strange.
package com.test;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
public class VMTest
{
public class IntTask implements Runnable
{
#Override
public void run()
{
int i = 0;
while (true)
{
i = i + 2;
}
}
}
public class StringTask implements Runnable
{
#Override
public void run()
{
int i = 0;
String s;
while (true)
{
i++;
s = "s" + Integer.valueOf(i);
}
}
}
public class ArrayTask implements Runnable
{
private final int size;
public ArrayTask(int size)
{
this.size = size;
}
#Override
public void run()
{
int i = 0;
String[] s;
while (true)
{
i++;
s = new String[size];
}
}
}
public void doIt(String[] args) throws InterruptedException
{
final String command = args[1].trim();
ExecutorService executor = Executors.newFixedThreadPool(Integer.valueOf(args[0]));
for (int i = 0; i < Integer.valueOf(args[0]); i++)
{
Runnable runnable = null;
if (command.equalsIgnoreCase("int"))
{
runnable = new IntTask();
}
else if (command.equalsIgnoreCase("string"))
{
runnable = new StringTask();
}
Future<?> submit = executor.submit(runnable);
}
executor.awaitTermination(1, TimeUnit.HOURS);
}
public static void main(String[] args) throws InterruptedException
{
if (args.length < 3)
{
System.err.println("Usage: VMTest threadCount taskDef size");
System.err.println("threadCount: Number 1..n");
System.err.println("taskDef: int string array");
System.err.println("size: size of memory allocation for array, ");
System.exit(-1);
}
new VMTest().doIt(args);
}
}
I don't see anything wrong with your code.
However, unfortunately, you can't specify the processor affinity in Java. So, this is actually left up to the OS, not the JVM. It's all about how your OS handles threads.
You could split your Java threads into separate processes and wrap them up in native code, to put one process per core. This does, of course, complicate communication, as it will be inter-process rather than inter-thread. Anyway, this is how popular grid computing applications like boink work.
Otherwise, you're at the mercy of the OS to schedule the threads.
I would guess this is inherent to the JVM/OS and not necessarily your code. Check the various JVM performance tuning docs from Sun, e.g. http://ch.sun.com/sunnews/events/2009/apr/adworkshop/pdf/5-1-Java-Performance.pdf which suggests using numactl on Linux to set the affinity.
Good luck!
Apparently your VM is running in so-called "client" mode, where all Java threads are mapped to one native OS thread and consequently are run by one single CPU core. Try to invoke the JVM with -server switch, this should correct the problem.
If you get an: Error: no 'server' JVM found, you'll have to copy the server directory from a JDK's jre\bin directory to JRE's bin.
uname -a
2.6.18-194.11.4.el5 #1 SMP Tue Sep 21 05:04:09 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
Intel(R) Xeon(R) CPU E5530 # 2.40GHz
http://browse.geekbench.ca/geekbench2/view/182101
Java 1.6.0_20-b02
16cores, the program consumed 100% cpu as shown by vmstat
Interestingly I came to this article because I am suspecting my application is not utilizing all the cores as the cpu utilisation never increases but the response time starts deteriorating
I've noticed even on C that a tight loop often has issues like that. You'll also see pretty vast differences depending on OS.
Depending on the reporting tool you are using, it may not report the CPU used by some core services.
Java tends to be pretty friendly. You might try the same thing in linux but set the process priority to some negative number and see how it acts.
Setting thread priorities inside the app may help a little too if your jvm isn't using green threads.
Lots of variables.