We Keep Coding

Java/JavaFX ProcessHandle possibly not finding all processes (Linux/Debian)

I've a JavaFX application where I've a list of a bunch of script files. Once the application loads, it reads it and and checks which ones are running.
To do that I use a ProcessHandle, as mentioned in various examples here on StackOverflow and other guides/tutorials on the internet.
The problem is, it never finds any of them. There for I programmatically started one, which I know for a fact that it will be running, via Process process = new ProcessBuilder("/path/to/file/my_script.sh").start(); - and it won't find this one either.
Contents of my_script.sh:
#!/bin/bash
echo "Wait for 5 seconds"
sleep 5
echo "Completed"
Java code:
// List of PIDs which correspond to the processes shown after "INFO COMMAND:"
System.out.println("ALL PROCESSES: " + ProcessHandle.allProcesses().toList());
Optional<ProcessHandle> scriptProcessHandle = ProcessHandle.allProcesses().filter(processHandle -> {
System.out.println("INFO COMMAND: " + processHandle.info().command());
Optional<String> processOptional = processHandle.info().command();
return processOptional.isPresent() && processOptional.get().equals("my_script.sh");
}).findFirst();
System.out.println("Script process handle is present: " + scriptProcessHandle.isPresent());
if (scriptProcessHandle.isPresent()) { // Always false
// Do stuff
}
Thanks to the good old fashioned System.out.println(), I noticed that I get this in my output console every time:
ALL PROCESSES: [1, 2, 28, 85, 128, 6944, 21174, 29029, 29071]
INFO COMMAND: Optional[/usr/bin/bwrap]
INFO COMMAND: Optional[/usr/bin/bash]
INFO COMMAND: Optional[/app/idea-IC/jbr/bin/java]
INFO COMMAND: Optional[/app/idea-IC/bin/fsnotifier]
INFO COMMAND: Optional[/home/username/.jdks/openjdk-17.0.2/bin/java]
INFO COMMAND: Optional[/usr/bin/bash]
INFO COMMAND: Optional[/home/username/.jdks/openjdk-17.0.2/bin/java]
INFO COMMAND: Optional[/home/username/.jdks/openjdk-17.0.2/bin/java]
INFO COMMAND: Optional[/usr/bin/bash]
Script process handle is present: false
The first line in the Javadoc of ProcessHandle.allProcess() reads:
Returns a snapshot of all processes visible to the current process.
So how come I can't see the rest of the operating system's processes?
I'm looking for a non-os-dependent solution, if possible. Why? For better portability and hopefully less maintenance in the future.
Notes:
A popular solution for GNU/Linux seems to be to check the proc entries, but I don't know if that would work for at least the majority of the most popular distributions - if it doesn't, adding support for them in a different way, would create more testing and maintenance workload.
I'm aware of ps, windir, tasklist.exe possible solutions (worst comes to worst).
I found the JavaSysMon library but it seems dead and unfortunately:
CPU speed on Linux only reports correct values for Intel CPUs
Edit 1:
I'm on Pop_OS! and installed IntelliJ via the PopShop as flatpak.
In order to start it as root as suggested by mr mcwolf, I went to /home/username/.local/share/flatpak/app/com.jetbrains.IntelliJ-IDEA-Community/x86_64/stable/active/export/bin and found com.jetbrains.IntelliJ-IDEA-Community file.
When I run sudo ./com.jetbrains.IntelliJ-IDEA-Community or sudo /usr/bin/flatpak run --branch=stable --arch=x86_64 com.jetbrains.IntelliJ-IDEA-Community in my terminal, I get error: app/com.jetbrains.IntelliJ-IDEA-Community/x86_64/stable not installed
So I opened the file and ran its contents:
exec /usr/bin/flatpak run --branch=stable --arch=x86_64 com.jetbrains.IntelliJ-IDEA-Community "$#"
This opens IntelliJ, but not as root, so instead I ran:
exec sudo /usr/bin/flatpak run --branch=stable --arch=x86_64 com.jetbrains.IntelliJ-IDEA-Community "$#"
Which prompts for a password and when I write it in, the terminal crashes.
Edit 1.1:
(╯°□°）╯︵ ┻━┻ "flatpak run" is not intended to be ran with sudo
Edit 2:
As mr mcwolf said, I downloaded the IntelliJ from the official website, extracted it and ran the idea.sh as root.
Now a lot more processes are shown. 1/3 of them show up as INFO COMMAND: Optional.empty.
scriptProcessHandle.isPresent() is still unfortunately returning false. I searched through them and my_script.sh is nowhere to be found. I also tried processOptional.isPresent() && processOptional.get().equals("/absolute/path/to/my_script.sh") but I still get false on isPresent() and it's not in the list of shown processes.
Though the last sentence might be a different problem. I'll do more digging.
Edit 3:
Combining .commandLine() and .contains() (instead of .equals()) solves the problem mentioned in "Edit 2".
Optional<ProcessHandle> scriptProcessHandle = ProcessHandle.allProcesses().filter(processHandle -> {
System.out.println("INFO COMMAND LINE: " + processHandle.info().commandLine());
Optional<String> processOptional = processHandle.info().commandLine();
return processOptional.isPresent() && processOptional.get().contains("/absolute/path/to/my_script.sh");
}).findFirst();
System.out.println("Script process handle is present: " + scriptProcessHandle.isPresent());
if (scriptProcessHandle.isPresent()) { // Returns true
// Do stuff
}
.commandLine() also shows script arguments, so that must be kept in mind.

Running the same Gradle task multiple times

So i have a gradle test task which runs all my tests. How can I set up gradle to run this task 100 times? It works and runs all my tests, I just need an option to choose how many times to run this.
The task in build.gradle:
test {
// enable JUnit Platform (a.k.a. JUnit 5) support
useJUnitPlatform()
// set a system property for the test JVM(s)
systemProperty 'some.prop', 'value'
// explicitly include or exclude tests
include 'com/company/calculator/**'
// show standard out and standard error of the test JVM(s) on the console
testLogging.showStandardStreams = true
// set heap size for the test JVM(s)
minHeapSize = "128m"
maxHeapSize = "512m"
// set JVM arguments for the test JVM(s)
jvmArgs '-XX:MaxPermSize=256m'
// listen to events in the test execution lifecycle
beforeTest { descriptor ->
logger.lifecycle("Running test: " + descriptor)
}
// Fail the 'test' task on the first test failure
failFast = true
// listen to standard out and standard error of the test JVM(s)
onOutput { descriptor, event ->
logger.lifecycle("Test: " + descriptor + " produced standard out/err: " + event.message )
}
The use case is that i want to test performance of different assertions and mocking libraries (i have multiple branches with tests written using different libraries), to do that i need to run test suite multiple times.
To test performance i need to measure the time it takes to run these tests for example 100 times (maybe 1000), on each libraries set.

One option might be this --rerun-tasks flag.
gradle test --rerun-tasks
From the Gradle user guide.
Another option, from a similar question, is creating a subclass of the Test class that returns a task with multiple copies of all tests, code here: https://stackoverflow.com/a/41650455/1686615 .
There really are many ways to do this at different levels, with Gradle code as in that link, or perhaps in .gradle files, with a parameter passed into the test code, or on the command line. Maybe indicate more about your use case or if there is a particular level at which you'd like to make the change.

java.io.tmpdir when running java_test with bazel

motivation:
I have a test that needs to write a short temp file (must be < 107 characters).
Currently the test is using
Files.createTempFile(null,".sock");
issue
which when running
I'm trying to figure out the java.io.tmp value when running java test using bazel. The different options I have is:
Setting $TEST_TMPDIR (or without)
Using "local"=True (or without)
Here is the result:
# local=True + TEST_TMPDIR=/btmp:
/btmp/_bazel_ors/719f891d5db9fd5e73ade25b0c847fd1/execroot/__main__/_tmp/8be6e61521c57d3cfc8585efa880e1ac/1638063256753562848.sock
# local=False + TEST_TMPDIR=/btmp:
/btmp/_bazel_ors/719f891d5db9fd5e73ade25b0c847fd1/bazel-sandbox/5561433121200492142/execroot/__main__/_tmp/8be6e61521c57d3cfc8585efa880e1ac/4867903879018296623.sock
# local=True , no TEST_TMPDIR:
/private/var/tmp/_bazel_ors/719f891d5db9fd5e73ade25b0c847fd1/execroot/__main__/_tmp/8be6e61521c57d3cfc8585efa880e1ac/984443110479498941.sock
# local=False , no TEST_TMPDIR:
/private/var/tmp/_bazel_ors/719f891d5db9fd5e73ade25b0c847fd1/bazel-sandbox/6199384508952843116/execroot/__main__/_tmp/8be6e61521c57d3cfc8585efa880e1ac/4588114364301475150.sock
Seems like the shortest temp prefix I can get is:
/private/var/tmp/_bazel_ors/719f891d5db9fd5e73ade25b0c847fd1/execroot/__main__/_tmp/
which is 85 char long (way too long for my needs).
How can I safely play with this configuration and make it a lot shorter?
note:
My env is mac osx sierra and I'm running bazel 0.5.1

Solvable by adding this to the jvm_flags of the test target:
"jvm_flags" = ["-Djava.io.tmpdir=/tmp"],
But note that it would make the test less hermetic

You can also tell bazel where it should store its outputs --output_base=/tmp/foo.

Closing a RandomAccessFile sometimes takes exactly 45 seconds

In my program, closing a java.util.RandomAccessFile sometimes takes exactly 45 seconds (well, almost exactly: between 44.998 and 45.003 seconds). The program creates and closes lots of small files. Usually closing the file is very quick (between 0 and 0.1 seconds). If I debug the program, it's stuck in the native method RandomAccessFile.close0.
The same problem also occurs when using FileOutputStream instead of RandomAccessFile (in which case the program is blocked in the native method FileOutputStream.close0).
Has somebody an idea what that could be? Can you reproduce the problem on your system (I can reproduce it only on a Mac, not on Windows XP; I didn't test yet on Linux)?
Update 2:
This only seems to happend on Mac OS X. I use JDK 1.6.0_22-b04. It happens on both 32-bit and 64-bit. On Windows XP it doesn't seem to occur.
My test case is:
import java.io.File;
import java.io.RandomAccessFile;
public class TestFileClose {
public static void main(String... args) throws Exception {
for (int i = 0; i < 100000; i++) {
String name = "test" + i;
RandomAccessFile r = new RandomAccessFile(name, "rw");
r.write(0);
long t = System.currentTimeMillis();
r.close();
long close = System.currentTimeMillis() - t;
if (close > 200) {
System.out.println("closing " + name +
" took " + close + " ms!");
}
if (i % 2000 == 0) {
System.out.println("test " + i + "/100000");
}
new File(name).delete();
}
}
}
Example output on my machine:
test 0/100000
test 2000/100000
test 4000/100000
test 6000/100000
test 8000/100000
test 10000/100000
closing test10030 took 44998 ms!
test 12000/100000
test 14000/100000
test 16000/100000
closing test16930 took 44998 ms!
test 18000/100000
test 20000/100000

In my case, it turned out to be McAfee antivirus installed on my machine. I had to install it (company policy)...
The problem also showed up if I disabled the on-access scan.

It could be garbage collection activity, triggered by opening/closing a large number of RandomAccessFile objects; there may be nothing magic about 45 seconds - it could just be the time it takes the JVM on your machine to traverse the heap scavenging for objects to free. Having said that, 45 seconds is an awfully long GC pause; One application I worked on recently always suffered full GC's of about 11 seconds.
Try monitoring your program using JConsole or JVisualVM, or when you start the program try adding the following options:
-verbose:gc -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Then have a look in the gc.log file produced to see what the application stop times are; if you instrument your code to print timestamps you may be able to tie the close() behaviour to specific GC activity:
...
if (close > 200) {
System.out.println(new Date());
System.out.println("closing " + name +
" took " + close + " ms!");
}
...
If it is GC related, in the gc.log file, you'd be looking for Full garbage collections and/or application stop times at around the timestamps your program outputs files.
Tinkering with the heap settings (-Xmx=... and XX:MaxPermSize=...) may then give you a completely different profile.
On a side note, if it is a temporary file, try using File file = File.createTempFile(prefix, suffix) and pass that into the RandomAccessFile - this may create files in /var/tmp (or whatever its called) on OS X, thus using an in-memory file system instead of a disk-based file system.

Why does output appear in wrong order?

I'm trying to write a Groovy script that wraps another command and am having trouble with the stdout/stderr order. My script is below:
#!/usr/bin/env groovy
synchronized def output = ""
def process = "qrsh ${args.join(' ')}".execute()
def outTh = Thread.start {
process.in.eachLine {
output += it
System.out.println "out: $it"
}
}
def errTh = Thread.start {
process.err.eachLine {
output += it
System.err.println "err: $it"
}
}
outTh.join()
errTh.join()
process.waitFor()
System.exit(process.exitValue())
My problem is that the output doesn't appear on the terminal in the correct order. Below is the wrapper's output.
[<cwd>] wrap.groovy -cwd -V -now n -b y -verbose ant target
waiting for interactive job to be scheduled ...
Your interactive job 2831303 has been successfully scheduled.
Establishing builtin session to host <host> ...
Buildfile: build.xml
BUILD FAILED
Target "target" does not exist in the project "null".
Total time: 0 seconds
Your job 2831303 ("wrap.groovy") has been submitted
Below is the unwrapped command output.
[<cwd>] qrsh -cwd -V -now n -b y -verbose ant target
Your job 2831304 ("ant") has been submitted
waiting for interactive job to be scheduled ...
Your interactive job 2831303 has been successfully scheduled.
Establishing builtin session to host host ...
Buildfile: build.xml
BUILD FAILED
Target "target" does not exist in the project "null".
Total time: 0 seconds
Why does the "Your job has been submitted" message appear as the first line in one cast and the last line in another? I'm guessing it's related to Java libraries, not Groovy.

This is because of buffering. The threads which read stdout and stderr will not process the output the moment it is written by the child process. Instead, both streams are buffered, so your process won't see anything unless the child flushes the streams).
When the data is on the way, which thread gets the CPU first? There is no way to tell. Even if the data for stderr arrives a few milliseconds before stdout, if the stdout thread has the CPU right now, it will get its data first.
What you could do is use Java NIO (channels) and a single thread and first process all output from stderr but that still wouldn't guarantee that the order is preserved. Because of the buffering between child and parent process, you could get 4KB of text from one stream before you see a single byte of the other.
Unfortunately, there is no cross-platform solution because Java doesn't have an API to merge the two streams into one. On Unix, you could run the command with sh -c cmd 2>&1. That would redirect stderr to stdout. In the parent process, you could then just read stdout and ignore stderr.
The same works for OS X (since it's Unix based). On Windows, you could install Perl or a similar tool to run the process; that allows you to mess with the file descriptors.
PS: Pray that args never contains spaces. String.execute() is a really bad way to run a process; use java.lang.ProcessBuilder instead.

Try putting System.out.flush after you do your println. If I am right, the messages are appearing in different orders because the System.out is being buffered.