I'm trying to spawn the following bash script and capture its stdout asynchronously:
#!/bin/bash
sleep 2;
echo "one";
sleep 2;
echo "two";
sleep 2;
echo "three";
# ... possibly infinite..
Here is the Java Code so far:
ProcessBuilder pb = new ProcessBuilder("sleeper");
Process process = pb.start();
InputStream input = process.getInputStream();
// now I continue in pseudo-code:
supervise the input-stream.
whenever a new line arrives:
check if the line equals "two";
then: doSomeAction();
Note: I'm not actually writing the program in Java. I'm writing it in Clojure, but I haven't found a Clojure approach to do so yet. So I'm trying to use the Java native API wrapped by Clojure.
a nodejs example
Tho clarify my intention a bit more, here's a JavaScript Code for Node.JS, that does exactly, what I want:
const spawn = require('child_process').spawn;
const sleeper = spawn('sleeper');
sleeper.stdout.on('data', (data) => {
if (data.toString() === "two\n") {
doSomeAction();
}
});
You can use the tools in clojure.java.io together with straightforward Java interop.
Here’s some code to get you started:
(require '[clojure.java.io :refer [reader]])
(let [process (.start (ProcessBuilder. ["./sleeper"]))]
(with-open [r (reader (.getInputStream process))]
(doseq [line (line-seq r)]
(when (= line "two")
(println line)))))
Paste this into your REPL and you should see two being output after the appropriate delay.
Related
I'm parsing the result of executing this composite command
ntpq -c peers | awk ' $0 ~ /^*/ {print $9}'
in order to obtain the offset of the active ntp server.
This is the java code used and executed periodically
public Double getClockOffset() {
Double localClockOffset = null;
try {
String[] cmd = {"/bin/sh",
"-c",
"ntpq -c peers | awk \' $0 ~ /^\\*/ {print $9}\'"};
Process p = Runtime.getRuntime().exec(cmd);
p.waitFor();
BufferedReader buf = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = buf.readLine();
if (!StringUtils.isEmpty(line)) {
localClockOffset = Double.parseDouble(line.trim());
} else {
// Log "NTP -> Empty line - No active servers - Unsynchronized"
}
} catch (Exception e) {
// Log exception
}
return localClockOffset;
}
ntpq result example
> remote refid st t when poll reach delay offset jitter
> ==============================================================================
> *server001s1 .LOCL. 1 u 33 64 377 0.111 -0.017 0.011
> +server002s1 10.30.10.6 2 u 42 64 377 0.106 -0.006 0.027
> +server003s1 10.30.10.6 2 u 13 64 377 0.120 -0.009 0.016
Notice that awk searchs the first line beginnig with '*' and extracts its ninth column. In the example: -0.017
The problem is that sometimes I'm obtaining the no-active-servers log message - intended to appear when there is no server with '*'- while the execution of the command through the console returns a number.
I know that I'm not closing the BufferedReader in that code but is that the reason of this behaviour? A new instance is being created (and left open until garbage collecting) in each method invocation but I think that it shouldn't be the cause of this problem.
Runtime.exec() simply invokes the ProcessBuilder inside it, like that:
public Process More ...exec(String[] cmdarray, String[] envp, File dir)
throws IOException {
return new ProcessBuilder(cmdarray)
.environment(envp)
.directory(dir)
.start();
}
see OpenJDK Runtime.java
So there is nothing wrong with using it instead of the ProcessBuilder as is.
The problem is that you invoke:
p.waitFor();
before you obtained the InputStream.
Which means that the process will be already terminated, by the time you obtain the InputStream, and the output stream data might be or might not be available to you, depending on the OS buffering implementation nuances and precise timing of the operations.
So, if you move the waitFor() to the bottom, your code should start working more reliably.
Under Linux however you should normally be able to read the remaining data from the PIPE buffer, even after the writing process has ended.
And the UNIXProcess implementation in OpenJDK, actually makes an explicit use of that, and tries to drain the remaining data, once the process has exited, so that file descriptor can be reclaimed:
/** Called by the process reaper thread when the process exits. */
synchronized void processExited() {
synchronized (closeLock) {
try {
InputStream in = this.in;
// this stream is closed if and only if: in == null
if (in != null) {
byte[] stragglers = drainInputStream(in);
in.close();
this.in = (stragglers == null) ?
ProcessBuilder.NullInputStream.INSTANCE :
new ByteArrayInputStream(stragglers);
}
} catch (IOException ignored) {}
}
}
And this seems to work reliable enough, at least in my tests, so it would be nice to know which specific version of Linux|Unix and JRE your are running.
Have you also considered the possibility of an application-level problem ?
I.e. ntpq is not really guaranteed to always return a * row.
So, it would be nice to remove the awk part from your pipe, to see if there will be some output at all the times.
Another thing to note is that if one of your shell pipeline steps fails (e.g. the ntpq itself), you will also get an empty output, so you will have to track the STDERR as well (e.g. by merging it with STDOUT via the ProcessBuilder).
Sidenote
Doing waitFor before you start consuming the data, is a bad idea in any case, as if your external process will produce enough output to fill the pipe buffer, it will just hang waiting for someone to read it, which will never happen, as your Java process will be locked in waitFor at the same time.
As pointed by Andrew Thompson, you shall try ProcessBuilder instead.
String[] cmd = {"/bin/sh",
"-c",
"ntpq -c peers | awk \' $0 ~ /^\\*/ {print $9}\'"};
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);
Process proc = pb.start();
BufferedReader buf = new BufferedReader(new
InputStreamReader(proc.getInputStream()));
String line = null;
while ((line = buf.readLine()) != null) {
localClockOffset = Double.parseDouble(line.trim());
break;
}
proc.destroy();
Ref ProcessBuilder
Finally we have found the real problem.
I'm not gonna change the accepted anwser, I think that it's useful too but maybe someone can learn from our experience.
My java program is launched with a shell script. When we execute the script manually, ntpq command is found and invoked successfully. The problem arises when the software is fully deployed. In the final environment we've got a cron scheduled demon that keeps our program alive but PATH established by cron is different from the PATH that our profile has got assigned.
PATH used by cron:
.:/usr/bin:/bin
PATH that we got login for launching the script manually:
/usr/sbin:/usr/bin:/bin:/sbin:/usr/lib:/usr/lib64:/local/users/nor:
/usr/local/bin:/usr/local/lib:.
Usually ntpq is in
/usr/sbin/ntpq
After we found the key of our problem, I search StackOverflow and got this relevant question where the problem is better explained and solved.
How to get CRON to call in the correct PATHs
Is there any sane way to run a system command from Java that ignores STDOUT and STDERR? For example, when I try the following:
Process p = Runtime.getRuntime().exec("some_executable_path param1 param2 >NUL 2>&1");
Java tries to parse the command, and ends up escaping the arguments (e.g., prevents the output from being redirected). If I don't redirect STDOUT/STDERR, the buffers fill up and prevent the system call from exiting. The following does what I want, but is extremely cumbersome and creates expensive resources just to throw the output of the system call away:
ProcessBuilder pb = new ProcessBuilder("some_executable_path", "param1", "param2");
pb.redirectErrorStream(true);
final Process p = pb.start();
final Thread redirectToNull = new Thread(() -> {
final InputStream stdout = process.getInputStream();
try {
while (stdout.read() != -1);
} catch (final Exception e) {
// Don't care
}
}, "Output Consumer Thread");
redirectToNull.setDaemon(true);
redirectToNull.start();
I realize the Java design team is known to be masochistic, but this is ridiculous. I would prefer to deliver a batch or Perl script that wraps the system call with my application rather than use the above code. There has to be an easier way to accomplish this.
So the question is, is there any sane way to run a system command from within Java and ignore the output printed to STDOUT/STDERR?
It's not that Java 'prevents' redirection, it just doesn't affirmatively do it, and neither does your program. When you give CMD a command like program arg1 arg2 >out 2>err <in, it is CMD that sets up those redirections and then invokes program with arg1 arg2 only, not >out etc. On Unix the shells do the same -- there is a choice of several shells, but all of them handle redirection like this. Similarly pipes are set up by CMD or shells, not by either or all of the programs run in those pipes.
Thus on Windows the way to do this is either run CMD and have it do the redirections:
Process p = new ProcessBuilder ("cmd", "/c", "program arg1 arg2 >NUL 2>&1").start();
// this uses CMD's default parsing for args, so they must not contain space
// unless you insert 'data' quotes, or things that look like a substitutable %var%
or (assuming Java7+) tell ProcessBuilder to do the redirections:
pb.redirectOutput (new File("NUL")).redirectErrorStream(true)
How do I start processes from a script in a way that also allows me to terminate them?
Basically, I can easily terminate the main script, but terminating the external processes that this main script starts has been the issue. I googled like crazy for Perl 6 solutions. I was just about to post my question and then thought I'd open the question up to solutions in other languages.
Starting external processes is easy with Perl 6:
my $proc = shell("possibly_long_running_command");
shell returns a process object after the process finishes. So, I don't know how to programmatically find out the PID of the running process because the variable $proc isn't even created until the external process finishes. (side note: after it finishes, $proc.pid returns an undefined Any, so it doesn't tell me what PID it used to have.)
Here is some code demonstrating some of my attempts to create a "self destructing" script:
#!/bin/env perl6
say "PID of the main script: $*PID";
# limit run time of this script
Promise.in(10).then( {
say "Took too long! Killing job with PID of $*PID";
shell "kill $*PID"
} );
my $example = shell('echo "PID of bash command: $$"; sleep 20; echo "PID of bash command after sleeping is still $$"');
say "This line is never printed";
This results in the following output which kills the main script, but not the externally created process (see output after the word Terminated):
[prompt]$ ./self_destruct.pl6
PID of the main script: 30432
PID of bash command: 30436
Took too long! Killing job with PID of 30432
Terminated
[prompt]$ my PID after sleeping is still 30436
By the way, the PID of sleep was also different (i.e. 30437) according to top.
I'm also not sure how to make this work with Proc::Async. Unlike the result of shell, the asynchronous process object it creates doesn't have a pid method.
I was originally looking for a Perl 6 solution, but I'm open to solutions in Python, Perl 5, Java, or any language that interacts with the "shell" reasonably well.
For Perl 6, there seems to be the Proc::Async module
Proc::Async allows you to run external commands asynchronously, capturing standard output and error handles, and optionally write to its standard input.
# command with arguments
my $proc = Proc::Async.new('echo', 'foo', 'bar');
# subscribe to new output from out and err handles:
$proc.stdout.tap(-> $v { print "Output: $v" });
$proc.stderr.tap(-> $v { print "Error: $v" });
say "Starting...";
my $promise = $proc.start;
# wait for the external program to terminate
await $promise;
say "Done.";
Method kill:
kill(Proc::Async:D: $signal = "HUP")
Sends a signal to the running program. The signal can be a signal name ("KILL" or "SIGKILL"), an integer (9) or an element of the Signal enum (Signal::SIGKILL).
An example on how to use it:
#!/usr/bin/env perl6
use v6;
say 'Start';
my $proc = Proc::Async.new('sleep', 10);
my $promise= $proc.start;
say 'Process started';
sleep 2;
$proc.kill;
await $promise;
say 'Process killed';
As you can see, $proc has a method to kill the process.
Neither Perl, Perl 6, nor Java, but bash:
timeout 5 bash -c "echo hello; sleep 10; echo goodbye" &
In Java you can create a process like this:
ProcessBuilder processBuilder = new ProcessBuilder("C:\\Path\program.exe", "param1", "param2", "ecc...");
Process process = processBuilder.start(); // start the process
process.waitFor(timeLimit, timeUnit); // This causes the current thread to wait until the process has terminated or the specified time elapses
// when you want to kill the process
if(process.isAlive()) {
process.destroy();
}
Or you can use process.destroyForcibly();, see the Process documentation for more info.
To execute a bash command point to the bash executable and set the command as a parameter.
I use Google Closure Compiler to compile automatically javascript using PHP (is needed to do it that way - in PHP, hovewer no security limitations on Windows machine). I wrote simple PHP script which calls process, pass .js content to stdin and receive recompiled .js via stdout. It works fine, problem is, when I compiling for example 40 .js files, it takes on strong machine almost 2 minutes. However, mayor delay is because java starts new instance of .jar app for every script. Is there any way how to modify script below to create process only one and send/receive .js content multiple times before process ends?
function compileJScript($s) {
$process = proc_open('java.exe -jar compiler.jar', array(
0 => array("pipe", "r"), 1 => array("pipe", "w")), $pipes);
if (is_resource($process)) {
fwrite($pipes[0], $s);
fclose($pipes[0]);
$output = stream_get_contents($pipes[1]);
fclose($pipes[1]);
if (proc_close($process) == 0) // If fails, keep $s intact
$s = $output;
}
return $s;
}
I can see several options, but don't know if it is possible and how to do it:
Create process once and recreate only pipes for every file
Force java to keep JIT-ed .jar in memory for much faster re-executing
If PHP can't do it, is possible to use bridge (another .exe file which will start fast every time, transfer stdin/out and redirects it to running compiler; if something like this even exists)
This is really a matter of coordination between the two process.
Here I wrote a quick 10-minutes script (just for the fun) that launches a JVM and sends an integer value, which java parses and returns incremented.. which PHP will just send it back ad-infinitum..
PHP.php
<?php
echo 'Compiling..', PHP_EOL;
system('javac Java.java');
echo 'Starting JVM..', PHP_EOL;
$pipes = null;
$process = proc_open('java Java', [0 => ['pipe', 'r'],
1 => ['pipe', 'w']], $pipes);
if (!is_resource($process)) {
exit('ERR: Cannot create java process');
}
list($javaIn, $javaOut) = $pipes;
$i = 1;
while (true) {
fwrite($javaIn, $i); // <-- send the number
fwrite($javaIn, PHP_EOL);
fflush($javaIn);
$reply = fgetss($javaOut); // <-- blocking read
$i = intval($reply);
echo $i, PHP_EOL;
sleep(1); // <-- wait 1 second
}
Java.java
import java.util.Scanner;
class Java {
public static void main(String[] args) {
Scanner s = new Scanner(System.in);
while (s.hasNextInt()) { // <-- blocking read
int i = s.nextInt();
System.out.print(i + 1); // <-- send it back
System.out.print('\n');
System.out.flush();
}
}
}
To run the script simply put those files in the same folder and do
$ php PHP.php
you should start seeing the numbers being printed like:
1
2
3
.
.
.
Note that while those numbers are printed by PHP, they are actually generated by Java
I don't think #1 from your list is possible because compiler.jar would need to have native support for keeping the process alive, which it doesn't (and if you consider that a compression algorithm needs the entire input before it can start processing data, it makes sense that the process doesn't stay alive).
According to Anyway to Boost java JVM Startup Speed? some people have been able to reduce their jvm startup times with nailgun
Nailgun is a client, protocol, and server for running Java programs
from the command line without incurring the JVM startup overhead.
Programs run in the server (which is implemented in Java), and are
triggered by the client (written in C), which handles all I/O.
How can I be notified when a process I did not start ends and is their a way to recover its exit code and or output? the process doing the watching will be running as root/administrator.
You can check whether a process is currently running from java by calling a shell command that lists all the current processes and parsing the output. Under linux/unix/mac os the command is ps, under windows it is tasklist.
For the ps version you would need to do something like:
ProcessBuilder pb = new ProcessBuilder("ps", "-A");
Process p = pb.start();
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
// Skip first (header) line: " PID TTY TIME CMD"
in.readLine();
// Extract process IDs from lines of output
// e.g. " 146 ? 00:03:45 pdflush"
List<String> runningProcessIds = new ArrayList<String>();
for (String line = in.readLine(); line != null; line = in.readLine()) {
runningProcessIds.add(line.trim().split("\\s+")[0]);
}
I don't know of any way that you could capture the exit code or output.
No (not on Unix/Windows, at least). You would have to be the parent process and spawn it off in order to collect the return code and output.
You can kind of do that. On Unix, you can write a script to continuously grep the list of running processes and notify you when the process you're searching for is no longer found.
This is pseudocode, but you can do something like this:
while ( true ) {
str = ps -Alh | grep "process_name"
if ( str == '' ) {
break
}
wait(5 seconds)
}
raise_alert("Alert!")
Check the man page for ps. You options may be different. Those are the ones I use on Mac OSX10.4.
looks like you could use jna to tie into the "C" way of waiting for a pid to end (in windows, poll OpenProcess( PROCESS_QUERY_INFORMATION ...) to see when it reports the process as dead, see ruby's win32.c