We Keep Coding

Java on Linux unable to read process inputstream

I'm spawning a process using ProcessBuilder in Java and trying to read its output:
public static void main(String... args) throws IOException, InterruptedException {
ProcessBuilder builder = new ProcessBuilder("test.exe");
Process process = builder.start();
InputStream stdout = process.getInputStream();
final BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
final List<String> queue = new ArrayList<String>();
Thread ioThread = new Thread() {
public void run() {
System.out.println("STARTED THREAD");
String line = null;
try {
while ((line = reader.readLine()) != null) {
System.out.println("ADD: " + line);
queue.add(line);
Thread.yield();
}
} catch (IOException exception) {
System.err.println("Fatal Error: " + exception.getMessage());
}
}
};
ioThread.start();
Thread.sleep(10000);
System.out.println("GOT: " + queue);
}
This works perfectly fine on my Windows machine, and also on a local Linux installation.
On my Linux VM's (DigitalOcean and Linode), it does not work, however, meaning that the process gets started, but the running thread is unable to add a single line to the queue (after waiting ten seconds in the main program, queue is empty whereas in Windows, the queue contains the read lines).
Thinks I've tried so far:
redirecting the error stream to the output stream (even although I'm sure the test program is outputting lines on STDOUT
using stdbuf with -oL and -eL... doesn't work
trying to read byte by byte manually instead of using BufferedReader, same issue
converting thread to a FutureTask
switching between OpenJDK and Oracle VM
It definitely works in Windows and on some Linux machines.

It turned out to be a buffering issue after all: one stdbuf couldn't fix in this case.
Using a pseudo terminal with script did the trick, however, using:
ProcessBuilder builder = new ProcessBuilder();
builder.command().add("script");
builder.command().add("-c");
builder.command().add(command);
builder.command().add("-f");
builder.command().add("-q");

How to call python script from java and return long text?

I need to call python script from Java application and return big text (json response). I successfully called script and i can return some text to Java application. But when i tried return full json response - my Java application freezes and nothing happend.
How i call python script:
ProcessBuilder pb = new ProcessBuilder(
"python", absolutePath, params
);
Process ps = pb.start();
BufferedReader reader = new BufferedReader(
new InputStreamReader(ps.getInputStream())
);
ps.waitFor();
int ec = ps.exitValue();
StringBuffer HTTPResponse = new StringBuffer();
String inputLine;
while ((inputLine = reader.readLine()) != null) {
logger.debug(String.format("DEBUG!!!: input = %s", inputLine));
HTTPResponse.append(inputLine);
}

You are experiencing a deadlock between parent and child processes.
This is because your java application does not read anything from the child process until it has terminated. Python will buffer it's output (up to 64k on my system), and once the buffer has filled it will block on the next write. Because it is blocked, it will not terminate. Meanwhile the java parent is also blocked waiting for the child to terminate, which will not happen naturally.
To fix this you need to restructure your Java code such that it, rather than waiting for the child to terminate before processing its output, reads from the child until the child terminates, something like this:
import java.io.*;
public class TestProcess {
public static void main(String[] args)
throws InterruptedException, IOException
{
ProcessBuilder pb = new ProcessBuilder(
"python", absolutePath, params
);
Process ps = pb.start();
BufferedReader reader = new BufferedReader(
new InputStreamReader(ps.getInputStream())
);
String inputLine;
while ((inputLine = reader.readLine()) != null) {
System.out.println(inputLine);
}
ps.waitFor();
int ec = ps.exitValue();
System.out.println("Child returned " + ec);
}
}
And your Python script could be:
print 'a' * (1024*1024)
which would produce 1MiB of data.

runtime.exec sending EOF immediately to input?

This is my code to start a process in Windows via java (and gobble the output).
public static void main(String[] args) throws Exception {
String[] command = new String[3];
command[0] = "cmd";
command[1] = "/C";
command[2] = "test.exe";
final Process child = Runtime.getRuntime().exec(command);
new StreamGobbler(child.getInputStream(), "out").start();
new StreamGobbler(child.getErrorStream(), "err").start();
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(
child.getOutputStream()));
out.write("exit\r\n");
out.flush();
child.waitFor();
}
private static class StreamGobbler extends Thread {
private final InputStream inputStream;
private final String name;
public StreamGobbler(InputStream inputStream, String name) {
this.inputStream = inputStream;
this.name = name;
}
public void run() {
try {
BufferedReader in = new BufferedReader(new InputStreamReader(
inputStream));
for (String s = in.readLine(); s != null; s = in.readLine()) {
System.out.println(name + ": " + s);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Somehow the program in question (process) is recieving an EOF right away (as in right after I step pas the "exec" line) and thus throwing an error ( detected, invalid) message immediately after runtime.exec is called. I can run this program manually via command prompt without this issue, but have confirmed that sending a ctrl-z on windows is what causes this message.
Anyone know what could be causing this?
If it matters, I have tried running the process directly as "test.exe" instead of cmd /c test.exe, but when I do that I can't see the output via the inputStream. And when I do cmd test.exe without the /c, there is no difference.

Your code looks like it should work (with one caveat, see below).
I took your code verbatim and replaced test.ext with sort, which can read from piped stdin.
If I run the code as-is, it starts the sort command, which waits for input. It hangs at child.waitFor() because you don't close the output stream to indicate EOF. When I add the close() call, everything works correctly.
I suggest you look at test.exe and determine if it is capable of reading from piped stdin, or is expecting console input.

Get rid of "cmd" and "/c". At present you are feeding output to cmd.exe, not to test.exe.

How do I run multiple commands in SSH through Java?

How do I run multiple commands in SSH using Java runtime?
the command: ssh user#127.0.0.1 'export MYVAR=this/dir/is/cool; /run/my/script
/myscript; echo $MYVAR'
#Test
public void testSSHcmd() throws Exception
{
StringBuilder cmd = new StringBuilder();
cmd.append("ssh ");
cmd.append("user#127.0.0.1 ");
cmd.append("'export ");
cmd.append("MYVAR=this/dir/is/cool; ");
cmd.append("/run/my/script/myScript; ");
cmd.append("echo $MYVAR'");
Process p = Runtime.getRuntime().exec(cmd.toString());
}
The command by its self will work but when trying to execute from java run-time it does not. Any suggestions or advice?

Use the newer ProcessBuilder class instead of Runtime.exec. You can construct one by specifying the program and its list of arguments as shown in my code below. You don't need to use single-quotes around the command. You should also read the stdout and stderr streams and waitFor for the process to finish.
ProcessBuilder pb = new ProcessBuilder("ssh",
"user#127.0.0.1",
"export MYVAR=this/dir/is/cool; /run/my/script/myScript; echo $MYVAR");
pb.redirectErrorStream(); //redirect stderr to stdout
Process process = pb.start();
InputStream inputStream = process.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line = null;
while((line = reader.readLine())!= null) {
System.out.println(line);
}
process.waitFor();

If the Process just hangs I suspect that /run/my/script/myScript outputs something to stderr. You need to handle that output aswell as stdout:
public static void main(String[] args) throws Exception {
String[] cmd = {"ssh", "root#localhost", "'ls asd; ls'" };
final Process p = Runtime.getRuntime().exec(cmd);
// ignore all errors (print to std err)
new Thread() {
#Override
public void run() {
try {
BufferedReader err = new BufferedReader(
new InputStreamReader(p.getErrorStream()));
String in;
while((in = err.readLine()) != null)
System.err.println(in);
err.close();
} catch (IOException e) {}
}
}.start();
// handle std out
InputStreamReader isr = new InputStreamReader(p.getInputStream());
BufferedReader reader = new BufferedReader(isr);
StringBuilder ret = new StringBuilder();
char[] data = new char[1024];
int read;
while ((read = reader.read(data)) != -1)
ret.append(data, 0, read);
reader.close();
// wait for the exit code
int exitCode = p.waitFor();
}

The veriant of Runtime.exec you are calling splits the command string into several tokens which are then passed to ssh. What you need is one of the variants where you can provide a string array. Put the complete remote part into one argument while stripping the outer quotes. Example
Runtime.exec(new String[]{
"ssh",
"user#127.0.0.1",
"export MYVAR=this/dir/is/cool; /run/my/script/myScript; echo $MYVAR"
});
That's it.

You might want to take a look at the JSch library. It allows you to do all sorts of SSH things with remote hosts including executing commands and scripts.
They have examples listed here: http://www.jcraft.com/jsch/examples/

Here is the right way to do it:
Runtime rt=Runtime.getRuntime();
rt.exec("cmd.exe /c start <full path>");
For example:
Runtime rt=Runtime.getRuntime();
rt.exec("cmd.exe /c start C:/aa.txt");

If you are using SSHJ from https://github.com/shikhar/sshj/
public static void main(String[] args) throws IOException {
final SSHClient ssh = new SSHClient();
ssh.loadKnownHosts();
ssh.connect("10.x.x.x");
try {
//ssh.authPublickey(System.getProperty("root"));
ssh.authPassword("user", "xxxx");
final Session session = ssh.startSession();
try {
final Command cmd = session.exec("cd /backup; ls; ./backup.sh");
System.out.println(IOUtils.readFully(cmd.getInputStream()).toString());
cmd.join(5, TimeUnit.SECONDS);
System.out.println("\n** exit status: " + cmd.getExitStatus());
} finally {
session.close();
}
} finally {
ssh.disconnect();
}
}

Redirect process output to stdout

I would like to execute foo.bat from within a Groovy program and have the resulting process' output redirected to stdout. Either a Java or Groovy code example would be fine.
foo.bat can take several minutes to run and generates a lot of output, so I would like to see the output as soon as it is generated, rather than having to wait until the process has completed before seeing all the output at once.

It is simple to redirect all your stream to standard output using inheritIO() method. This will print the output to the stdout of the process from which you are running this command.
ProcessBuilder pb = new ProcessBuilder("command", "argument");
pb.directory(new File(<directory from where you want to run the command>));
pb.inheritIO();
Process p = pb.start();
p.waitFor();
There exist other methods too, like as mentioned below. These individual methods will help redirect only required stream.
pb.redirectInput(Redirect.INHERIT)
pb.redirectOutput(Redirect.INHERIT)
pb.redirectError(Redirect.INHERIT)

This uses a class which reads all output the executed program generates and displays it in it's own stdout.
class StreamGobbler extends Thread {
InputStream is;
// reads everything from is until empty.
StreamGobbler(InputStream is) {
this.is = is;
}
public void run() {
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
System.out.println(line);
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("javac");
//output both stdout and stderr data from proc to stdout of this process
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream());
StreamGobbler outputGobbler = new StreamGobbler(proc.getInputStream());
errorGobbler.start();
outputGobbler.start();
proc.waitFor();

If you're looking to do this with more Groovy and less java, this will print each line as it happens:
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.in.eachLine { line -> println line }
Alternatively, if you want to see both stdout and stderr
def cmd = "./longRunningProcess"
def process = cmd.execute()
process.waitForProcessOutput( System.out, System.err )

Here's something a little simpler if you're just trying to grab the output of a simple command. You'll need to use threads like jitter does if you want to process in parallel or if your command takes stdin or generates stderr.
Use a buffered copy (like this) if you're getting lots of output.
import java.io.*;
public class test {
static void copy(InputStream in, OutputStream out) throws IOException {
while (true) {
int c = in.read();
if (c == -1) break;
out.write((char)c);
}
}
public static void main(String[] args) throws IOException, InterruptedException {
String cmd = "echo foo";
Process p = Runtime.getRuntime().exec(cmd);
copy(p.getInputStream(), System.out);
p.waitFor();
}
}

The following Groovy code will execute foo.bat and send the output to stdout:
println "foo.bat".execute().text

Asynchronous way to achieve it.
void inputStreamToOutputStream(final InputStream inputStream, final OutputStream out) {
Thread t = new Thread(new Runnable() {
public void run() {
try {
int d;
while ((d = inputStream.read()) != -1) {
out.write(d);
}
} catch (IOException ex) {
//TODO make a callback on exception.
}
}
});
t.setDaemon(true);
t.start();
}
{
Process p = ...;
inputStreamToOutputStream(p.getErrorStream(), System.out);
inputStreamToOutputStream(p.getInputStream(), System.out);
}

VerboseProcess from jcabi-log can help you:
String output = new VerboseProcess(new ProcessBuilder("foo.bat")).stdout();