We Keep Coding

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Executor framework future.get() hangs

I have the following code that runs a process and I want to know if there was an exception while running it. It hangs for no reason
Runtime runtime = Runtime.getRuntime();
proc = runtime.exec(command.toString());
ProcessHandler errorStream = new ProcessHandler(proc.getErrorStream(),"ERROR", rdyFilePath);
ExecutorService pool = Executors.newSingleThreadExecutor();
Future future = pool.submit(errorStream);
pool.shutdown();
try {
if(future.get() == null) {
log.info("Done completing error thread");
}
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
Here is the class for process handler
public class ProcessHandler implements Callable<Integer> {
private static Logger log = Logger.getLogger(ProcessHandler.class.getName());
InputStream inpStr;
String strType;
String rdyFile;
public ProcessHandler(InputStream inpStr, String strType, String rdyFile) {
this.inpStr = inpStr;
this.strType = strType;
this.rdyFile = rdyFile;
}
public Integer call() throws FileMetadataException {
StringBuffer sb = new StringBuffer();
try {
InputStreamReader inpStrd = new InputStreamReader(inpStr);
BufferedReader buffRd = new BufferedReader(inpStrd);
String line = null;
while((line = buffRd.readLine()) != null) {
if("ERROR".equalsIgnoreCase(strType)) {
sb.append(line + "\n");
log.info(strType + "->" + line);
}
}
if(sb != null) {
log.info("Error Stream length : " + sb.length());
throw new RuntimeException();
}
buffRd.close();
} catch(IOException e) {
log.error("IOException in ProcessHandler Thread" + e.fillInStackTrace());
System.err.println(e);
throw new FileMetadataException();
} finally {
if(sb != null) {
if(sb.toString().length() > 0 ) {
log.error("Error string buffer length " + sb.length());
// do not create rdy file
} else {
log.error("Error string buffer length : " + sb.length());
File f = new File(rdyFile);
try {
f.createNewFile();
} catch(IOException e) {
log.error("IOException while creating rdy file");
}
}
// create rdy file.
}
}
return sb.length();
}
}
}

I have the following code that runs a process and I want to know if there was an exception while running it. It hangs for no reason
If there was an exception your future.get() should have thrown with an ExecutionException -- it would not "hang". Are you sure that your exception is not being printed but somehow lost in the logs or console output?
In tracing your code, I see no way for your program to not finish after it finishes reading the stream. Maybe the process whose error-stream you are reading from is still running and the InputStream has not been closed yet? Maybe there is so much output that you are filling up core with the StringBuffer (which should be changed to a StringBuilder btw).
Can you attach to your application with jconsole to see if the thread is still running and if it is, what is it doing?
if(future.get() == null) {
log.info("Done completing error thread");
}
So this will only log output if you return null from your call() method. That will never happen since the only return is return sb.length();. So either you will get an exception from your call() or your result will be a non-null Integer.

From the Process javadoc:
Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock.
Per this warning, your future.get() method is hanging because you are only consuming the error stream from the process object. On all platforms I have ever utilized Process objects, I have observed that you need to consume both the Error and Standard out streams.
This is a good tool that uses a multithreaded approach to ensure that streams of a process are consumed.
In this case, since it appears that you don't care about the standard output of the process you might add something like this, borrowing the StreamHelper class:
StreamHelper inStreamHelper = new StreamHelper(proc.getInputStream());
inStreamHelper.start();

Heap size issue - Memory management using java

I have the following code in my application which does two things:
Parse the file which has 'n' number of data.
For each data in the file, there will be two web service calls.
public static List<String> parseFile(String fileName) {
List<String> idList = new ArrayList<String>();
try {
BufferedReader cfgFile = new BufferedReader(new FileReader(new File(fileName)));
String line = null;
cfgFile.readLine();
while ((line = cfgFile.readLine()) != null) {
if (!line.trim().equals("")) {
String [] fields = line.split("\\|");
idList.add(fields[0]);
}
}
cfgFile.close();
} catch (IOException e) {
System.out.println(e+" Unexpected File IO Error.");
}
return idList;
}
When i try parse the file having 1 million lines of record, the java process fails after processing certain amount of data. I got java.lang.OutOfMemoryError: Java heap space error. I can partly figure out that the java process stops because of this huge data being provided. Kindly suggest me how to proceed with this huge data.
EDIT: Will this part of code new BufferedReader(new FileReader(new File(fileName))); parse the whole file and gets affected to the size of the file.

The problem you have is you are accumulating all the data on the list. The best way to approach this is to do it on a streaming fashion. This means do not accumulate all the ids on the list, but call your web service on each row or accumulate a smaller buffer and then do the call.
Opening the file and creating the BufferedReader will have no impact on memory consumption, as the bytes from the file will be read (more or less) line by line. The problem is at this point in the code idList.add(fields[0]);, the list will grow as large as the file as you keep accumulating all of the file data into it.
Your code should do something like this:
while ((line = cfgFile.readLine()) != null) {
if (!line.trim().equals("")) {
String [] fields = line.split("\\|");
callToRemoteWebService(fields[0]);
}
}

Increase your java heap memory size using the -Xms and -Xmx options. If not set explicitly, the jvm sets the heap size to the ergonomic defaults which in your case is not enough. Read this paper to find out more about tuning the memory in jvm: http://www.oracle.com/technetwork/java/javase/tech/memorymanagement-whitepaper-1-150020.pdf
EDIT: Alternative way on doing this in a producer-consumer way to exploit parallel processing. The general idea is to create a producer thread that reads the file and queues tasks for processing and n consumer threads that consume them. A very general idea (for illustrative purposes) is the following:
// blocking queue holding the tasks to be executed
final SynchronousQueue<Callable<String[]> queue = // ...
// reads the file and submit tasks for processing
final Runnable producer = new Runnable() {
public void run() {
BufferedReader in = null;
try {
in = new BufferedReader(new FileReader(new File(fileName)));
String line = null;
while ((line = file.readLine()) != null) {
if (!line.trim().equals("")) {
String[] fields = line.split("\\|");
// this will block if there are not available consumer threads to process it...
queue.put(new Callable<Void>() {
public Void call() {
process(fields);
}
});
}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt());
} finally {
// close the buffered reader here...
}
}
}
// Consumes the tasks submitted from the producer. Consumers can be pooled
// for parallel processing.
final Runnable consumer = new Runnable() {
public void run() {
try {
while (true) {
// this method blocks if there are no items left for processing in the queue...
Callable<Void> task = queue.take();
taks.call();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
Of course you have to write code that manages the lifecycle of the consumer and producer threads. The right way to do this would be by implementing it using an Executor.

When you want to work with big data, you have 2 choices:
use a big enough heap to fit all the data. this will "work" for a while, but if your data size is unbounded, it will eventually fail.
work with the data incrementally. only keep part of the data (of a bounded size) in memory at any one time. this is the ideal solution as it will scale to any amount of data.

Implementing the "system" command in Java

I have need for a "system" function call, the same as those in Python, Perl, PHP, Ruby, &c. It will be a component of a JavaScript standard library called Narwhal, when it's run on the Rhino JavaScript engine, which is in turn run on Java.
The trouble is that Java's standard library appears to have abstracted away the ability to spawn a subprocess that shares the parent process's stdio. This means that you can't defer interactivity to the subprocess.
My first crack at this was to implement Python's subprocess.popen. This uses three "pumper" threads to actively copy the parent process's stdio independently (to prevent deadlock). Unfortunately this is giving us two problems. First, the input does not close automatically when the sub-process voluntarily exits. Second, the streams to the child process do not buffer and flush properly.
I'm looking for solutions that would make our require("os").system() command work as one would expect.
The project is at http://narwhaljs.org
Relevant code:
http://github.com/tlrobinson/narwhal/blob/d147c160f11fdfb7f3c0763acf352b2b0e2713f7/lib/os.js#L10
http://github.com/tlrobinson/narwhal/blob/d147c160f11fdfb7f3c0763acf352b2b0e2713f7/engines/rhino/lib/os-engine.js#L37

Not sure if this is what you're looking for, but you can invoke the C system function through the JNA library:
public class System {
public interface C extends Library {
C INSTANCE = (C) Native.loadLibrary(
(Platform.isWindows() ? "msvcrt" : "c"), C.class);
public int system(String format);
}
public static void main(String[] args) {
C.INSTANCE.system("vi");
}
}
Cursory testing worked on Windows, anyhow.

If I'm understanding you properly, you want something like that:
import java.util.*;
import java.io.*;
class StreamGobbler extends Thread
{
InputStream is;
String type;
OutputStream os;
StreamGobbler(InputStream is, String type)
{
this(is, type, null);
}
StreamGobbler(InputStream is, String type, OutputStream redirect)
{
this.is = is;
this.type = type;
this.os = redirect;
}
public void run()
{
try
{
PrintWriter pw = null;
if (os != null)
pw = new PrintWriter(os);
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line=null;
while ( (line = br.readLine()) != null)
{
if (pw != null)
pw.println(line);
System.out.println(type + ">" + line);
}
if (pw != null)
pw.flush();
} catch (IOException ioe)
{
ioe.printStackTrace();
}
}
}
public class GoodWinRedirect
{
public static void main(String args[])
{
if (args.length < 1)
{
System.out.println("USAGE java GoodWinRedirect <outputfile>");
System.exit(1);
}
try
{
FileOutputStream fos = new FileOutputStream(args[0]);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("java jecho 'Hello World'");
// any error message?
StreamGobbler errorGobbler = new
StreamGobbler(proc.getErrorStream(), "ERROR");
// any output?
StreamGobbler outputGobbler = new
StreamGobbler(proc.getInputStream(), "OUTPUT", fos);
// kick them off
errorGobbler.start();
outputGobbler.start();
// any error???
int exitVal = proc.waitFor();
System.out.println("ExitValue: " + exitVal);
fos.flush();
fos.close();
} catch (Throwable t)
{
t.printStackTrace();
}
}
}
I found this piece of code in: JavaWorld sometime ago when I was looking for a similar solution to wrap system calls to some exe files.
My code has evolved a bit since then, but I think is a good example.

You need to monitor the process exit-status separately, either by polling the exit code or by having a separate thread waiting in the Process.waitFor() method.
On the matter of buffering and flushing the streams, I don't thing there is an easy solution. There are several Java-classes which do buffering in different forms (BufferedInputStream, etc). Maybe one of them can help?

It's really important to consume the process standard output and error concurrently. See Carlos Tasada's example code elsewhere here.
If you don't do this your code may (or may not) work depending on the output from your spawned process. As and when that output changes (if your spawned process encounters an error, say) then without the concurrent consumption your process will deadlock. Most of the issues I see on SO related to Process.exec() are blocking related.

Run external program from Java, read output, allow interruption

I want to launch a process from Java, read its output, and get its return code. But while it's executing, I want to be able to cancel it. I start out by launching the process:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
If I call proc.waitFor(), I can't do anything until the process exits. So I'm assuming I need to something like this:
while (true) {
see if process has exited
capture some output from the process
decide if I want to cancel it, and if so, cancel it
sleep for a while
}
Is this right? Can someone give me an example of how to do this in Java?

Here's an example of what I think you want to do:
ProcessBuilder pb = new ProcessBuilder(args);
pb.redirectErrorStream(true);
Process proc = pb.start();
InputStream is = proc.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
int exit = -1;
while ((line = br.readLine()) != null) {
// Outputs your process execution
System.out.println(line);
try {
exit = proc.exitValue();
if (exit == 0) {
// Process finished
}
} catch (IllegalThreadStateException t) {
// The process has not yet finished.
// Should we stop it?
if (processMustStop())
// processMustStop can return true
// after time out, for example.
proc.destroy();
}
}
You can improve it :-) I don't have a real environment to test it now, but you can find some more information here.

I recommend checking out Apache Commons Exec to avoid recreating the wheel. It has some nice features like choosing between synchronous vs. asynchronous execution, as well as a standard solution to spawning a watchdog process that can help in timing out the execution in case it gets stuck.

A helper class like this would do the trick:
public class ProcessWatcher implements Runnable {
private Process p;
private volatile boolean finished = false;
public ProcessWatcher(Process p) {
this.p = p;
new Thread(this).start();
}
public boolean isFinished() {
return finished;
}
public void run() {
try {
p.waitFor();
} catch (Exception e) {}
finished = true;
}
}
You would then implement your loop exactly as you describe:
Process p = Runtime.getRuntime().exec("whatever command");
ProcessWatcher pw = new ProcessWatcher(p);
InputStream output = p.getInputStream();
while(!pw.isFinished()) {
processOutput(output);
if(shouldCancel()) p.destroy();
Thread.sleep(500);
}
Depending upon what conditions would make you want to destroy the process, you might want to do that in a separate thread. Otherwise, you may block while waiting for more program output to process and never really get the option to destroy it.
EDIT: McDowell is 100% right in his comment below, so I've made the finished variable volatile.

How about this (see how it works in jcabi-heroku-maven-plugin):
/**
* Wait for the process to stop, logging its output in parallel.
* #param process The process to wait for
* #return Stdout produced by the process
* #throws InterruptedException If interrupted in between
*/
private String waitFor(final Process process) throws InterruptedException {
final BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream())
);
final CountDownLatch done = new CountDownLatch(1);
final StringBuffer stdout = new StringBuffer();
new Thread(
new VerboseRunnable(
new Callable<Void>() {
#Override
public Void call() throws Exception {
while (true) {
final String line = reader.readLine();
if (line == null) {
break;
}
System.out.println(">> " + line);
stdout.append(line);
}
done.countDown();
return null;
}
},
false
)
).start();
try {
process.waitFor();
} finally {
done.await();
IOUtils.closeQuietly(reader);
}
return stdout.toString();
}
ps. Now this implementation is available as com.jcabi.log.VerboseProcess class from jcabi-log artifact.

What would make you decide to kill the process -- an asynchronous event (such as input from the user), or a synchronous event (e.g., the process has done what you wanted it to do)? I'm guessing it's the former -- input from the user makes you decide to cancel the subprocess.
Also, how much output do you expect the subprocess to produce? If it's a lot, then the subprocess may block if you don't read from its output stream quickly enough.
Your situation may vary, but it seems that you're likely going to need at least two different threads -- one to decide whether to cancel the process, and one that handles the output from the subprocess.
Have a look here for a bit more detail: http://java.sun.com/developer/JDCTechTips/2005/tt0727.html#2