We Keep Coding

Java - Use Input and OutputStream of ProcessBuilder continuously

I want to use an external tool while extracting some data (loop through lines).
For that I first used Runtime.getRuntime().exec() to execute it.
But then my extraction got really slow. So I am searching for a possibility to exec the external tool in each instance of the loop, using the same instance of shell.
I found out, that I should use ProcessBuilder. But it's not working yet.
Here is my code to test the execution (with input from the answers here in the forum already):
public class ExecuteShell {
ProcessBuilder builder;
Process process = null;
BufferedWriter process_stdin;
BufferedReader reader, errReader;
public ExecuteShell() {
String command;
command = getShellCommandForOperatingSystem();
if(command.equals("")) {
return; //Fehler! No error handling yet
}
//init shell
builder = new ProcessBuilder( command);
builder.redirectErrorStream(true);
try {
process = builder.start();
} catch (IOException e) {
System.out.println(e);
}
//get stdout of shell
reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
errReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
//get stdin of shell
process_stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
System.out.println("ExecuteShell: Constructor successfully finished");
}
public String executeCommand(String commands) {
StringBuffer output;
String line;
try {
//single execution
process_stdin.write(commands);
process_stdin.newLine();
process_stdin.flush();
} catch (IOException e) {
System.out.println(e);
}
output = new StringBuffer();
line = "";
try {
if (!reader.ready()) {
output.append("Reader empty \n");
return output.toString();
}
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
return output.toString();
}
if (!reader.ready()) {
output.append("errReader empty \n");
return output.toString();
}
while ((line = errReader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
System.out.println("ExecuteShell: error in executeShell2File");
e.printStackTrace();
return "";
}
return output.toString();
}
public int close() {
// finally close the shell by execution exit command
try {
process_stdin.write("exit");
process_stdin.newLine();
process_stdin.flush();
}
catch (IOException e) {
System.out.println(e);
return 1;
}
return 0;
}
private static String getShellCommandForOperatingSystem() {
Properties prop = System.getProperties( );
String os = prop.getProperty( "os.name" );
if ( os.startsWith("Windows") ) {
//System.out.println("WINDOWS!");
return "C:/cygwin64/bin/bash";
} else if (os.startsWith("Linux") ) {
//System.out.println("Linux!");
return"/bin/sh";
}
return "";
}
}
I want to call it in another Class like this Testclass:
public class TestExec{
public static void main(String[] args) {
String result = "";
ExecuteShell es = new ExecuteShell();
for (int i=0; i<5; i++) {
// do something
result = es.executeCommand("date"); //execute some command
System.out.println("result:\n" + result); //do something with result
// do something
}
es.close();
}
}
My Problem is, that the output stream is always empty:
ExecuteShell: Constructor successfully finished
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
result:
Reader empty
I read the thread here: Java Process with Input/Output Stream
But the code snippets were not enough to get me going, I am missing something. I have not really worked with different threads much. And I am not sure if/how a Scanner is of any help to me. I would really appreciate some help.
Ultimatively, my goal is to call an external command repeatetly and make it fast.
EDIT:
I changed the loop, so that the es.close() is outside. And I wanted to add, that I do not want only this inside the loop.
EDIT:
The problem with the time was, that the command I called caused an error. When the command does not cause an error, the time is acceptable.
Thank you for your answers

You are probably experiencing a race condition: after writing the command to the shell, your Java program continues to run, and almost immediately calls reader.ready(). The command you wanted to execute has probably not yet output anything, so the reader has no data available. An alternative explanation would be that the command does not write anything to stdout, but only to stderr (or the shell, maybe it has failed to start the command?). You are however not reading from stderr in practice.
To properly handle output and error streams, you cannot check reader.ready() but need to call readLine() (which waits until data is available) in a loop. With your code, even if the program would come to that point, you would read only exactly one line from the output. If the program would output more than one line, this data would get interpreted as the output of the next command. The typical solution is to read in a loop until readLine() returns null, but this does not work here because this would mean your program would wait in this loop until the shell terminates (which would never happen, so it would just hang infinitely).
Fixing this would be pretty much impossible, if you do not know exactly how many lines each command will write to stdout and stderr.
However, your complicated approach of using a shell and sending commands to it is probably completely unnecessary. Starting a command from within your Java program and from within the shell is equally fast, and much easier to write. Similarly, there is no performance difference between Runtime.exec() and ProcessBuilder (the former just calls the latter), you only need ProcessBuilder if you need its advanced features.
If you are experiencing performance problems when calling external programs, you should find out where they are exactly and try to solve them, but not with this approach. For example, normally one starts a thread for reading from both the output and the error stream (if you do not start separate threads and the command produces large output, everything might hang). This could be slow, so you could use a thread pool to avoid repeated spawning of processes.

Opening exe with parameters in Java, don't understand parameters

I'm aware how to open an exe program with parameters in Java from finding the answer online. However my specific shortcut is a bit complicated for me to understand.
I'm trying to open a shortcut which has the following target:
C:\Windows\System32\javaw.exe -cp jts.jar;total.2012.jar -Dsun.java2d.noddraw=true -Dswing.boldMetal=false -Dsun.locale.formatasdefault=true -Xmx768M -XX:MaxPermSize=128M jclient/LoginFrame C:\Jts
In my program I've split up the location and what I think are the parameters. However when I run the program I get the error 'Could not create Java Virtual Machine, Program will Exit'. Can someone with a better understanding of whats going on explain what I might be doing wrong or point me in a direction where I can read up?
String location = "C:\\Windows\\System32\\javaw.exe";
String p1="-cp jts.jar;total.2012.jar";
String p2="-Dsun.java2d.noddraw=true";
String p3="-Dswing.boldMetal=false";
String p4="-Dsun.locale.formatasdefault=true";
String p5="-Xmx768M";
String p6="-XX:MaxPermSize=128M";
String p7="jclient/LoginFrame" ;
String p8 = "C:\\Jts";
try {
Process p = new ProcessBuilder(location,p1,p2,p3,p4,p5,p6,p7,p8).start();
} catch (IOException ex) {
Logger.getLogger(Openprogramtest.class.getName()).log(Level.SEVERE, null, ex);
}

Each String you pass to ProcessBuilder is a separate argument (except the first one, which is the command).
Think of it like the args[] which are passed to your main method. Each String would be a separate element in the array.
I suspect that p1 is been interpreted as a single argument, when it should actually be two...
Try separating this argument into two separate parameters
String location = "C:\\Windows\\System32\\javaw.exe";
String p1="-cp";
String p2="jts.jar;total.2012.jar";
String p3="-Dsun.java2d.noddraw=true";
String p4="-Dswing.boldMetal=false";
String p5="-Dsun.locale.formatasdefault=true";
String p6="-Xmx768M";
String p7="-XX:MaxPermSize=128M";
String p8="jclient/LoginFrame" ;
String p9 = "C:\\Jts";
Amendment
Look at the -cp parameter, it appears that the class path elements are relative to the location that the command is executed. This suggests that you need to use the ProcessBuilder#directory(File) to specify the location that the command should executed from.
For example, if you program is installed in C:\Program Files\MyAwesomeApp, but you run it from the context of C:\Desktop, then Java won't be able to find the Jar files it needs, generally raising a ClassNotFound exception.
Instead, you need to tell ProcessBuilder that you want the command to executed from within the C:\Program Files\MyAwesomeApp context.
For example...
ProcessBuilder pb = new ProcessBuilder(...);
pb.directory(new File("C:\Program Files\MyAwesomeApp"));
// Other settings...
Process p = pb.start();
Updated from running example
Just to make the point. I built myself a little Java program that simple printed a simple message to the standard out.
When I run this, it works as expected...
try {
String params[] = new String[]{
"C:\\Windows\\System32\\javaw.exe",
"-cp",
"C:\\...\\TestSimpleProcessBuilder\\build\\classes",
"-Dsun.java2d.noddraw=true",
"-Dswing.boldMetal=false",
"-Dsun.locale.formatasdefault=true",
"-Xmx768M",
"-XX:MaxPermSize=128M",
"testsimpleprocessbuilder/HelloWorld",
"Boo"
};
ProcessBuilder pb = new ProcessBuilder(params);
pb.redirectErrorStream();
Process p = pb.start();
InputStream is = p.getInputStream();
int in = -1;
while ((in = is.read()) != -1) {
System.out.print((char) in);
}
is = p.getErrorStream();
in = -1;
while ((in = is.read()) != -1) {
System.out.print((char) in);
}
System.out.println("p exited with " + p.exitValue());
} catch (IOException ex) {
Logger.getLogger(TestSimpleProcessBuilder.class.getName()).log(Level.SEVERE, null, ex);
}
When I change the arguments from
"-cp",
"C:\\...\\TestSimpleProcessBuilder\\build\\classes",
to
"-cp C:\\...\\TestSimpleProcessBuilder\\build\\classes",
It fails with...
And outputs
Unrecognized option: -cp
C:\DevWork\personal\java\projects\wip\StackOverflow\TestSimpleProcessBuilder\build\classes
And if you're wondering, this is the little test program I wrote that gets run...
package testsimpleprocessbuilder;
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello world - world says " + (args.length > 0 ? args[0] : "Nothing"));
}
}

Java Process.waitFor() and Readline hangs

First, this is my code :
import java.io.*;
import java.util.Date;
import com.banctecmtl.ca.vlp.shared.exceptions.*;
public class PowershellTest implements Runnable {
public static final String PATH_TO_SCRIPT = "C:\\Scripts\\ScriptTest.ps1";
public static final String SERVER_IP = "XX.XX.XX.XXX";
public static final String MACHINE_TO_MOD = "MachineTest";
/**
* #param args
* #throws OperationException
*/
public static void main(String[] args) throws OperationException {
new PowershellTest().run();
}
public PowershellTest(){}
#Override
public synchronized void run() {
String input = "";
String error = "";
boolean isHanging = false;
try {
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec("powershell -file " + PATH_TO_SCRIPT +" "+ SERVER_IP +" "+ MACHINE_TO_MOD);
proc.getOutputStream().close();
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
proc.waitFor();
String line;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
input += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
inputstream = proc.getErrorStream();
inputstreamreader = new InputStreamReader(inputstream);
bufferedreader = new BufferedReader(inputstreamreader);
isHanging = false;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
error += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
proc.destroy();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
} catch (InterruptedException e) {
//throw new OperationException("Script thread problem.",e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
I'm currently trying to run a powershell script that will start/stop a vm (VMWARE) on a remote server. The script work from command line and so does this code. The thing is, I hate how I have to use a thread (and make it wait for the script to respond, as explained further) for such a job. I had to do it because both BufferedReader.readline() and proc.waitFor() hang forever.
The script, when ran from cmd, is long to execute. it stall for 30 sec to 1 min from validating authentification with the server and executing the actual script. From what I saw from debugging, the readline hang when it start receiving those delays from the script.
I'm also pretty sure it's not a memory problem since I never had any OOM error in any debugging session.
Now I understand that Process.waitFor() requires me to flush the buffer from both the error stream and the regular stream to work and so that's mainly why I don't use it (I need the output to manage VM specific errors, certificates issues, etc.).
I would like to know if someone could explain to me why it hangs and if there is a way to just use the typical readline() without having it to hang so hard. Even if the script should have ended since a while, it still hang (I tried to run both the java application and a cmd command using the exact same thing I use in the java application at the same time, left it runingfor 1h, nothing worked). It is not just stuck in the while loop, the readline() is where the hanging is.
Also this is a test version, nowhere close to the final code, so please spare me the : this should be a constant, this is useless, etc. I will clean the code later. Also the IP is not XX.XX.XX.XXX in my code, obviously.
Either explanation or suggestion on how to fix would be greatly appreciated.
Ho btw here is the script I currently use :
Add-PSSnapin vmware.vimautomation.core
Connect-VIServer -server $args[0]
Start-VM -VM "MachineTest"
If you need more details I will try to give as much as I can.
Thanks in advance for your help!
EDIT : I also previously tested the code with a less demanding script, which job was to get the content of a file and print it. Since no waiting was needed to get the information, the readline() worked well. I'm thus fairly certain that the problem reside on the wait time coming from the script execution.
Also, forgive my errors, English is not my main language.
Thanks in advance for your help!
EDIT2 : Since I cannot answer to my own Question :
Here is my "final" code, after using threads :
import java.io.*;
public class PowershellTest implements Runnable {
public InputStream is;
public PowershellTest(InputStream newIs){
this.is = newIs;
}
#Override
public synchronized void run() {
String input = "";
String error = "";
try {
InputStreamReader inputstreamreader = new InputStreamReader(is);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String line;
while ((line = bufferedreader.readLine()) != null) {
input += (line + "\n");
}
is.close();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
And the main simply create and start 2 thread (PowerShellTest instances), 1 with the errorStream and 1 with the inputStream.
I believe I made a dumb error when I first coded the app and fixed it somehow as I reworked the code over and over. It still take a good 5-6 mins to run, which is somehow similar if not longer than my previous code (which is logical since the errorStream and inputStream get their information sequentially in my case).
Anyway, thanks to all your answer and especially Miserable Variable for his hint on threading.

First, don't call waitFor() until after you've finished reading the streams. I would highly recommend you look at ProcessBuilder instead of simply using Runtime.exec, and split the command up yourself rather than relying on Java to do it for you:
ProcessBuilder pb = new ProcessBuilder("powershell", "-file", PATH_TO_SCRIPT,
SERVER_IP, MACHINE_TO_MOD);
pb.redirectErrorStream(true); // merge stdout and stderr
Process proc = pb.start();
redirectErrorStream merges the error output into the normal output, so you only have to read proc.getInputStream(). You should then be able to just read that stream until EOF, then call proc.waitFor().

You are currently waiting to complete reading from inputStream before starting to read from errorStream. If the process writes to its stderr before stdout maybe you are getting into a deadlock situation.
Try reading from both streams from concurrently running threads. While you are at it, also remove proc.getOutputStream().close();. It shouldn't affect the behavior, but it is not required either.

Runtime.exec on argument containing multiple spaces

How can I make the following run?
public class ExecTest {
public static void main(String[] args) {
try {
//Notice the multiple spaces in the argument
String[] cmd = {"explorer.exe", "/select,\"C:\\New Folder\\file.txt\""};
//btw this works
//String cmd = "explorer.exe /select,\"C:\\New Folder\\file.txt\"";
//and surprisingly this doesn't work
//String[] cmd = {"explorer.exe", "/select,\"C:\\New Folder\\file.txt\""};
//Update: and (as crazy as it seems) the following also worked
//String[] cmd = {"explorer.exe", "/select,\"C:\\New", "Folder\\file.txt\""};
Runtime.getRuntime().exec(cmd);
} catch (Exception e) {
e.printStackTrace();
}
}
}
Using Java 6. Tested under Vista x64. By the way, taking the string that gets executed (you'll have to use the String version of exec to get it) and using it in the Search field of Vista's start menu will run as expected.

Ok, this is not simply an update but also an answer so I'm filing it as one. According to all information I could find, the following should theoretically do it:
String[] cmd = {"explorer.exe", "/select,\"C:\New", "", "", "", "", "", "", "Folder\file.txt\""};
The multiple spaces have been broken into empty strings and the array version of exec is used.
Using the above array, I debugged the loop in lines 50-75 of java.lang.ProcessImpl where a string is finally constructed. The resulting string was:
explorer.exe /select,"C:\New Folder\file.txt"
This is what is passed as the 1st argument to ProcessImpl's native create method (line 118 same class), which as it seems fails to run properly this command.
So I guess it all ends here... sadly.
Thnx prunge for pointing out the java bug.
Thnx everyone for their time and interest!

A miracle, it works!
Don't ask me why, but when i, after quite a while of nerve-wrecking research in the internets, was close to give up and use a temporary batch file as a workaround, i forgot to add the /select, parameter to the command, and, who would have thought, the following works on my Win 7 32Bit System.
String param = "\"C:\\Users\\ME\\AppData\\Local\\Microsoft\\Windows\\Temporary Internet Files\\\"";
try {
String[]commands = new String[]{"explorer.exe", param};
Process child = Runtime.getRuntime().exec(commands);
} catch (IOException e1) {
System.out.println("...");
}
General Solution:
The solution of the bug-database mentioned by prunge in his post (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6511002) worked fine for me.
Reason:
Apparently the problem lies with the commenting of some characters done by java which it does before actually executing the command string.
You have to do the commenting yourself by tokenizing your command string, to prevent the faulty java one to spring into action and mess everything up.
How to fix:
So, in my case i had to do the following (tokenizing my command string, so that no spaces are left inside the string):
String param[] = {
"explorer.exe",
"/select,C:\\Users\\ME\\AppData\\Local\\Microsoft\\Windows\\Temporary",
"Internet",
"Files\\"};
try {
Process child = Runtime.getRuntime().exec(param);
} catch (IOException e1) {
System.out.println("...");
}
As you can see i basically started a new String wherever a space occured, so "Temporary Internet Files" became "Temporary","Internet","Files".

Always use Runtime.exec(String[]), not Runtime.exec(String) unless the command line is extremely simple.

Use new File(pathName).canExecute() first to check whether it's executable or not
EDIT:
public static void runAll(String... cmd)
{
for(String s : cmd)
{
try
{
Runtime.getRuntime().exec(cmd);
}
catch(Exception e)
{
e.printStackTrace();
}
}
}
and then you can use it like: runAll("explorer.exe", "taskmgr.exe");

The characters ,-& and double spaces, all combined are a nightmare!
All the answers exposed here failed for "\\NAS\media\Music\Artistes\E\Earth, Wind & Fire\1992 - The eternal dance - Vol. 1 (1971-1975) (double space between 'Vol. 1' and '(1971').
I have no other choice than writing a temporary batch file:
void openFolderOf( Album album ) {
try {
final String path = album._playList.getParent();
final File batch = File.createTempFile( getClass().getSimpleName(), ".bat" );
try( PrintStream ps = new PrintStream( batch )) {
ps.println( "explorer.exe \"" + path + '"' );
}
Runtime.getRuntime().exec( batch.getAbsolutePath());
}
catch( final Throwable t ) {
t.printStackTrace();
}
}
Note: on cmd.exe, the line explorer "\\NAS..." works well but not with Runtime.exec() nor ProcessBuilder.

Could be a Java bug. See:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6511002
Did a bit of debugging out of curiosity, I think things are becoming unstuck in java.lang.ProcessImpl (see the constructor). Noticed that when it got to actually calling the underlying Windows API the string had turned into
explorer.exe "/select,"c:\New Folder\test.txt""
So that might explain why, as for workarounds see the bug database link.

For your specific case of needing the reveal/select command, I get around the windows quote nightmare by using cmd /c start:
String[] cmd = {"cmd", "/c", "start explorer.exe /select," + path};
Where path is the absolute path from a File object.

A better way to do it would be using ProcessBuilder object:
Process p;
p = new ProcessBuilder("/Applications/Sublime Text.app/Contents/MacOS/sublime_text", homeDir + _CURL_POST_PUT_CMDS).start();
int exitValue = p.waitFor();
if (exitValue != 0){
System.out.println("Error to open " + homeDir + _CURL_POST_PUT_CMDS);
}

Simple way to resolve this problem for files is java.awt.Desktop Since 1.6
Example:
Desktop.getDesktop().open(new File(fullFileName));

How to run command line within java using thread async

I am new to java. I am tasked to write java program to run the command lines. I tested the command line under the DOS prompt since i do not have have access to Linux box yet. it worked fine. See the PROGRAM below for full command line syntax. the job will take 6 input files and generate some output files. Next i tried to create a class to and using getruntime and process to process this job. Even it compiled without error but when i run it just show the cursor blinking... i thought i need to use Thread async technique. please provide some advices since i do not have enough time for the projects. I also would like to implement a call back or return values when the job is done. an example would be greatly appreciated. Thanks
import java.io.*;
public class RunJob {
// public static final String PROGRAM = "c:\\wrk\\java.exe Hello";
//one command line below
public static final String PROGRAM = "c:/java.exe -cp \"wrk/jmp.jar;wrk/colt.jar\" gov.lanl.yadas.reliability.UltimateMissileReliabilityModel 10000 \"wrk/\" x1.dat x2c.dat x3.dat x4.dat x5.dat x6.dat true";
// Set to true to end the loop
static boolean done = false;
public static void main(String argv[]) throws IOException {
BufferedReader is;
String line;
String returnMsg = "Start ";
final Process p = Runtime.getRuntime().exec(PROGRAM);
System.out.println("start");
Thread waiter = new Thread() {
public void run() {
try {
p.waitFor();
} catch (InterruptedException ex) {
System.out.println("InterruptedException");
return;
}
System.out.println("Program terminated!");
done = true;
}
};
waiter.start();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
while (!done && ((line = is.readLine()) != null))
{
System.out.println(line);
returnMsg = returnMsg + line;
}
System.out.println(returnMsg);
System.out.println("End");
return;
}// main
}

I assume that there is a good reason why you want to run a java program from another java program and not just from a shell script, or by invoking an API - but if not - please reconsider.
As to your problem - if your application produces a lot of output (the one you are running as a process) - your application will hang. The p.waitFor() will halt until the process ends. But if you don't read the information from the InputStream - it will overflow and hang!
Advice #1: put the p.waitFor() at the end.
Advice #2: read this article. If I remember correctly it is the one I read when I had a similar problem. You can also google for "StreamGobbler" - it is a common name for a separate thread that "gobbles" your streams.
Advice #3: Don't forget the ErrorStream - if your application will produce too many errors - that stream will cause the process to hang as well.