We Keep Coding

Calling Python from Java (Tomcat6) as sub-process

I am trying to call a python script from a java/tomcat6 webapp. I am currently using the following code:
Process p = Runtime.getRuntime().exec("python <file.py>");
InputStream in = p.getInputStream();
InputStreamReader isr = new InputStreamReader(in);
BufferedReader b = new BufferedReader(isr);
logger.info("PYTHON OUTPUT");
String line = null;
while ( (line = b.readLine()) != null){
logger.info(line);
}
p.waitFor();
logger.info("COMPLETE PYTHON OUTPUT");
logger.info("EXIT VALUE: "+p.exitValue());
I can't really see any output in the catalinia.out file from the python script and using an adapter library like jython is not possible as the script relies on several machine learning libraries that need python's Numpy module to work.
Help?

The explanation is probably one (or more) of following:
The command is failing and writing error messages to its "stderr" fd ... which you are not looking at.
The command is failing to launch because the command name is incorrect; e.g. it can't be found on $PATH.
The command is trying to read from its stdin fd ... but you haven't provided any input (yet).
It could be a problem with command-line splitting; e.g if you are using pathnames with embedded spaces, or other things that would normally be handled by the shell.
Also, since this is python, this could be a problem with python-specific environment variables, the current directory and/or the effective user that is executing the command.
How to proceed:
Determine if the python command is actually starting. For instance. "hack" the "" to write something to a temporary file on startup.
Change to using ProcessBuilder to create the Process object. This will give you more control over the streams and how they are handled.
Find out what is going to the child processes "stderr". (ProcessBuilder allows you to redirect it to "stdout" ...)

Cannot run exe file with ProcessBuilder in Java

I am trying to run an exe file while setting some parameters for it like this:
myExePath -ini myIniPath -x myConfigFilePath
When I run it from the command line it works perfectly. But when I try running it from my Java code the process starts but after a while is not responding anymore so I have to forcibly close it. I am using this Java code:
List<String> parameters = new ArrayList<String>();
parameters.add(myexePath);
parameters.add("-ini ");
parameters.add(myIniPath);
parameters.add("-x ");
parameters.add(myConfigPath
ProcessBuilder builder = new ProcessBuilder(parameters);
Process process = builder.start();
try {
process.waitFor();
} catch (InterruptedException e) {
System.err.println("Process was interrupted");
}
Any ideas what I am doing wrong?

Does the exe use stdout, stderr, stdin? You should always read from them or close them. Depending on the implementation and buffer size not reading from them could lead to blocking.

I'm not sure if it helps, but why you use spaces?
e.g.: parameters.add("-x ");
You don't need them.
What you can also try is to put all your parameters in an array and use another constructor of ProcessBuilder which takes an array as argument.

I guess you should first get a reference to the Runtime.
You could do this
Runtime.getRuntime().exec(parameters.toString());
Your string from the parameters list may need a bit formatting.

Problem with copy files using cmd in Java

I'm trying to copy a bunch of files with a specific extension from one folder to another using the copy command, heres wat im doing,
String[] command = new String[3];
command[0] = "cmd";
command[1] = "/c";
command[2] = "copy C:\\output\\html\\*.txt C:\\output\\";
ProcessBuilder copyFiles = new ProcessBuilder(command);
p = copyFiles.start();
p.waitFor();
the thing is, this code works fine for files less than some 5 or so, but just stops responding wen the number of files are more (even for 15 files) !! and the files are not copied either!!
I dont know what the problem is, will be glad if someone could help! :)

You're not reading the output the copy command is generating.
When spawning a child process using ProcessBuilder, output generated by your child process gets written to a buffer. If this buffer isn't read from, it eventually fills up. When it fills up, the copy command can't write any more to it and so is blocked by the operating system. It is then forced to wait until space is made in the buffer by reading from it.
I ran your code with 20 files and I found that it did indeed hang.
One way to solve your problem is to redirect the output from copy to NUL. Most of the output from copy is a list of all the files it has copied, which you probably don't care too much for. To do this redirection, modify the line that assigns to command[2] to the following:
command[2] = "copy C:\\output\\html\\*.txt C:\\output\\ >NUL 2>NUL";
However, if there is a problem copying files, you might not know about it if you do this.
Alternatively, you can read the output that the copy command generates. The following code sends it to System.out, but you can easily send it elsewhere or completely ignore it if you wish:
String[] command = { "cmd", "/c", "copy C:\\output\\html\\*.txt C:\\output\\" };
ProcessBuilder copyFiles = new ProcessBuilder(command);
copyFiles.redirectErrorStream(true);
p = copyFiles.start();
// The InputStream we get from the Process reads from the standard output
// of the process (and also the standard error, by virtue of the line
// copyFiles.redirectErrorStream(true) ).
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
do {
line = reader.readLine();
if (line != null) { System.out.println(line); }
} while (line != null);
reader.close();
p.waitFor();
I gave each approach a quick test with the same 20 files and neither approach hung.
EDIT: You might also want to try a 'hybrid' approach, by throwing away what copy writes to standard output (e.g. the list of files it's copying) but using the second approach to read in what it writes to standard error (e.g. error messages). To do this, you'd add the >NUL, which redirects the standard output of copy to NUL, but you wouldn't add the 2>NUL, since that redirects standard error to NUL.

Running other programs from Java

I need to run a couple of other programs from my own Java program, basically I need to run these command line statements.
svn log --xml -v > svn.log
and
java -jar example.jar arg1 arg2
and I need to use the text outputs written to the console from these programs in my own program. I've tried Runtime.getRuntime().exec() with the svn, but it doesn't seem to be doing anything because it doesn't make a svn.log file. Also both programs need to be called in different places, the svn line needs to be called from inside one folder and the java line needs to be called from another.
Any ideas on how to go about this? If this is not possible in Java, is there a way to do it in C#?
Thanks

Here:
ProcessBuilder processbuilder
try
{
processbuilder.directory(file);
processbuilder.redirectErrorStream(true);
process = processbuilder.start();
String readLine;
BufferedReader output = new BufferedReader(new InputStreamReader(process.getInputStream()));
// include this too:
// BufferedReader output = new BufferedReader(new InputStreamReader(process.getErrorStream()));
while((readLine = output.readLine()) != null)
{
m_Logger.info(readLine);
}
process.waitFor();
}
I've used something similar. You'll actually want to do something with the readLine. I just copied and pasted from code where I didn't care what it said.

The redirection > (like the pipe |) is a shell construct and only works when you execute stuff via /bin/sh (or equivalent). So the above isn't really going to work. You could execute
/bin/sh -c "svn log --xml -v > svn.log"
and read svn.log.
Alternatively, you can read the output from the process execution and dump that to a file (if you need to dump it to a file, or just consume it directly as you read it). If you choose this route and consume stdout/stderr separately, note that when you consume the output (stdout), you need to consume stderr as well, and concurrently, otherwise buffers will block (and your spawned process) waiting for your process to consume this. See this answer for more details.

instead of piping in your command, just let it print to standard output and error output. You can access those streams from your process object that is returned from exec.

For the svn stuff use java SVNKit API.

Seeing your two commands, why don't you do it directly from Java, without executing ? You could use SVNKit for the svn part, and include directly the jars in your classpath.

Try this
public static void main(String[] args) {
try {
// Execute a command with an argument that contains a space
System.out.println(args[0]);
String[]commands = new String[]{"svn", "info", args[0]};
Process process = Runtime.getRuntime().exec(commands);
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
StringBuilder builder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
builder.append(line);
builder.append(System.getProperty("line.separator"));
}
String result = builder.toString();
System.out.println(result);
}catch(Exception e){
System.out.print(e);
}
}

External program from our Java program

How can I write a program in Java that will execute another program? Also, the input of that program should be given from our program and the output of that program should be written into a file.
This is my small set of code to get its output:
Process p = Runtime.getRuntime().exec("C:\\j2sdk1.4.0\bin\\helloworld.java");
BufferedReader input =
new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = input.readLine()) != null)
System.out.println(line);
input.close();
This was my set of code but this throws an IOException.

The API that Java offers for this is the ProcessBuilder. It is relatively straightforward to set working directory and pass parameters.
What is a little tricky is passing STDIN and reading STDERR and STDOUT, at least for non-trivial sizes thereof, because you need to start seperate threads to make sure the respective buffers get cleared. Otherwise the application that you called might block until it can write more output, and if you also wait for that process to finish (without making sure that STDOUT gets read), you will deadlock.

You can use java.lang.Process and java.lang.ProcessBuilder. You interact with the input/output of the process using getInputStream/getOutputStream/getErrorStream.
However, there's an Apache Commons library called Exec which is designed to make all of this easier. (It can normally get quite hairy when it comes to quoting command line parameters etc.) I haven't used Exec myself, but it's worth checking out.

When you only want to start other programms, you can use the exec method like this:
Runtime r = Runtime.getRuntime();
mStartProcess = r.exec(applicationName, null, fileToExecute);
StreamLogger outputGobbler = new StreamLogger(mStartProcess.getInputStream());
outputGobbler.start();
int returnCode = mStartProcess.waitFor();
class StreamLogger extends Thread{
private InputStream mInputStream;
public StreamLogger(InputStream is) {
this.mInputStream = is;
}
public void run() {
try {
InputStreamReader isr = new InputStreamReader(mInputStream);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
}
exec:
public Process exec(String command, String envp[], File dir)
#param command a specified system command.
#param envp array of strings, each element of which
has environment variable settings in format
<i>name</i>=<i>value</i>.
#param dir the working directory of the subprocess, or
<tt>null</tt> if the subprocess should inherit
the working directory of the current process.

Please do not edit your question so that it does not fit the original answers anymore.
If you have follow-up question, clearly mark them as such, or ask them as a seperate questions, or use comments or something.
As for your IOException, please give the error message it shows.
Also, it seems as if you are trying to run a ".java" file directly. That will not work. The methods described here are to launch native binary executables. If you want to run a ".java" file, you have to compile it to a class, and the invoke that class' main method.

What platform are you in?
If you are on *nix you can type:
java MyProgram | myexternalprogram > myfilename.txt