Executing a python script inside java map reduce - java

I need to execute a python script inside a java mapreduce program.
Here, in the mapper class , I need execute the the python command :
python methratio.py --ref=../refernce/referncefile -r -g --out=Ouputfile ./Inputfile
Here the Inputfile is the input file in hdfs and the outputfile (in hdfs) is where the python script writes the ouput.
Can I use process builder or any other better options are there ??

I don't know if this can help you, but you can execute system commands in this way in java:
public static void main(String[] args) throws IOException {
String s = null;
String command = "python <your_command>";
Process p = Runtime.getRuntime().exec(command);
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
// read the output from the command
System.out.println("Here is the standard output of the command:\n");
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
}
You can see a detailed example in http://alvinalexander.com/java/edu/pj/pj010016
Hope this help you :D

Related

How to print command line execution output using java

While running below code prompt will open and it will show java version information. I want to print that java version information. please help me on this
public class Test {
public static void main(String[] args) throws IOException {
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec("cmd.exe /c start cmd.exe /k \"java -version\"");
BufferedReader stdInput = new BufferedReader(new InputStreamReader(proc.getInputStream()));
// Read the output from the command
System.out.println("Here is the standard output of the command:\n");
String s = null;
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
}
}
At the moment you read the stream the process might not actualy be done yet. I'm pretty sure you first have to wait int exitVal = proc.waitFor(); on the process to finish before reading the outputStream.
for debugging wrapping part of this in a try with logging on the proc.getErrorStream() might help to

Issue in calling Python code from Java (without using jython)

I found this as one of the ways to run (using exec() method) python script from java. I have one simple print statement in python file. However, my program is doing nothing when I run it. It neither prints the statement written in python file nor throws an exception. The program just terminates doing nothing:
Process p = Runtime.getRuntime().exec("C:\\Python\\Python36-32\\python.exe C:\\test2.py");
Even this is not creating the output file:
Process p = Runtime.getRuntime().exec("C:\\Python\\Python36-32\\python.exe C:\\test2.py output.txt 2>&1");
What is the issue?
I think you could try your luck with the ProcessBuilder class.
If I read the Oracle documentation correctly, the std inputs and outputs are directed to pipes by default but the ProcessBuilder has an easy method for you to explicitly set output (or input) to a file on your system or something else.
If you want your Python program to use the same output as your Java program (likely stdout and stderr), you can use stg like this:
ProcessBuilder pb = new ProcessBuilder("C:\\Python\\Python36-32\\python.exe", "C:\\test2.py");
pb.redirectOutput(Redirect.INHERIT);
Process p = pb.start();
You can use the ProcessBuilder API, redirecting the output to a file and then wait for the result.
public class Main {
public static final String PYTHON_PATH = "D:\\Anaconda3\\python.exe";
public static final String PATH_TO_SCRIPT = "D:\\projects\\StartScript\\test.py";
public static void main(String[] args) throws IOException, InterruptedException {
ProcessBuilder builder = new ProcessBuilder();
builder.command(PYTHON_PATH, PATH_TO_SCRIPT);
// Redirect output to a file
builder.redirectOutput(new File("output.txt"));
builder.start().waitFor();
// Print output to console
ProcessBuilder.Redirect output = builder.redirectOutput();
File outputFile = output.file();
BufferedReader br = new BufferedReader(new FileReader(outputFile));
String st;
while ((st = br.readLine()) != null) {
System.out.println(st);
}
}
}
The python file test.py contains a simple print statement:
print("Hello from python")
I guess it would be even simpler, if you do not need to wait for the result.
Using the Process API should work, too.
Like in your example (I am using the same constants declared above):
Process p = Runtime.getRuntime().exec(PYTHON_PATH + " " + PATH_TO_SCRIPT);
p.waitFor();
byte[] buffer = new byte[1024];
byte[] errBuffer = new byte[1024];
p.getInputStream().read(buffer);
p.getErrorStream().read(errBuffer);
System.out.println(new String(buffer));
System.out.println(new String(errBuffer));
To see the output of the print statement, you need to wait and redirect the streams. Same for the error stream.
Now if you break the python script like this:
print("Hello from python')
you should be able to see the error printed as well.
One way to start a python process is using an entrypoint - test.cmd
echo Hello
python hello.py
here is hello.py
#!/usr/bin/env python3
import os
if not os.path.exists('dir'):
os.makedirs('dir')
Here is my Java code:
public static void main(String[] args) throws IOException {
try {
Process p = Runtime.getRuntime().exec("test.cmd");
p.waitFor();
Scanner sc = new Scanner(p.getInputStream());
while(sc.hasNextLine()){
System.out.println(sc.nextLine());
}
sc.close();
} catch (Exception err) {
err.printStackTrace();
}
}

How can I run a shell script, written in Bash on Ubuntu, from Java in a Windows 10 environment?

How can I run a shell script, written in Bash on Ubuntu, from Java in a Windows 10 environment?
I'm trying to use this code but it is not running nor executing the script.
public static void main(String[] args) throws IOException {
Runtime rt = Runtime.getRuntime();
ProcessBuilder builder = new ProcessBuilder(
"bash.exe", "/mnt/d/Kaldi-Java/kaldi-trunk/tester.sh");
Process p = builder.start();
BufferedReader r = new BufferedReader(new
InputStreamReader(p.getInputStream()));
String line;
while (true) {
line = r.readLine();
if (line != null) { System.out.print(line);}
else{break;}
}
}
First of all: did you try to execute this command from command line? If you did and it worked it means that problem is not with bash on windows but with your java program.
If you cannot execute it from command line then fix this problem first
I cannot test my program because I use Ubuntu but can advice you to try smth like this: (wait until program is over)
public class Main {
public static void main(String[] args) throws IOException, InterruptedException {
ProcessBuilder builder = new ProcessBuilder(
"bash.exe", "/mnt/d/Kaldi-Java/kaldi-trunk/tester.sh");
Process p = builder.start();
/* waitFor() method stops current thread until this process is over */
p.waitFor();
// I think that scanner is a nicer way of parsing output
Scanner scanner = new Scanner(p.getInputStream());
while (scanner.hasNextLine()) {
// you do not have to create `line` outside the loop
// it does not change performance of a program
String line = scanner.nextLine();
System.out.println(line);
}
}
}
If you are trying to run a script using Java in a Windows environment, I would suggested executing it differently.
I have adapted you code from a precious question asked here :
How to run Unix shell script from Java code?
Also, this question will help you with your question:
Unable to read InputStream from Java Process (Runtime.getRuntime().exec() or ProcessBuilder)
public static void main(String[] args) throws IOException {
Process proc = Runtime.getRuntime().exec(
"/mnt/d/Kaldi-Java/kaldi-trunk/tester.sh");
BufferedReader read = new BufferedReader(
new InputStreamReader(proc.getInputStream()));
while (read.ready())
{
System.out.println(read.readLine());
}
}
I believe this is what you are looking for. I believe that your Java code was a little off and these edits should help you. After you build the java executable, you could be able to run it via Window's Command Prompt.

Execute external Java code and get output

I want to execute a Java CLI-program from within another Java program and get the output of the CLI-program. I've tried two different implementations (using runtime.exec() and ProcessBuilder) and they don't quite work.
Here's the peculiar part; the implementations work (catch the output) for when executing commands such as pwd but for some reason they do not catch the output of a Hello World java program executed with java Hello.
Execution code:
public static void executeCommand(String command)
{
System.out.println("Command: \"" + command + "\"");
Runtime runtime = Runtime.getRuntime();
try
{
Process process = runtime.exec(command);
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(process.getInputStream()));
BufferedReader stdError = new BufferedReader(new
InputStreamReader(process.getErrorStream()));
// read the output from the command
System.out.println("Standard output of the command:\n");
String s = null;
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
// read any errors from the attempted command
System.out.println("Standard error of the command (if any):\n");
while ((s = stdError.readLine()) != null) {
System.out.println(s);
}
} catch (Exception e)
{
e.printStackTrace();
}
}
Example output
Command: "cd /Users/axelkennedal/Desktop && java Hello"
Standard output of the command:
Standard error of the command (if any):
Command: "pwd"
Standard output of the command:
/Users/axelkennedal/Dropbox/Programmering/Java/JavaFX/Kode
Standard error of the command (if any):
I have verified that Hello does indeed print "Hello world" to the CLI when running Hello directly from the CLI instead of via executeCommand().
Hello world
public class Hello
{
public static void main(String[] args)
{
System.out.println("Hello world!");
}
}
This "cd /Users/axelkennedal/Desktop && java Hello" is not one command but two commands separated by the &&. In general it means do the first command and if the first command succeeds do the second command. You can't pass this as a single command but you can implement the logic yourself:
eg to execute "cmd1 && cmd2"
if (Runtime.getRuntime().exec("cmd1").waitFor() == 0) {
Runtime.getRuntime().exec("cmd2").waitFor();
}
However, because in this case cmd1 is to change directories there is a better way, which is to use the directory function of ProcessBuilder instead of the first command.
Process p = new ProcessBuilder("java","hello")
.directory(new File("/Users/axelkennedal/Desktop"))
.start();

to run shell script through java

I want to run a shell script through java .I am using license generation tool,It can be call with the help of ./LicenseGen.sh command,under it I require to execute another command
create licensekey -x license-input.xml
which create a new licensekey.xml file where license-input.xml is a input file and licensekey is a output xml file how it is posssible in java please help me.
my code is
import java.io.*;
import java.util.*;
public class ProcessExample {
/**
* #param args
*/
public static void main(String args[]) throws IOException {
File file=new File("/opt");
// List<String> list=new List<String>();
ProcessBuilder processBuilder = new ProcessBuilder("./LicenseGen.sh");
processBuilder.directory(file);
Process process=processBuilder.start();
//processBuilder.command("create licensekey -x license-input.xml");
//process=processBuilder.start();
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
System.out.printf("Output of running %s is:",
Arrays.toString(args));
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
}
You can't execute the script directly since it has to be interpreted by a shell like bash.
Note that bash is an executeable.
ProcessBuilder pb = new ProcessBuilder("/bin/bash", "/path/LicenseGen.sh");
Use commons cli http://commons.apache.org/cli/
Good luck!
I have used JSch extensively for remote login and script executions. I used google Expect4j with Jsch for executing scripts on remote machines in expect mode(send/wait). Since, you have to execute command one after another, you can try this.
It can also be used for local execution that you require. The only worry is that you need to login (into your local machine) for execution.
For jsch, go to http://www.jcraft.com/jsch/
For Expect4j, go to http://code.google.com/p/expect4j/
Thanks.

Categories