Java Fails to Run Large Python Files - java

I have a Python script which I am attempting to run via code in Java.
The Python script runs fine when run through a Linux terminal command on my Ubuntu virtual machine using an identical command to the one being passed through the Java script.
The Java code runs fine when running a different Python script that runs faster than the Python script I'm attempting to run..
However, despite both the Python script running fine and the Java script running fine, somehow, when I put the two together, nothing happens: The .txt file isn't updated, so the Java script prints out whatever old value it contains.
System.out.println("starting...");
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py abc def");
Thread.sleep(900000);
# Or try System.out.println(process.waitFor());
File file = new File("/home/.../PycharmProjects/.../output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch(Exception e) {
System.out.println(e);
}
The code above should run the Python3 script at the absolute directory provided, using two arguments. The Python3 script completes after around 13 minutes and updates the output.txt file, which is then read by the Java program after waiting 15 minutes (or you can tell the thread to wait for completion-- process.WaitFor() returns 1).
def testScript():
time.sleep(780)
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
outputFile = open(os.path.dirname(os.path.abspath(__file__)) + "/output/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()
The script above is a good stand-in for the Python script. If you lower the sleep time to 10 minutes for the Python script, it runs when Java sends the command. But, at the sleep times shown above, Java apparently fails to run the script, or the script run attempt ends in failure.
Additional info: the Java command is activated using a JavaFX button. The Java script has been developed in IntelliJ IDEA and the Python script was created using PyCharm.
My question is, what are possible causes for this problem, when both scripts work fine on their own?

As a simple suggestion, you should not rely on Thread.sleep method with a fixed parameter such as 15 minutes. Your data may grow or shrink and that way of proceeding is not efficient.
You could try to call the Process.waitFor() method so that when the python process is over, your thread continues.
Moreover, you could try to use ProcessBuilder that sometimes helps when facing buggy System exec cases.
Here is some code. in sub(), you can not change the python program, but for sub2() to work, you have to modify the python program so that its output is on the standard out and Java would do the redirect to the output.txt file.
public void sub() {
System.out.println("startig...");
Scanner newLineReader = null;
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py /home/.../PycharmProjects/.../fraudAnalysis.db 500");
process.waitFor();
File file = new File("/home/.../PycharmProjects/.../output.txt");
newLineReader = new Scanner(file);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
} catch(IOException ioe) {
ioe.printStackTrace();
}catch(InterruptedException ie) {
ie.printStackTrace();
}finally {
newLineReader.close();
}
}
public void sub2() {
ProcessBuilder pb =
new ProcessBuilder("python3",
"/home/.../PycharmProjects/.../fraudanalysis.py",
"/home/.../PycharmProjects/.../fraudAnalysis.db", "500");
File log = new File("/home/.../PycharmProjects/.../output.txt");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = null;
try {
p = pb.start();
} catch (IOException e) {
e.printStackTrace();
}
try {
p.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
Scanner newLineReader = null;
try{
newLineReader = new Scanner(log);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
}catch(IOException ioe) {
ioe.printStackTrace();
}
}

I was able to get it to work with a small modification. I used relative file locations and TimeUnit.MINUTES.sleep(15);
package org.openjfx;
import java.io.File;
import java.util.Scanner;
import java.util.concurrent.TimeUnit;
public class TestWait {
public static void main(String[] args) {
System.out.println("starting...");
String dir="src/main/resources/org/openjfx/";//location of the python script
try {
System.out.println("Working Directory = " + System.getProperty("user.dir"));
//System.out.println("python3 " + dir+"fraudanalysis.py abc def");
Process process = Runtime.getRuntime().exec("python3 " + dir+"fraudanalysis.py abc def");
System.out.println(process.waitFor());
TimeUnit.MINUTES.sleep(15);
File file = new File("src/main/resources/org/openjfx/output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch (Exception e) {
System.out.println(e);
}
}
}
Here is the python I used.
import sys
import time
def testScript():
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
time.sleep(780)
outputFile = open("src/main/resources/org/openjfx/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()

it's a timeout error. can't be fixed. just pick between Java and Python and write everything in it. no reason to use both.

Related

Printing commands to bat/cmd file with Java

How to print commands to bat/cmd file using Java? I have created a method that opens this bat file and now the program should write commands to this bat file. For instance, I have a string variable "Command" and the program must write this command to bat file.
Here I attach the code.
private static void openBat(){
File file = new File(lockerPath);
try {
if (file.exists()) {
Process pro = Runtime.getRuntime().exec("rundll32 url.dll,FileProtocolHandler " + lockerPath);
pro.waitFor();
} else {
System.out.println("file does not exist");
}
} catch (Exception e) {
System.out.println(e);
}
}
This is the code to open bat file, and the next code is to write commands:
private static void printing(int password ){
try {
ProcessBuilder processBuilder = new ProcessBuilder("cmd", "/C", lockerPath);
Process process = processBuilder.start();
process.waitFor();
List<String> commands = new ArrayList<>();
commands.add(String.valueOf(password));
processBuilder.command(commands);
}catch (Exception ex){
ex.printStackTrace();
}
}
It doesn't write anything to the file.
I will be very grateful for your help.
I still don't think I completely understand your question. If you just want to simulate the user entering a value to a batch file via a java program, then the below code does that.
First I wrote a batch file.
#echo off
set /P pw=
echo You entered: %pw%
It simply waits for the user to enter a value and assigns that value to a variable named pw. After the user enters the value, the batch file displays the entered value.
Here is the java code that runs the above batch file and enters a value.
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
public class Main {
public static void main(String[] args) {
ProcessBuilder pb = new ProcessBuilder("cmd.exe", "/C", "getusrpw.bat");
try {
Process p = pb.start(); // throws java.io.IOException
BufferedReader stdout = p.inputReader();
BufferedReader stderr = p.errorReader();
BufferedWriter stdin = p.outputWriter();
stdin.write("secret");
stdin.newLine();
stdin.flush();
String output = stdout.readLine();
while (output != null) {
System.out.println("OUT> " + output);
output = stdout.readLine();
}
String error = stderr.readLine();
while (error != null) {
System.out.println("ERR> " + error);
error = stderr.readLine();
}
int exitStatus = p.waitFor(); // throws java.lang.InterruptedException
System.out.println("Process exit status = " + exitStatus);
}
catch (InterruptedException | IOException x) {
x.printStackTrace();
}
}
}
The name of the batch file is getusrpw.bat.
stdout is for reading output generated by the batch file.
stderr is for reading error output. Note that there may be no error output.
stdin is for sending input to the batch file.
Note that methods inputReader, outputWriter and errorReader were added in JDK 17. If you are using an earlier version, use methods getInputStream, getOutputStream and getErrorStream, respectively.
The Java program enters the value secret.
The Java program prints the output generated by the batch file.
When I run the above Java code, it produces the following output:
OUT> You entered: secret
Process exit status = 0

Java CommandBuilder incomplete execution of copy command

I built a web crawler that records a livestream. After the recording is finished, the individual *.ts files are to be merged into a video. Since after my research FFMPEG has only a complex procedure over a text file as input parameter, it seems to me clearly eifnacher to use the available functions of the operating system.
I can use cmd.exe /c copy /b *.ts J:\final\output.mp4
to merge my ts files. However, as soon as I execute the command via the CommandBuilder, only the first 5 to 10 files are merged. So the video will be only a few seconds long. If I execute the command manually via Windows, all files are merged correctly. How does this happen? My commandExecture method looks like this:
private int execCommand(String dirPath, String command) {
try {
String[] commandArr = command.split(" ");
ProcessBuilder pb = new ProcessBuilder(commandArr);
pb.directory(new File(dirPath));
pb. redirectErrorStream(true);
Process p = pb.start();
return p.exitValue();
} catch (IOException e) {
e.printStackTrace();
}
return -9999;
}
I suspected that the program exited before the command was fully executed. So I added the following after pb.start() :
while(p.isAlive()) {
try {
Thread.sleep(500);
System.out.println("running");
} catch (InterruptedException e) {
e.printStackTrace();
}
}
also this did not change anything. How does this happen and how can I make it merge automatically?

how to launch a shell script in a new gnome terminal, from a java program

I'm trying to run a shell script (say myscript.sh) from a java program.
when i run the script from terminal, like this :
./myscript.sh
it works fine.
But when i call it from the java program, with the following code :
try
{
ProcessBuilder pb = new ProcessBuilder("/bin/bash","./myScript.sh",someParam);
pb.environment().put("PATH", "OtherPath");
Process p = pb.start();
InputStreamReader isr = new InputStreamReader(p.getInputStream());
BufferedReader br = new BufferedReader(isr);
String line ;
while((line = br.readLine()) != null)
System.out.println(line);
int exitVal = p.waitFor();
}catch(Exception e)
{ e.printStackTrace(); }
}
It doesnt goes the same way.
Several shell commands (like sed, awk and similar commands) get skipped and donot give any output at all.
Question : Is there some way to launch this script in a new terminal using java.
PS : i've found that "gnome-terminal" command launches a new terminal in shell,
But, i'm unable to figure out, how to use the same in a java code.
i'm quite new to using shell scripting. Please help
Thanks in advance
In java:
import java.lang.Runtime;
class CLI {
public static void main(String args[]) {
String command[] = {"/bin/sh", "-c",
"gnome-terminal --execute ./myscript.sh"};
Runtime rt = Runtime.getRuntime();
try {
rt.exec(command);
} catch(Exception ex) {
// handle ex
}
}
}
And the contents of the script are:
#!/bin/bash
echo 'hello!'
bash
Notes:
You'll do this in a background thread or a worker
The last command, in the shell script, is bash; otherwise execution completes and the terminal is closed.
The shell script is located in the same path as the calling Java class.
Don't overrwrite your entire PATH...
pb.environment().put("PATH", "OtherPath"); // This drops the existing PATH... ouch.
Try this instead
pb.environment().put("PATH", "OtherPath:" + pb.environment().get("PATH"));
Or, use the full directories to your commands in your script file.
You must set your shell script file as executable first and then add the below code,
shellScriptFile.setExecutable(true);
//Running sh file
Process exec = Runtime.getRuntime().exec(PATH_OF_PARENT_FOLDER_OF_SHELL_SCRIPT_FILE+File.separator+shellScriptFile.getName());
byte []buf = new byte[300];
InputStream errorStream = exec.getErrorStream();
errorStream.read(buf);
logger.debug(new String(buf));
int waitFor = exec.waitFor();
if(waitFor==0) {
System.out.println("Shell script executed properly");
}
This worked for me on Ubuntu and Java 8
Process pr =new ProcessBuilder("gnome-terminal", "-e",
"./progrm").directory(new File("/directory/for/the/program/to/be/executed/from")).start();
The previous code creates a new terminal in a specificied directory and executes a command
script.sh Must have executable permissions
public class ShellFileInNewTerminalFromJava {
public static void main(String[] arg) {
try{
Process pr =new ProcessBuilder("gnome-terminal", "-e", "pathToScript/script.sh").start();
}catch(Exception e){
e.printStackTrace();
}
}
}

Java program that calls external program behaving asynchronously?

All,
I originally had a shell script that called SQLLoader (Oracles data upload tool).
The problem was that SQLLoader takes a plain text password as input so I decided to build a Java application to call SQLLoader internally passing a decrypted password into the command string.
e.g.
sqlldr user/pass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv
So with my java wrapper it became this in my shell script
java -jar sqlloader.jar sqlloader.ctl mydata.csv
However a new problem developed when SQLLoader complained there was no file to load. After some head scratching it was discovered that a subsequent command in my shell script seemed to be executing while my java application was still running. Therefore it was behaving asynchronously.
The next command was moving the input file sqlloader was using before it could get a chance to use it. So I put a sleep command in of 20 seconds to give my java application time to run.
java -jar sqlloader.jar sqlloader.ctl mydata.csv
echo $?
sleep 20
if [ $? -ne 0 ]
then
echo "SQLLoader failed during execution, please check the log : "
mv mydata.csv
else
echo "SQLLoader successfully processed file : "
mv mydata.csv
fi
Does anyone know why unix is behaving this way, does Java execute my SQLLoader as a different user/ thread?
This is my java code:
Runtime Rt;
Process Prc;
Prc = Rt.exec("sqlldr user/decryptedpass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv);
system.exit(0);
I checked the Runtime Class for anything about it being Asynchronous but couldnt find anything
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
Any theories or suggestions?
Thanks
Yes. If you look at Runtime.exec again it does specify that it will launch a new process in the specified environment (e.g. independently of the current "environment" or as you put it asynchronously). You should use ProcessBuilder to create a Process and then waitFor that Process to finish before calling System.exit - which certainly isn't mandatory. Something like this
public static void main(String[] args) {
// String command = "/usr/bin/sleep 5";
List<String> command = new ArrayList<String>();
command.add("c:/cygwin/bin/sleep");
command.add("5");
ProcessBuilder pb = new ProcessBuilder(command);
BufferedReader is = null;
try {
System.out.println("Starting command " + command);
Process p = pb.start();
int ret = p.waitFor();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = is.readLine()) != null) {
System.out.println(line);
}
if (ret == 0) {
System.out.println("Command has completed.");
System.exit(ret);
} else {
System.out.println("Command completed with return code " + ret);
System.exit(ret);
}
} catch (Exception e) {
System.out.println("Caught Exception " + e.getMessage()
+ " running command " + command);
e.printStackTrace();
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
}
}
}
System.out.println("COMMAND FAILED");
System.exit(1);
}
You need to wait for process completion, you should also read all output (stdout and stderr) from the process you are starting.
If you call exit() after exec(), Java will do just that - exit immediatedly.
Here is an article that explains Runtime.exec pitfalls: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4 (also consider the other pages).

Run monkey from java

I have this script in file script.txt
And I run this like this
monkeyrunner /home/user/script.txt
this is my script.txt
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
import time
device = MonkeyRunner.waitForConnection("wait forever","emulator-5554")
package = 'com.pak.pak1'
activity = 'com.pak.pak1.MyActivity'
runComponent = package + '/' + activity
# Runs the component
device.startActivity(component=runComponent)
time.sleep(1)
The thing I want to do is to run the script from java
This code runs a shell command for example to srart the script
try {
new Thread() {
public void run() {
Process p;
try {
p = Runtime.getRuntime().exec("monkeyrunner /home/user/script.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
br.close();
p.waitFor();
} catch (Exception e) {
//e.printStackTrace();
}
}
}.start();
} catch (Exception ie) {
}
And finally mu question is how can I directly from java run the monkey runner commands, I do not want to have the script.txt file. Is this possible ? My goal is to run the monkey runner but I do not want to have the script.txt file
Apparently, if you include the MonkeyRunner chimpchat.jar (and it's jar depedencies) on your classpath, then you can call the monkey runner Java classes directly inside your Java application. Check out this class and this class that make up an example:
Another thread on this subject
This looks awfully complicated, but still..
monkeyrunner can run interactively, so write directly to stdin (get it from p.getOutputStream()) all strings you want it to run.
you might need to exhaust the stdout before issuing any command, but I don't think that will be the case.

Categories