We Keep Coding

Java Fails to Run Large Python Files

I have a Python script which I am attempting to run via code in Java.
The Python script runs fine when run through a Linux terminal command on my Ubuntu virtual machine using an identical command to the one being passed through the Java script.
The Java code runs fine when running a different Python script that runs faster than the Python script I'm attempting to run..
However, despite both the Python script running fine and the Java script running fine, somehow, when I put the two together, nothing happens: The .txt file isn't updated, so the Java script prints out whatever old value it contains.
System.out.println("starting...");
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py abc def");
Thread.sleep(900000);
# Or try System.out.println(process.waitFor());
File file = new File("/home/.../PycharmProjects/.../output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch(Exception e) {
System.out.println(e);
}
The code above should run the Python3 script at the absolute directory provided, using two arguments. The Python3 script completes after around 13 minutes and updates the output.txt file, which is then read by the Java program after waiting 15 minutes (or you can tell the thread to wait for completion-- process.WaitFor() returns 1).
def testScript():
time.sleep(780)
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
outputFile = open(os.path.dirname(os.path.abspath(__file__)) + "/output/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()
The script above is a good stand-in for the Python script. If you lower the sleep time to 10 minutes for the Python script, it runs when Java sends the command. But, at the sleep times shown above, Java apparently fails to run the script, or the script run attempt ends in failure.
Additional info: the Java command is activated using a JavaFX button. The Java script has been developed in IntelliJ IDEA and the Python script was created using PyCharm.
My question is, what are possible causes for this problem, when both scripts work fine on their own?

As a simple suggestion, you should not rely on Thread.sleep method with a fixed parameter such as 15 minutes. Your data may grow or shrink and that way of proceeding is not efficient.
You could try to call the Process.waitFor() method so that when the python process is over, your thread continues.
Moreover, you could try to use ProcessBuilder that sometimes helps when facing buggy System exec cases.
Here is some code. in sub(), you can not change the python program, but for sub2() to work, you have to modify the python program so that its output is on the standard out and Java would do the redirect to the output.txt file.
public void sub() {
System.out.println("startig...");
Scanner newLineReader = null;
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py /home/.../PycharmProjects/.../fraudAnalysis.db 500");
process.waitFor();
File file = new File("/home/.../PycharmProjects/.../output.txt");
newLineReader = new Scanner(file);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
} catch(IOException ioe) {
ioe.printStackTrace();
}catch(InterruptedException ie) {
ie.printStackTrace();
}finally {
newLineReader.close();
}
}
public void sub2() {
ProcessBuilder pb =
new ProcessBuilder("python3",
"/home/.../PycharmProjects/.../fraudanalysis.py",
"/home/.../PycharmProjects/.../fraudAnalysis.db", "500");
File log = new File("/home/.../PycharmProjects/.../output.txt");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = null;
try {
p = pb.start();
} catch (IOException e) {
e.printStackTrace();
}
try {
p.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
Scanner newLineReader = null;
try{
newLineReader = new Scanner(log);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
}catch(IOException ioe) {
ioe.printStackTrace();
}
}

I was able to get it to work with a small modification. I used relative file locations and TimeUnit.MINUTES.sleep(15);
package org.openjfx;
import java.io.File;
import java.util.Scanner;
import java.util.concurrent.TimeUnit;
public class TestWait {
public static void main(String[] args) {
System.out.println("starting...");
String dir="src/main/resources/org/openjfx/";//location of the python script
try {
System.out.println("Working Directory = " + System.getProperty("user.dir"));
//System.out.println("python3 " + dir+"fraudanalysis.py abc def");
Process process = Runtime.getRuntime().exec("python3 " + dir+"fraudanalysis.py abc def");
System.out.println(process.waitFor());
TimeUnit.MINUTES.sleep(15);
File file = new File("src/main/resources/org/openjfx/output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch (Exception e) {
System.out.println(e);
}
}
}
Here is the python I used.
import sys
import time
def testScript():
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
time.sleep(780)
outputFile = open("src/main/resources/org/openjfx/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()

it's a timeout error. can't be fixed. just pick between Java and Python and write everything in it. no reason to use both.

how can I execute "java" command by java code?

I'm trying to run a multi-agent system using JADE by code.
Usually, I can run the jade system by command line using the following command:
java jade.Boot -gui
It also works well using Netbeans IDE by changing the project properties and pointing the run configuration to the jade.Boot class.
My problem is: I want to run this system by java code... let's say, when a user clicks a button, and as far as I know, this command specified above should work using the following code:
Process p=null;
try {
p = Runtime.getRuntime().exec("java jade.Boot -gui;");
}
catch (IOException ex) {
Logger.getLogger(SimulationCreator.class.getName()).log(Level.SEVERE, null, ex);
}
BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
String s;
try {
while ((s = stdInput.readLine()) != null) {
System.out.println(s);
}
} catch (IOException ex) {
Logger.getLogger(SimulationCreator.class.getName()).log(Level.SEVERE, null, ex);
}
But I'm getting nothing, neither the JADE system is running nor I'm getting any error.
What am I missing ?
P.S.: I'm working on ubuntu 20.0.
P.S.: running other commands by code like "ps -aux" works very well!

Your problem may be a difference between PATH of the current running VM, compared the shell (bash etc) you use and implicit passing of CLASSPATH. Check the location of java which which java in the shell that works and try using in your exec.
Also you won't see all issues when running as you don't access the error stream at same time as the output stream, changing to ProcessBuilder allows easier debugging as you can redirect the out/err streams to a file. But if JADE runs for a long time or produces a lot of output you should consumer STDOUT+ERR in background threads.
Try this in jshell:
String[] cmd = new String[]{"java", "jade.Boot", "-gui"};
ProcessBuilder pb = new ProcessBuilder(cmd);
File fileOut = new File("stdout.log");
File fileErr = new File("stderr.log");
pb.redirectError(fileErr);
pb.redirectOutput(fileOut);
Process p = pb.start();
int rc = p.waitFor();
String stdout = Files.readString(fileOut.toPath());
String stderr = Files.readString(fileErr.toPath());
System.out.println("Exit : "+rc +' '+(rc == 0 ? "OK":"**** ERROR ****"));
System.out.println("STDOUT : "+stdout);
System.out.println("STDERR : "+stderr);

How do I run GDB and accept GDB commands through Java output console?

I am trying to build a GUI for GCC which has some basic functionalities like compile, link, execute, debug, etc for C++ programs using Java. I am creating strings of command which I pass to the ProcessBuilder and run it via command prompt and GCC.
command = "cd src & cd Resources & g++ " + compileFile.cpp +" -Wall "+ " -o "+ "tempOut";
This is a sample code for compiling the file.
Part of this is the debug functionality for which I am using GDB. Now the problem is GDB needs additional input to add breakpoints, remove breakpoints and so on. I am having trouble on how to pass these necessary inputs to GDB via Java terminal. If I pass the commands in the command prompt, it is working fine and I get the desired output.
enter image description here
But whenever I fire the GDB command from the Java program, I cannot pass any inputs from the IDE's terminal. I am aware that each GDB command uses a different process and I tried attaching Java's process ID to GDB but I just get a blank output console. It seems that the GDB session has started but there is no way to interact with that process through the IDE's output console.
int pid = Integer.parseInt(ManagementFactory.getRuntimeMXBean().getName().split("#")[0]);
command = "gdb attach "+ pid;
fireCommand();
EDIT
This is the method that interacts with command prompt to take input and display output in the IDE's output console:
public void fireCommand() {
String line;
String os = System.getProperty("os.name");
ProcessBuilder builder;
if(os.startsWith("Win")) {
builder = new ProcessBuilder("cmd.exe", "/c", command);
}
else{
builder = new ProcessBuilder("bash", "-c", command);
}
try {
process = builder.start();
BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream()));
while ((line = reader.readLine()) != null) {
if(line.contains("Input the value")) {
//any other user input in non debug execution
String value = JOptionPane.showInputDialog(line);
BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(process.getOutputStream()));
writer.write(value, 0, value.length());
writer.newLine();
writer.close();
}
else {
output.append(line).append("\n");
}
}
int exitVal = process.waitFor();
if (exitVal == 0) {
//display("Success!");
display(output.toString());
} else {
String lineErr;
BufferedReader readerErr = new BufferedReader(
new InputStreamReader(process.getErrorStream()));
while ((lineErr = readerErr.readLine()) != null) {
outputErr.append(lineErr).append("\n");
}
//display(exitVal);
display(outputErr.toString()); //Display the uncatched errors
}
} catch (IOException e) {
display("There was a problem with the I/O");
e.printStackTrace();
} catch (InterruptedException e) {
display("There was a interruption with the execution");
e.printStackTrace();
}
if(!outputErr.toString().isEmpty())
errorFormatDisplay(); //display Error output function
}
Any leads on this would be very helpful. Thank you.

I am aware that each GDB command uses a different process
No, gdb runs as one process. Did you mean to say that you are creating a new gdb process every time you try to pass it a gdb command ?
It seems that the GDB session has started but there is no way to interact with that process through the IDE's output console.
Maybe you should use The GDB/MI Interface, which e.g. the Emacs debugger mode gud uses.

Java Runtime.exec() to run a java program

Here's the situation. Im creating a UI which will allow make using a genetic programming system (ECJ) easier to use.
Currently you need to run a command prompt within the ECJ folder and use the commands similar to this to execute a parameter file.
java ec.Evolve -file ec\app\tutorial5\tutorial5.params
Where the full path of tutorial5 is
C:\Users\Eric\Documents\COSC\ecj\ec\app\tutorial5\tutorial5.params
and the command prompt must be executed from
C:\Users\Eric\Documents\COSC\ecj
My program makes the user select a .params file (which is located in a ecj subdirectory) and then use the Runtime.exec() to execute
java ec.Evolve -file ec\app\tutorial5\tutorial5.params
What i have so far
// Command to be executed
String cmd = "cd " + ecjDirectory;
String cmd2 = "java ec.Evolve -file " + executeDirectory;
System.out.println(cmd);
try {
Process p = Runtime.getRuntime().exec(
new String[]{"cmd.exe", "/c", cmd, cmd2});
BufferedReader r = new BufferedReader(new InputStreamReader(p.getInputStream()));
statusTF.append(r.readLine());
p.waitFor();
} catch (IOException | InterruptedException ex) {
System.out.println("FAILED: " + ex.getMessage());
statusTF.append("Failed\n");
}
Currently it outputs the change directory command but nothing else.
Can this be done?

First, the 'cd' command can't be executed by Runtime.exec() in the first place (see How to use "cd" command using Java runtime?). You should be able to just set the working directory for the process when you call exec (see below).
Second, running 'cmd.exe /c' to execute your process isn't what you want here. You won't be able to get the results of your process running, because that is returned to the command window -- which eats the error and then closes without passing the error along to you.
Your exec command should look more like this:
Process p = Runtime.getRuntime().exec(
command, null, "C:\Users\Eric\Documents\COSC\ecj");
Where 'command' looks like this:
String command = "java ec.Evolve -file ec\app\tutorial5\tutorial5.params"
Edit: For reading error messages, try this:
String error = "";
try (InputStream is = proc.getErrorStream()) {
error = IOUtils.toString(is, "UTF-8");
}
int exit = proc.waitFor();
if (exit != 0) {
System.out.println(error);
} else {
System.out.println("Success!");
}

You can use Java processbuilder:
processBuilder documentation!
you can define the working directory of the process and all other stuff.

Each call to exec() runs in a new environment, this means that the call to cd will work, but will not exist to the next call to exec().
I prefer to use Apache's Commons Exec, it's provides an excellent facade over Java's Runtime.exec() and gives a nice way to specify the working directory. Another very nice thing is they provide utilities to capture standard out and standard err. These can be difficult to properly capture yourself.
Here's a template I use. Note that this sample expects an exit code of 0, your application may be different.
String sJavaPath = "full\path\to\java\executable";
String sTutorialPath = "C:\Users\Eric\Documents\COSC\ecj\ec\app\tutorial5\tutorial5.params";
String sWorkingDir = "C:\Users\Eric\Documents\COSC\ecj";
try (
OutputStream out = new ByteArrayOutputStream();
OutputStream err = new ByteArrayOutputStream();
)
{
// setup watchdog and stream handler
ExecuteWatchdog watchdog = new ExecuteWatchdog(Config.TEN_SECONDS);
PumpStreamHandler streamHandler = new PumpStreamHandler(out, err);
// build the command line
CommandLine cmdLine = new CommandLine(sJavaPath);
cmdLine.addArgument("ec.Evolve");
cmdLine.addArgument("-file");
cmdLine.addArgument(sTutorialPath);
// create the executor and setup the working directory
Executor exec = new DefaultExecutor();
exec.setExitValue(0); // tells Executor we expect a 0 for success
exec.setWatchdog(watchdog);
exec.setStreamHandler(streamHandler);
exec.setWorkingDirectory(sWorkingDir);
// run it
int iExitValue = exec.execute(cmdLine);
String sOutput = out.toString();
String sErrOutput = err.toString();
if (iExitValue == 0)
{
// successful execution
}
else
{
// exit code was not 0
// report the unexpected results...
}
}
catch (IOException ex)
{
// report the exception...
}

How do i run a Linux terminal cmd from a java program [duplicate]

This question already has answers here:
How to run Linux commands in Java?
(9 answers)
Closed 3 years ago.
I am trying to call my rrdtool cmd from a java class, not sure how to go about it.
I have tested my RRDTool cmd from my terminal and it is successful, see below.
rrdtool update mydb.rrd 1385056701:6:5
How do i execute this cmd from a java class?

You can use the below command format to run your Linux command.
Runtime r = Runtime.getRuntime();
Process p = r.exec(yourcmd);
Please go through Running unix command from Java and Unable to run Unix command in Java-Stackoverflow
Hope you get your answers here.

try this
public class ShellTest {
public static void main(String[] args) throws java.io.IOException, java.lang.InterruptedException {
// Get runtime
java.lang.Runtime rt = java.lang.Runtime.getRuntime();
// Start a new process: UNIX command ls
java.lang.Process p = rt.exec("ls");
// Show exit code of process
System.out.println("Process exited with code = " + rt.exitValue());
}
}
also check here for more details

Try like this(As answered by paxdiablo):
public static void main(String args[]) {
String s;
Process p;
try {
p = Runtime.getRuntime().exec("ls -aF");
BufferedReader br = new BufferedReader(
new InputStreamReader(p.getInputStream()));
while ((s = br.readLine()) != null)
System.out.println("line: " + s);
p.waitFor();
System.out.println ("exit: " + p.exitValue());
p.destroy();
} catch (Exception e) {}
}
Also check java.lang.Runtime.exec for details.
Executes the specified string command in a separate process.
This is a convenience method. An invocation of the form exec(command)
behaves in exactly the same way as the invocation exec(command, null,
null).

You can use Runtime#exec for this purpose. If you want a Java-like control over the started process, there is a great library called zt-exec that makes handeling processes much easier. The ProcessBuilder is also offering a minor API improvement over Runtime#exec that ships with the Java standard library.
Something you have to take care of is that Java processes come with very little buffer for the in- and output streams what blocks processes once these buffers run full. This happens silently. zt-exec can help you with that.

You can have a look at :
https://github.com/OpenNMS/opennms/blob/master/opennms-rrd/opennms-rrd-rrdtool/opennms-rrdtool-api/src/main/java/org/opennms/netmgt/rrd/rrdtool/JniRrdStrategy.java
for a real life use of rrdtool in java.
Or at
https://code.google.com/p/rrd4j/
For a native version of rrdtool in java.

I runned my rrdtool command in java program as follow:
Process p = null;
try {
ProcessBuilder pb = new ProcessBuilder("/usr/bin/rrdtool","lastupdate", rrdPath);
pb.redirectErrorStream(true);
p = pb.start();
int exitVal = p.waitFor();
if (exitVal == 0)
System.out.println("exitVal of rrdLastUpdate is Successful");
else
System.out.println("exitVal of rrdLastUpdate is Abnormal");
} catch (Exception e) {
System.out.println("Problem in executing rrdlastupdate()");
e.printStackTrace();
}//end of try-catch
I hope this be useful for U :) I worked with some other rrdtool commands in java. if you need more help, I will be happy to help.