We Keep Coding

Java Fails to Run Large Python Files

I have a Python script which I am attempting to run via code in Java.
The Python script runs fine when run through a Linux terminal command on my Ubuntu virtual machine using an identical command to the one being passed through the Java script.
The Java code runs fine when running a different Python script that runs faster than the Python script I'm attempting to run..
However, despite both the Python script running fine and the Java script running fine, somehow, when I put the two together, nothing happens: The .txt file isn't updated, so the Java script prints out whatever old value it contains.
System.out.println("starting...");
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py abc def");
Thread.sleep(900000);
# Or try System.out.println(process.waitFor());
File file = new File("/home/.../PycharmProjects/.../output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch(Exception e) {
System.out.println(e);
}
The code above should run the Python3 script at the absolute directory provided, using two arguments. The Python3 script completes after around 13 minutes and updates the output.txt file, which is then read by the Java program after waiting 15 minutes (or you can tell the thread to wait for completion-- process.WaitFor() returns 1).
def testScript():
time.sleep(780)
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
outputFile = open(os.path.dirname(os.path.abspath(__file__)) + "/output/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()
The script above is a good stand-in for the Python script. If you lower the sleep time to 10 minutes for the Python script, it runs when Java sends the command. But, at the sleep times shown above, Java apparently fails to run the script, or the script run attempt ends in failure.
Additional info: the Java command is activated using a JavaFX button. The Java script has been developed in IntelliJ IDEA and the Python script was created using PyCharm.
My question is, what are possible causes for this problem, when both scripts work fine on their own?

As a simple suggestion, you should not rely on Thread.sleep method with a fixed parameter such as 15 minutes. Your data may grow or shrink and that way of proceeding is not efficient.
You could try to call the Process.waitFor() method so that when the python process is over, your thread continues.
Moreover, you could try to use ProcessBuilder that sometimes helps when facing buggy System exec cases.
Here is some code. in sub(), you can not change the python program, but for sub2() to work, you have to modify the python program so that its output is on the standard out and Java would do the redirect to the output.txt file.
public void sub() {
System.out.println("startig...");
Scanner newLineReader = null;
try {
Process process = Runtime.getRuntime().exec("python3 /home/.../PycharmProjects/.../fraudanalysis.py /home/.../PycharmProjects/.../fraudAnalysis.db 500");
process.waitFor();
File file = new File("/home/.../PycharmProjects/.../output.txt");
newLineReader = new Scanner(file);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
} catch(IOException ioe) {
ioe.printStackTrace();
}catch(InterruptedException ie) {
ie.printStackTrace();
}finally {
newLineReader.close();
}
}
public void sub2() {
ProcessBuilder pb =
new ProcessBuilder("python3",
"/home/.../PycharmProjects/.../fraudanalysis.py",
"/home/.../PycharmProjects/.../fraudAnalysis.db", "500");
File log = new File("/home/.../PycharmProjects/.../output.txt");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = null;
try {
p = pb.start();
} catch (IOException e) {
e.printStackTrace();
}
try {
p.waitFor();
} catch (InterruptedException e) {
e.printStackTrace();
}
Scanner newLineReader = null;
try{
newLineReader = new Scanner(log);
String line;
while((line=newLineReader.nextLine())!=null) {
System.out.println(line);
}
}catch(IOException ioe) {
ioe.printStackTrace();
}
}

I was able to get it to work with a small modification. I used relative file locations and TimeUnit.MINUTES.sleep(15);
package org.openjfx;
import java.io.File;
import java.util.Scanner;
import java.util.concurrent.TimeUnit;
public class TestWait {
public static void main(String[] args) {
System.out.println("starting...");
String dir="src/main/resources/org/openjfx/";//location of the python script
try {
System.out.println("Working Directory = " + System.getProperty("user.dir"));
//System.out.println("python3 " + dir+"fraudanalysis.py abc def");
Process process = Runtime.getRuntime().exec("python3 " + dir+"fraudanalysis.py abc def");
System.out.println(process.waitFor());
TimeUnit.MINUTES.sleep(15);
File file = new File("src/main/resources/org/openjfx/output.txt");
Scanner newLineReader = new Scanner(file);
System.out.println(newLineReader.nextLine());
} catch (Exception e) {
System.out.println(e);
}
}
}
Here is the python I used.
import sys
import time
def testScript():
return_string1 = sys.argv[1]
return_string2 = sys.argv[2]
time.sleep(780)
outputFile = open("src/main/resources/org/openjfx/output.txt", "w+")
outputFile.write(return_string1 + " " + return_string2)
print("Python run complete")
if __name__ == "__main__":
testScript()

it's a timeout error. can't be fixed. just pick between Java and Python and write everything in it. no reason to use both.

Stopping Minecraft Server in Eclipse

I have been trying this for hours and I don't know what to do. I'm creating a Minecraft Bukkit plugin which it's supposed to have a "restart" command, which sends some warning messages and after that I would like to close the server. I created a batch file to do it but, I want Java to do it for me when the warnings are over, something like, idk:
open("C:/Users/Blablabla/Desktop/close.bat");
And for the moment I have this:
public String document = "C:/Users/Joan-Server/Desktop/Server/closer.bat";
if (args[0].equalsIgnoreCase("restart")) {
while(x != 0) {
Bukkit.broadcastMessage(plugin.name + ChatColor.GOLD + "¡ATENCIÓN! " + ChatColor.WHITE + "Reinicio del servidor en "
+ ChatColor.RED + x + " segundos.");
//Try--
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
x = x - 1;
}
//Try--
}

You could use something like this to execute your bat file:
try{
Process p = Runtime.getRuntime().exec("C:/path/to/file.bat");
p.waitFor();
}catch( IOException ex ){
//Catch if file doesn't exist or is inaccessible
}
catch( InterruptedException ex ){
//If the p.waitFor() is interrupted
}
Use this to execute your restart script and you should be good to go! Assuming your restart script stops the server, waits, and starts the server again.
Let me know if you need anything else!

You could also do your restart directly from your plugin via
getServer().dispatchCommand(getServer().getConsoleSender(), "restart");
If you really want to run the batch file, use
Runtime.getRuntime().exec("C:\\Users\\Blablabla\\Desktop\\close.bat");
If you also want to handle the console output of the bat file, check out this answer.

Java - executing external exe file

I have a executable program made on C++ (cf.exe) that takes parameters from a few text files (.txt), performs some calculations, and creates a file with the results (results.txt). This program needs no interaction. Once executed, when the calculations are done, it's automatically closed.
Then, I have other program made on Java, which contains a button that executes the cf.exe. When the button is pressed the following method is called:
public void executeExternalProgram(){
String filePath = "C++" + File.separator + "cf.exe";
try {
System.out.println("Executing C++ exe...");
Process p = Runtime.getRuntime().exec(filePath);;
} catch (Exception e) {
e.printStackTrace();
}
}
I don't get any error, but the cf.exe is not being executed when I press the button. When I execute manually the cf.exe, the results file is correctly created. Furthermore, I've tried to copy other programs, such as notepad.exe, to the same folder and execute it through Java without any problem. Can anyone see why it's not working?

I found that changing the command given to the process makes it work correctly. The change is the following:
Process p = Runtime.getRuntime().exec("cmd /c start "+ filePath);
Then, the updated code for the method called when the button is pushed:
public void executeExternalProgram(){
String filePath = "C++" + File.separator + "cf.exe";
try {
System.out.println("Executing C++ exe...");
Process p = Runtime.getRuntime().exec("cmd /c start "+ filePath);
} catch (Exception e) {
e.printStackTrace();
}
}

Java exec stdin not flushing

For a school project I am trying to create a terminal in Java. The terminal works in the following way:
User types a command
Program grabs command and replaces <command> with the command in the string
/bin/bash -c "cd current/directory/; <command>; echo kjsfdjkadhlga; pwd
Program starts the process created via a ProcessBuilder object
Program spawns a thread that reads from stdout and stderr
Program continues looking for user input, and if the command is done running, then whatever the user entered is run as a command, otherwise it is fed to the currently running command as input.
As output is generated, program looks through the output for the kjsfdjkadhlga string so it knows when the user's command is done being run, and then grabs the remaining output and stores it as the current path that the user is at.
How this works/reasons for everything:
In order to avoid myself having to implement my own input parser to handle things like multiple commands on a line, IO redirection, and whatnot to work with the ProcessBuilder, I just essentially convert the command to a bash script and let bash execute it.
Since every process executes only a single command (or whatever it was given at the time of creation, which is a single user command in this case) then terminates, no process specific information is stored, such as the current working directory. To transfer that information, I call pwd after the user's command and then in the process of the next command, but before the user's command is run, I cd to that directory, effectively allowing the value of $PWD to persist between processes.
The Problem:
It all works well, except for when user interaction is required. If the user just types cat, it is supposed to wait for a line of user input, then print it, then wait for a line of user input, then print it, and repeat forever (I don't handle Crtl+C yet...). However, what actually happens is that the terminal waits for a line of user input, then prints it, then terminates without waiting for more input.
What I have tried:
Currently, I provide input to the command being run with:
BufferedWriter stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
If instead of calling close(), I call flush(), then cat ends up waiting for user input and not doing anything until I terminate my Terminal program, at which point it then prints everything the user had input.
It appears that the flush() function doesn't actually do anything. A Stack Overflow question mentioned using the raw OutputStream and calling write() instead of using a BufferedWriter. However, that has the same effect. In the OutputStream documentation for flush(), it states that "The flush method of OutputStream does nothing."
I have also tried using a BufferedOutputStream, but the documentation says that its flush function simply forces the buffered data to be written to the underlying OutputStream, which doesn't change the fact that the OutputStream is not flushing its stream.
This question seems to be the most promising, but I couldn't get it to work when implementing it. It may be because I am on Mac OS instead of Windows.
Does anybody know how to do this if keeping stdin open long enough to submit multiple lines of input is possible, or if I am going about it wrong?
Code
main()
Terminal terminal = new Terminal();
Scanner in = new Scanner(System.in);
while (in.hasNextLine())
{
String line = in.nextLine();
terminal.sendInput(line, terminal);
}
terminal.sendInput() called by main
// ProcessReaderDelegate implements functions called when receiving output on stdout, stderr, and when the process terminates.
public int sendInput(String text, ProcessReaderDelegate delegate)
{
if (processes.size() > 0)
{
processes.get(0).sendInput(text); // Is a ProcessReader object
return 1;
}
run(text, delegate); // runs the given text as the <command> text described above
return 2;
}
ProcessReader's sendInput() called by terminal.sendInput()
public boolean sendInput(String input)
{
try
{
// stdin and process are a instance fields
// tried this and doesn't seem to work (with either flush or close)
stdin = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
stdin.write(input);
stdin.newLine();
stdin.close();
// tried this and doesn't seem to work (with either flush or close)
//BufferedOutputStream os = new BufferedOutputStream(process.getOutputStream());
//os.write(input.getBytes());
//os.write("\n".getBytes());
//os.flush();
//os.close();
return true;
}
catch (IOException e)
{
System.out.println("ERROR: this should never happen: " + e.getMessage());
return false;
}
}
terminal.run() called by terminal.sendInput()
public void run(String command, ProcessReaderDelegate delegate)
{
// don't do anything with empty command since it screws up the command concatentaion later
if (command.equals(""))
{
delegate.receivedOutput(null, prompt);
return;
}
try
{
// create the command
List<String> list = new ArrayList<String>();
list.add(shellPath);
list.add(UNIX_BASED ? "-c" : "Command : ");
String cmd = (UNIX_BASED ? getUnixCommand(command) : getWindowsCommand(command));
list.add(cmd);
//System.out.println("command='" + list.get(0) + " " + list.get(1) + " " + list.get(2) + "'");
// create the process and run it
ProcessBuilder builder = new ProcessBuilder(list);
Process p = builder.start();
ProcessReader stdout = new ProcessReader(p, delegate, this);
new Thread(stdout).start();
processes.add(stdout);
}
catch (IOException e)
{
System.out.println(e.getMessage());
}
}
ProcessReader.run() executed in thread and reads stdout and stderr
public void run()
{
try
{
boolean hitend = false;
String buffer = "";
while (true)
{
int c;
String text;
// ======================================================
// read from stdout
// read the next character
c = stdout.read();
// build the string
while (c != -1) // while data available in the stream
{
buffer += (char)c;
c = stdout.read();
}
// send the string to the delegate
if ((!hitend) && (buffer.length() > 0))
{
// END_STRING is the "kjsfdjkadhlga" echoed after the command executes
int index = buffer.indexOf(END_STRING);
if (index >= 0)
{
hitend = true;
text = buffer.substring(0, index);
buffer = buffer.substring(index + END_STRING.length());
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
else
{
for (int i = END_STRING.length() - 1; i >= 0; i--)
{
index = buffer.indexOf(END_STRING.substring(0, i));
if (i == 0)
{
index = buffer.length();
}
if (index >= 0)
{
text = buffer.substring(0, index);
buffer = buffer.substring(index + i);
if (outputDelegate != null)
{
outputDelegate.receivedOutput(process, text);
}
}
}
}
}
// ======================================================
// read from stderr
// read the next character
c = stderr.read();
text = ""; // slow method; make faster with array
// build the string
while (c != -1) // while data available in the stream
{
text += (char)c;
c = stderr.read();
}
// send the string to the delegate
if ((text.length() > 0) && (outputDelegate != null))
{
outputDelegate.receivedError(process, text);
}
// ======================================================
// check if the process is done (and hence no more output)
boolean done = false;
try
{
int value = process.exitValue();
done = true; // if got to this point, then process is done
// read the ending environment variables
Map<String, String> env = new HashMap<String, String>();
String[] words = buffer.split(" ");
env.put(ENV_WORKING_DIR, words[0]);
if (envDelegate != null)
{
envDelegate.processTerminatedWithEnvironment(process, env);
}
// end the process
outputDelegate.processEnded(process);
stdout.close();
stderr.close();
break;
}
catch (Exception e) {System.out.println(e.getMessage());} // no exit value --> process not done
if (done) // just on the off chance that closing the streams crashes everything
{
break;
}
}
}
catch (IOException e)
{
System.out.println("ERROR: ProcessReader: " + e.getMessage());
}
}

Java program that calls external program behaving asynchronously?

All,
I originally had a shell script that called SQLLoader (Oracles data upload tool).
The problem was that SQLLoader takes a plain text password as input so I decided to build a Java application to call SQLLoader internally passing a decrypted password into the command string.
e.g.
sqlldr user/pass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv
So with my java wrapper it became this in my shell script
java -jar sqlloader.jar sqlloader.ctl mydata.csv
However a new problem developed when SQLLoader complained there was no file to load. After some head scratching it was discovered that a subsequent command in my shell script seemed to be executing while my java application was still running. Therefore it was behaving asynchronously.
The next command was moving the input file sqlloader was using before it could get a chance to use it. So I put a sleep command in of 20 seconds to give my java application time to run.
java -jar sqlloader.jar sqlloader.ctl mydata.csv
echo $?
sleep 20
if [ $? -ne 0 ]
then
echo "SQLLoader failed during execution, please check the log : "
mv mydata.csv
else
echo "SQLLoader successfully processed file : "
mv mydata.csv
fi
Does anyone know why unix is behaving this way, does Java execute my SQLLoader as a different user/ thread?
This is my java code:
Runtime Rt;
Process Prc;
Prc = Rt.exec("sqlldr user/decryptedpass#DBServer control=../sqlloader.ctl log=sqlloader.log data=mydata.csv);
system.exit(0);
I checked the Runtime Class for anything about it being Asynchronous but couldnt find anything
http://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html
Any theories or suggestions?
Thanks

Yes. If you look at Runtime.exec again it does specify that it will launch a new process in the specified environment (e.g. independently of the current "environment" or as you put it asynchronously). You should use ProcessBuilder to create a Process and then waitFor that Process to finish before calling System.exit - which certainly isn't mandatory. Something like this
public static void main(String[] args) {
// String command = "/usr/bin/sleep 5";
List<String> command = new ArrayList<String>();
command.add("c:/cygwin/bin/sleep");
command.add("5");
ProcessBuilder pb = new ProcessBuilder(command);
BufferedReader is = null;
try {
System.out.println("Starting command " + command);
Process p = pb.start();
int ret = p.waitFor();
is = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = is.readLine()) != null) {
System.out.println(line);
}
if (ret == 0) {
System.out.println("Command has completed.");
System.exit(ret);
} else {
System.out.println("Command completed with return code " + ret);
System.exit(ret);
}
} catch (Exception e) {
System.out.println("Caught Exception " + e.getMessage()
+ " running command " + command);
e.printStackTrace();
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
}
}
}
System.out.println("COMMAND FAILED");
System.exit(1);
}

You need to wait for process completion, you should also read all output (stdout and stderr) from the process you are starting.
If you call exit() after exec(), Java will do just that - exit immediatedly.
Here is an article that explains Runtime.exec pitfalls: http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4 (also consider the other pages).