I written a program in C++ where multithreading is used. Call this program using a bash script or something like that is no problem, everything is working fine. Now I want to call this program in Java. This is the Java code sample I tried:
Process v2j = new ProcessBuilder("./src/main/cpp/vcd_converter",
"-vcd", vcdfilePath, "-out", logfilePath).start();
InputStream is = v2j.getInputStream();
InputStream es = v2j.getErrorStream();
for(int i = 0; i < is.available(); i++) {
System.out.println("Input: " + br.readLine());
}
for(int i = 0; i < es.available(); i++) {
System.out.println("Error: " + ebr.readLine());
}
v2j.waitFor();
So if I run this code, following will happen: It seems like the waitFor() instruction will not realy wait for "all" threads, maybe it will wait only for one. However, the java program executed with a error because it can't find files which are generated by the C++ program. So the waitFor() don't wait until my c++ program is finished and has generated the files.
The output is really weird too. I only get one output from std::cout. Here's an example:
std::cout << out_folder_path << "\n" << vcd_filename << "\n";
std::cout << std::flush;
This will print only the out_folder_path output. A problem is: In the section of the std::cout I don't even use multi-threading. All other std::cout in the program are don't printed too.
So my questions: How can I wait for really ALL threads? And how can I get the real output of my program?
Your program doesn't get all output from the C++ program because it doesn't read until the input streams are depleted. It jumps to waitFor if the C++ program has not produced any output when your program reaches is.available().
Here's an alternative approach doing reading until the input streams are closed. It uses an InputStreamReader and a BufferedReader and calls the BufferedReader's readline() until the input stream closes.
Disclaimer. I haven't written any java programs before so I'm not sure I've gotten everything correct. The general idea should work though.
var v2j = new ProcessBuilder("./src/main/cpp/vcd_converter", "-vcd",
vcdfilePath, "-out", logfilePath).start();
InputStream is = v2j.getInputStream();
try(var reader = new BufferedReader(new InputStreamReader(is))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println("Input: " + line);
}
}
InputStream es = v2j.getErrorStream();
try(var reader = new BufferedReader(new InputStreamReader(es))) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println("Error: " + line);
}
}
v2j.waitFor();
Related
I'm using proc_open in php to call java application, pass it text to be processed and read output text. Java execution time is quite long and I found the reason for that is reading input takes most of the time. I'm not sure whether it's php's or java's fault.
My PHP code:
$process_cmd = "java -Dfile.encoding=UTF-8 -jar test.jar";
$env = NULL;
$options = ["bypass_shell" => true];
$cwd = NULL;
$descriptorspec = [
0 => ["pipe", "r"], //stdin is a pipe that the child will read from
1 => ["pipe", "w"], //stdout is a pipe that the child will write to
2 => ["file", "java.error", "a"]
];
$process = proc_open($process_cmd, $descriptorspec, $pipes, $cwd, $env, $options);
if (is_resource($process)) {
//feeding text to java
fwrite($pipes[0], $input);
fclose($pipes[0]);
//reading output text from java
$output = stream_get_contents($pipes[1]);
fclose($pipes[1]);
$return_value = proc_close($process);
}
My java code:
public static void main(String[] args) throws Exception {
long start;
long end;
start = System.currentTimeMillis();
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String in;
String input = "";
br = new BufferedReader(new InputStreamReader(System.in));
while ((in = br.readLine()) != null) {
input += in + "\n";
}
end = System.currentTimeMillis();
log("Input: " + Long.toString(end - start) + " ms");
start = System.currentTimeMillis();
org.jsoup.nodes.Document doc = Jsoup.parse(input);
end = System.currentTimeMillis();
log("Parser: " + Long.toString(end - start) + " ms");
start = System.currentTimeMillis();
System.out.print(doc);
end = System.currentTimeMillis();
log("Output: " + Long.toString(end - start) + " ms");
}
I'm passing to java html file of 3800 lines (~200KB in size as a standalone file). These are broken down execution times in the log file:
Input: 1169 ms
Parser: 98 ms
Output: 12 ms
My question is this: why does input take 100 times longer than output? Is there a way to make it faster?
Inspect your read block in the Java program: Try to use a StringBuilder to concat the data (instead of using += on a String):
String in;
StringBuilder input = new StringBulider();
br = new BufferedReader(new InputStreamReader(System.in));
while ((in = br.readLine()) != null) {
input.append(in + "\n");
}
Details are covered here: Why using StringBuilder explicitly
Generally speaking, to make it faster, consider using an application server (or a simple socket based server), to have a permanently running JVM. There is always some overhead when you start a JVM, on top of it the JIT needs some time as well to optimize your code. This effort is lost, after the the JVM exits.
As for the PHP program: Try to feed the Java program from the shell, just use cat to pipe the data (on a UNIX system like Linux). As an alternative, rewrite your Java program to accept a command line parameter for the file as well. Then you can judge, if your PHP code pipes the data fast enough.
As for the Java program: If you do performance analysis, consider the recommendations in How do I write a correct micro-benchmark in Java
I'm trying to get input from the console of a .exe process started by a Java script. Nothing appears in the console window, and nothing is read by the program until the process is terminated.
blServ = new ProcessBuilder(blPath + "Blockland.exe", "ptlaaxobimwroe", "-dedicated", "-port " + port, "-profilepath " + blPath.substring(0, blPath.length() - 1)).start();
System.out.println("Attempting to start server...\n" + blPath);
consoleIn = new BufferedReader(new InputStreamReader(blServ.getInputStream()));
'blServ' is a Process object. And yes, the program is starting successfully.
public void blStreamConsole() //called once every 500 milliseconds
{
String lineStr = "";
String line = "";
int lines = 0;
try
{
if (consoleIn != null)
{
while ((line = consoleIn.readLine()) != null)
{
//if (!line.equals("%"));
//{
lineStr += line + wordSym;
lines++;
//}
}
}
}
catch (IOException e)
{
netOut.println("notify" + wordSym + "ERROR: An I/O exception occured when trying to get data from the remote console. Some lines may not be displayed.");
}
if (!lineStr.equals("") && !(lineStr == null))
netOut.println("streamconsole" + wordSym + lines + wordSym + lineStr);
}
Basically, this method sees if there is more input waiting in the consoleIn object, and if there is, it appends every line it has to another string, and that other string is sent to a client. Unfortunately, it is all sent in one big chunk right when Blockland.exe is closed. Sorry about the indenting issues. The Stackoverflow editor re-arranged all of the code.
It seems to me that there are two possibilities here:
readLine blocks, waiting for input (and doesn't return null as you expect). You may be able to fix it by not using BufferedReader and instead using the InputStream
The output stream doesn't flush until all the input has been written. Try putting a flush there:
Also note that if lineStr is null, you'll get a NullPointerException as your code currently is (you need to swap your conditions), but it can't even be null.
if (!lineStr.isEmpty())
{
netOut.println("streamconsole" + wordSym + lines + wordSym + lineStr);
netOut.flush();
}
while ((line = consoleIn.readLine()) != null){
lineStr += line + wordSym;
lines++;
}
The problem with this piece of code is that it will keep running until the program exits. It will append every single line to lineStr until the program exits (when console.readLine() is null). The whole lineStr is then printed afterwards, containing the whole console.
If you want to continuously print the output, you will need to print it immediatly:
while ((line = consoleIn.readLine()) != null){
netOut.println(line);
}
You can run this in one separate thread, and it will keep outputting the console to the output stream until the program exits.
I have the following class file. This start a command prompt and print the responses. weird thing is after the first print i.e. dir the subsequent doesn't print. Please advice.
import java.io.*;
public class JavaApplication14 {
static Process p;
public static void main(String[] args) {
try {
String line;
p = Runtime.getRuntime().exec("cmd.exe");
OutputStream stdin = p.getOutputStream();
InputStream stderr = p.getErrorStream();
InputStream stdout = p.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(stdout));
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stdin));
String input = "dir";
input += "\n";
writer.write(input);
writer.flush();
while ((line = reader.readLine()) != null) {
System.out.println("Stdout: " + line);
}
input = "cd..";
input += "\n";
writer.write(input);
writer.flush();
input = "dir";
input += "\n";
writer.write(input);
writer.close();
while ((line = reader.readLine()) != null) {
System.out.println("Stdout: " + line);
}
} catch (IOException ex) {
Logger.getLogger(JavaApplication14.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
Your (first) while() loop never terminates:
while ((line = reader.readLine()) != null) {
System.out.println("Stdout: " + line);
}
readLine() returns null when the stream is closed, but since your sub process is still running, the stream never gets closed.
To solve this, you can either move the reading part into a separate thread (which requires additional synchronization), or a simpler solution would be to see if a specific line contents is read, for example if the command line prompt was printed by cmd.exe:
while ( !(line = reader.readLine()).startsWith("C:\\") ) {
System.out.println("Stdout: " + line);
}
This should work for your particular use case, and might be sufficient to do some learning - for real applications, you might want to have a look into the Apache Commons Exec project.
You are trying to do fundamentally asynchronous work from just one thread, using synchronous I/O operations. Your approach is bound to fail.
Specifically, readLine() blocks until there is a full line to be read, or until the underlying stream is closed.
You'll have to write quite a bit more code, involving threads, to make this work. This is a pain point in Java.
You could also use the ProcessBuilder, especially its redirectOutput method with the argument value INHERIT, to make the subprocess inherit your main process's stdout. In this scenario you won't have the opportunity to analyze the subprocess's output in Java.
In a java program, I am generating an sh script for use on a centOS machine, which will use sox and lame to decode an MP3 audio file, then apply some gain to the file respectively. Im having some issues getting the Process.waitFor() method to do anything other than hang indefinitely. Here is the code:
try
{
// TODO code application logic here
String reviewPath = "/SomeDirectory/";
String fileName = "FileName";
String extension = ".mp3";
StringBuilder sb = new StringBuilder();
sb.append("#!/bin/bash\n");
sb.append("cd " + reviewPath + "\n");
sb.append("lame --decode " + fileName + extension + "\n");
File script = new File(reviewPath + fileName + ".sh");
script.createNewFile();
script.setExecutable(true);
FileWriter writer = new FileWriter(script);
writer.write(sb.toString());
writer.close();
Process p = Runtime.getRuntime().exec(script.getAbsolutePath());
String line;
BufferedReader bri = new BufferedReader
(new InputStreamReader(p.getInputStream()));
BufferedReader bre = new BufferedReader
(new InputStreamReader(p.getErrorStream()));
while ((line = bri.readLine()) != null) {
System.out.println(line);
}
bri.close();
while ((line = bre.readLine()) != null) {
System.out.println(line);
}
bre.close();
p.waitFor();
System.out.println("Done.");
}
catch (Exception e)
{
System.out.println(e.getMessage());
}
The odd part is that when I run the .sh file it generates by hand, it runs and exits nicely, but when I execute it from a process object in java, it never exits. The exitValue of the process is always "Process has not exited". Ive tried adding set -e to the script, and exit to the end of the script. Short of using the kill command (which I dont really think I can do here) Im at a loss as to what is going on here. Any suggestions?
Add something like while(p.getInputStream().read() != -1); after starting the process. The buffer will get filled and the process will stop waiting for something (in this case, your program) to read from it to free up space.
I figured it out! The problem here was indeed that the output streams needed to be flushed for the application to exit, but simply reading from the streams is not enough. I used Suresh Koya's suggestion and used the processBuilder api, and redirected the error stream on the process before starting it, and read from the streams. This fixed the issues I was having :D
First, this is my code :
import java.io.*;
import java.util.Date;
import com.banctecmtl.ca.vlp.shared.exceptions.*;
public class PowershellTest implements Runnable {
public static final String PATH_TO_SCRIPT = "C:\\Scripts\\ScriptTest.ps1";
public static final String SERVER_IP = "XX.XX.XX.XXX";
public static final String MACHINE_TO_MOD = "MachineTest";
/**
* #param args
* #throws OperationException
*/
public static void main(String[] args) throws OperationException {
new PowershellTest().run();
}
public PowershellTest(){}
#Override
public synchronized void run() {
String input = "";
String error = "";
boolean isHanging = false;
try {
Runtime runtime = Runtime.getRuntime();
Process proc = runtime.exec("powershell -file " + PATH_TO_SCRIPT +" "+ SERVER_IP +" "+ MACHINE_TO_MOD);
proc.getOutputStream().close();
InputStream inputstream = proc.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
proc.waitFor();
String line;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
input += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
inputstream = proc.getErrorStream();
inputstreamreader = new InputStreamReader(inputstream);
bufferedreader = new BufferedReader(inputstreamreader);
isHanging = false;
while (!isHanging && (line = bufferedreader.readLine()) != null) {
error += (line + "\n");
Date date = new Date();
while(!bufferedreader.ready()){
this.wait(1000);
//if its been more then 1 minute since a line has been read, its hanging.
if(new Date().getTime() - date.getTime() >= 60000){
isHanging = true;
break;
}
}
}
inputstream.close();
proc.destroy();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
} catch (InterruptedException e) {
//throw new OperationException("Script thread problem.",e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
I'm currently trying to run a powershell script that will start/stop a vm (VMWARE) on a remote server. The script work from command line and so does this code. The thing is, I hate how I have to use a thread (and make it wait for the script to respond, as explained further) for such a job. I had to do it because both BufferedReader.readline() and proc.waitFor() hang forever.
The script, when ran from cmd, is long to execute. it stall for 30 sec to 1 min from validating authentification with the server and executing the actual script. From what I saw from debugging, the readline hang when it start receiving those delays from the script.
I'm also pretty sure it's not a memory problem since I never had any OOM error in any debugging session.
Now I understand that Process.waitFor() requires me to flush the buffer from both the error stream and the regular stream to work and so that's mainly why I don't use it (I need the output to manage VM specific errors, certificates issues, etc.).
I would like to know if someone could explain to me why it hangs and if there is a way to just use the typical readline() without having it to hang so hard. Even if the script should have ended since a while, it still hang (I tried to run both the java application and a cmd command using the exact same thing I use in the java application at the same time, left it runingfor 1h, nothing worked). It is not just stuck in the while loop, the readline() is where the hanging is.
Also this is a test version, nowhere close to the final code, so please spare me the : this should be a constant, this is useless, etc. I will clean the code later. Also the IP is not XX.XX.XX.XXX in my code, obviously.
Either explanation or suggestion on how to fix would be greatly appreciated.
Ho btw here is the script I currently use :
Add-PSSnapin vmware.vimautomation.core
Connect-VIServer -server $args[0]
Start-VM -VM "MachineTest"
If you need more details I will try to give as much as I can.
Thanks in advance for your help!
EDIT : I also previously tested the code with a less demanding script, which job was to get the content of a file and print it. Since no waiting was needed to get the information, the readline() worked well. I'm thus fairly certain that the problem reside on the wait time coming from the script execution.
Also, forgive my errors, English is not my main language.
Thanks in advance for your help!
EDIT2 : Since I cannot answer to my own Question :
Here is my "final" code, after using threads :
import java.io.*;
public class PowershellTest implements Runnable {
public InputStream is;
public PowershellTest(InputStream newIs){
this.is = newIs;
}
#Override
public synchronized void run() {
String input = "";
String error = "";
try {
InputStreamReader inputstreamreader = new InputStreamReader(is);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String line;
while ((line = bufferedreader.readLine()) != null) {
input += (line + "\n");
}
is.close();
} catch (IOException e) {
//throw new OperationException("File IO problem.", e);
}
System.out.println("Error : " + error + "\nInput : " + input);
}
}
And the main simply create and start 2 thread (PowerShellTest instances), 1 with the errorStream and 1 with the inputStream.
I believe I made a dumb error when I first coded the app and fixed it somehow as I reworked the code over and over. It still take a good 5-6 mins to run, which is somehow similar if not longer than my previous code (which is logical since the errorStream and inputStream get their information sequentially in my case).
Anyway, thanks to all your answer and especially Miserable Variable for his hint on threading.
First, don't call waitFor() until after you've finished reading the streams. I would highly recommend you look at ProcessBuilder instead of simply using Runtime.exec, and split the command up yourself rather than relying on Java to do it for you:
ProcessBuilder pb = new ProcessBuilder("powershell", "-file", PATH_TO_SCRIPT,
SERVER_IP, MACHINE_TO_MOD);
pb.redirectErrorStream(true); // merge stdout and stderr
Process proc = pb.start();
redirectErrorStream merges the error output into the normal output, so you only have to read proc.getInputStream(). You should then be able to just read that stream until EOF, then call proc.waitFor().
You are currently waiting to complete reading from inputStream before starting to read from errorStream. If the process writes to its stderr before stdout maybe you are getting into a deadlock situation.
Try reading from both streams from concurrently running threads. While you are at it, also remove proc.getOutputStream().close();. It shouldn't affect the behavior, but it is not required either.