I'm using the zeroturnaround zt-exec library (https://github.com/zeroturnaround/zt-exec), and I can't figure out why the output lines are not processed as they come in. It will wait until the script is done before spitting them all out at once. Is this something where they are being buffered? Seems like decreasing that would be pretty hacky.
Tried using 'start' instead of 'execute', and looked into buffer size but can't figure out what the issue is.
This answer seems to have what I'm looking for but I still get the issue
Read live output of process in java
Java code:
new ProcessExecutor().command("python", "printTest.py")
.redirectOutput(new LogOutputStream() {
#Override
protected void processLine(String line) {
logger.info("**** Line:" + line);
webSocketActorRef.tell(String.format("line:%s", line), self());
}
})
.execute();
python code:
print("**********INFO:MOCK SCRIPT STARTING **********")
print("STATUS: Starting the Mock simulation")
time.sleep(5)
print("STATUS: Message 1")
time.sleep(5)
print("STATUS: Message 2")
time.sleep(5)
print("STATUS: Message 3")
I am running Matlab2017 on windows 10.
I call a python script that runs some Speech Recognition task on cloud with something like this:
userAuthCode=1;% authentication code for user account to be run on cloud
cmd = ['C:\Python27\python.exe runASR.py userAuthCode];
system(cmd);
When the above command is called, the python script runs the input audio file on the ASR cloud engine, and as it runs, I can see Speech Recognition scores for the audio file from Python in the Matlab console.
I want to do the following:
(1) Execute multiple such commands in parallel. Lets say, I have 2 input audio files (each having different audio segments), and I would like to run the above command 2 times, but in parallel, using separate processes. I was able to create a code snippet that should be able to do this:
for i=1: 2
userAuthCode=i;
cmd = ['C:\Python27\python.exe runASR.py userAuthCode];
runtime = java.lang.Runtime.getRuntime();
pid(i) = runtime.exec(cmd);
end
for i=1:2
pid(i).waitFor();
% get exit status
rc(i) = pid(i).exitValue();
end
Now, when the above code is executed, I can see ASRE scores for data1 ,but not for data 2.
The exit status in variable rc, is 0,1, which confirms this.
The problem is I do not know the cause of the error, as nothing is printed in
Matlab. How can I get error message from Python captured in a java/Matlab
variable so that i could take a look?
The issue could be that multiple Calls to
ASRE in parallel (with different user accounts of course) may not
be supported but I won't know unless I can see the error.
(2) When I run a single command standalone, as mentioned at the start of the post, I am able to see Score messages for each audio segment being printed in the Matlab console, as they are obtained from Python. However, with multi-processing using java.lang.Runtime.getRuntime() and the associated code, no messages appears in the Matlab console. Is there a way to display those messages (I am assuming display might be asynchronous?)
Thanks
sedy
One approach is to use multiprocessing in Python. The results and any error messages can be returned in a list.
Example:
Assuming you have three audio files, your_function will run 3 times in parallel with error messages returned.
import subprocess
from multiprocessing import Pool, cpu_count
def multi_processor(function_name):
# Use a regex to make a list of full paths for audio files in /some/directory
# You could also just pass in a list of audio files as a parameter to this function
file_list = []
file_list = str(subprocess.check_output("find ./some/directory -type f -iname \"*a_string_in_your_aud_file_name*\" ",shell=True)).split('\\n')
file_list = sorted(file_list)
# Test, comment out two lines above and put 3 strings in the list so your_function should run three times with 3 processors in parallel
file_list.append("test1")
file_list.append("test2")
file_list.append("test3")
# Use max number of system processors - 1
pool = Pool(processes=cpu_count()-1)
pool.daemon = True
results = {}
# for every audio file in the file list, start a new process
for aud_file in file_list:
results[aud_file] = pool.apply_async(your_function, args=("arg1", "arg2"))
# Wait for all processes to finish before proceeding
pool.close()
pool.join()
# Results and any errors are returned
return {your_function: result.get() for your_function, result in results.items()}
def your_function(arg1, arg2):
try:
print("put your stuff in this function")
your_results = ""
return your_results
except Exception as e:
return str(e)
if __name__ == "__main__":
multi_processor("your_function")
I am new to ExpectJ Java programming. I downloaded jar's and able to do few send() and expect() methods. send() would fire a command on console and expect() would identify any prompt's so inputs can be provided. Expect only reads is there are prompts, and not other info. For example, if want to fire, spawn.send("ls") and get list of all file names and so certain action, is that possible?.
Is there way so I can read normal output of spawn.send("ls") for example, without expect which only captures prompts?
You can indeed capture the output stream:
It is one of the methods of the ExpectJ.Spawn class
I am also very new to Java, but I got the output, however, I am still struggling on getting the prompt recognized as I get extra control characters from Unix so do not trust what comes after the second System.out.println (the sh,expect part)
the output works fine, just set it in a variable if you want, or if you use swing, send it to a textarea with a listener.
BTW! if you know how to do the expect without these bloody control characters, 1m,34 [001 and so on, I welcome your input
import expectj.ExpectJ;
import expectj.Spawn;
public class myExpectinator {
public myExpectinator(){
}
public void connect(){
try {
ExpectJ ex = new ExpectJ(10);
Spawn sh = ex.spawn("10.10.10.10", 22, "name", "password");
System.out.println(sh.getCurrentStandardOutContents());
System.out.println(sh.getCurrentStandardErrContents());
sh.expect("~ $");
sh.send("ps\n");
System.out.println(sh.getCurrentStandardOutContents());
//sh.expectClose();
sh.stop();
}
catch(Exception e) {
System.out.println(e);
}
}
}
Is there an easy way to read output from subprocess unbuffered? I'm trying to call a C program from Java, but apparently it's stdout block-buffered when connected to pipe and line-buffered only when connected to console. I cannot modify C program.
Maybe there is a way to fool the program into thinking it is connected to console? Bonus points for a solution that works on linux as well.
It's not a great solution, but the runtime library probably doesn't buffer serial ports so if you're desperate enough you could use a null-modem emulator such as com0com or a derivative thereof.
Have you tried using environment variable BUF_1_=0?
OK -
1) "Line buffered input" is a common idiom for console-mode programs, dating back to the original Unix and serial-mode VTxx terminals.
2) You can read "raw, unbuffered" I/O (a keystroke at a time, instead of a line at a time), but the details are OS specific. Whatever you need to do on a specific OS, you almost certainly can do from Java.
3) It sounds like you want to be able to intercept an "arrow up" key or "page down" key, as it happens, on a Windows keyboard. Perhaps for playing a game, or for interacting with a console-mode user interface.
4) There are several options. One you might wish to consider is the "robot" API, used for testing:
http://www.java-tips.org/java-se-tips/java.awt/how-to-use-robot-class-in-java.html
If that's not sufficient, please give more details about exactly how you're trying to get your Java program to interact with the C program (and clarify if the platform is indeed Windows and a DOS prompt).
You could redirect the c output to a file and then tail the file from another thread in java using
org.apache.commons.io.input.Tailer
The best way to do it is using Threads to read the output
What I would do is: using a Callable (a Runnable will also work) where I give the input from the process (process.getInputStream ()) and the output where I want to store the output. The same should be done for the StdErr. The resulting output can be read using whatever you like.
public Object call () throws IOException {
int bytesRead;
byte[] b = new byte [ this.maxBlockSize ];
try {
while ( ( bytesRead = this.input.read ( b ) ) != -1 ) {
this.output.write ( b, 0, bytesRead );
}
} finally {
if ( this.output != null ) {
this.output.close ();
}
}
return null;
}
(This is a question my coworker posted elsewhere, but I thought I'd post it here to see if I could hit a different audience.)
Hello all,
I'm testing the possibility of writing a small java application the will use Psexec to kick off remote jobs. In the course of testing binding the stdin and stdout of a java program to psexec I came across an odd bug.
My test program is a basic echo program. It starts a thread to read from stdin and then pipes the read output directly back to stdout. When run on the local machine, not from psexec, it works beautifully. Exactly as it should.
However, when I call it from PsExec the first time the input is piped directly into stdout it is lost. What makes the bug really bizzare is that it is only the first time the input is piped directly into stdout that it is lost. If the input String is appended to another string it works fine. Either a String literal or a String variable. However, if the input String is sent directly to stdout it doesn't go through. The second time it is sent to stdout it goes through fine - and everytime there after.
I'm at a complete loss as to what's going on here. I've tried to test for every possible bug I can think of. I'm out of ideas. Did I miss one or is this just something inside psexec?
Here is the code in question, it's in three classes (one of which implements an interface which is a single function interace).
The Main class:
public class Main {
public static void main(String[] args) {
System.out.println("Starting up.");
CReader input = new CReader(new BufferedReader(
new InputStreamReader(System.in)));
CEcho echo = new CEcho();
input.addInputStreamListener(echo);
input.start();
System.out.println("Successfully started up. Awaiting input.");
}
}
The CReader class which is the thread that reads from stdin:
public class CReader extends Thread {
private ArrayList<InputStreamListener> listeners =
new ArrayList<InputStreamListener>();
private boolean exit = false;
private Reader in;
public CReader(Reader in) {
this.in = in;
}
public void addInputStreamListener(InputStreamListener listener) {
listeners.add(listener);
}
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(input);
}
#Override
public void run() {
StringBuilder sb = new StringBuilder();
int current = 0, last = 0;
while (!exit) {
try {
current = in.read();
}
catch (IOException e) {
System.out.println("Encountered IOException.");
}
if (current == -1) {
break;
}
else if (current == (int) '\r') {
if(sb.toString().length() == 0) {
// Extra \r, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else if(current == (int) '\n') {
if(sb.toString().length() == 0) {
// Extra \n, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else {
System.out.println("Recieved character: " + (char)current);
sb.append((char) current);
last = current;
}
}
}
}
The CEcho class, which is the class that pipes it back to stdout:
public class CEcho implements InputStreamListener {
public void inputRecieved(String input) {
System.out.println("\n\nSTART INPUT RECIEVED");
System.out.println("The input that has been recieved is: "+input);
System.out.println("It is a String, that has been copied from a " +
"StringBuilder's toString().");
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
System.out.println("Finished example outputs of input: "+input);
System.out.println("END INPUT RECIEVED\n\n");
}
}
And finally, here is the program output:
>psexec \\remotecomputer "C:\Program Files\Java\jre1.6.0_05\bin\java.exe" -jar "C:\Documents and Settings\testProram.jar"
PsExec v1.96 - Execute processes remotely
Copyright (C) 2001-2009 Mark Russinovich
Sysinternals - www.sysinternals.com
Starting up.
Successfully started up. Awaiting input.
Test
Recieved character: T
Recieved character: e
Recieved character: s
Recieved character: t
Input string has made it to fireInputRecieved: Test
START INPUT RECIEVED
The input that has been recieved is: Test
It is a String, that has been copied from a StringBuilder's toString().
Outputting it cleanly to standard out:
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
END INPUT RECIEVED
have you tried redirecting the output into a file ( java... >c:\output.txt )? this way you could doublecheck if everything is going into stdout and maybe just getting eaten by psexec
PsExec is eating the output. Next interesting thing might be where it's eating the output. You could check this by getting a copy of Wireshark and checking whether the output in question is traversing the network or not. If it's not, then it's being eaten on the remote side. If it is, it's being eaten locally.
Not that I'm really sure where to go from there, but collecting more information certainly seems like a good path to be following...
I was having the same issue and tried multiple combinations of redirects.
This is what worked:
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(Redirect.PIPE);
processBuilder.redirectInput(Redirect.INHERIT);
final Process process = processBuilder.start();
// Using Apache Commons IOUtils to get output in String
StringWriter writer = new StringWriter();
IOUtils.copy(process.getInputStream(), writer, StandardCharsets.UTF_8);
String result = writer.toString();
logger.info(result);
final int exitStatus = process.waitFor();
The Redirect.INHERIT for processBuilder.redirectInput got me the missing remote command output.
Is System.out not configured for autoflush? After the first print try System.out.flush() and see if the first line appears without more lines being printed.
(oh yeah, seriously, it is "RECEIVED", not "RECIEVED".)
OK, I've been thinking about this over the weekend and I since you are jumping from machine to machine I wonder if maybe there is a CharSet issue? Maybe it is eating the string the first time and dealing with a different code page or character set issue? Java is 16bit characters normally and windows is either 8bit with code pages or utf-8 these days.
Any chance the local and remote machines have different default character sets? If you are sending localized data over the net it might misbehave.
What I see when running psexec is that it spawns a child window to do the work but doesnt return that program's output to it's console window. I would suggest using WMI or some form of windows process API framework to gain a level of control you appear to lack with psexec. Surely java has an equivalent to .Net's System.Diagnotics.Process class.
Maybe you could try passing a copy of input to your listeners:
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
String inputCopy = new String(input);
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(inputCopy);
}
I had similar problems with listeners where a passed variable would end up empty unless I did pass an explicit copy of it.
I don't necessarily have an answer, but some comments may prove helpful.
The "pass a copy" idea shouldn't matter, since your output successfully prints the string twice before the failure, then succeeds again afterward.
auto-flush shouldn't matter either, as you've already mentioned
Niko's suggestion has some merit, for diagnostic purposes. Mixed with Mark's suggestion, it makes me wonder if there aren't some invisible control characters getting involved somewhere. What if you printed the characters byte values as a diagnostic step?
You know that the value is "Test" (at least in the output you gave us). What happens if you pass "Test" directly to the failing printLn statement?
In situations like this, you want to gain as much information as possible. Insert breakpoints and analyze characters. Send the bytes to files and open them in hex editors. Do whatever you can to trace things as accurately and as precisely as possible.
Come up with weird test scenarios and try them, even if they shouldn't possibly help. You never know what good idea you might have while analyzing the results of the hopeless idea.
I'd guess that there is a bogus byte in there prefacing the T. According to JavaDocs, an InputStreamReader will read one or more bytes, and decode them into characters.
You could have an escape sequence or spurious byte in there, masquerading as a multibyte character.
Quick check - see if "current" is ever > 128 or < 33.
What if you used a CharArrayReader to get individual bytes, without any charset translation?
The theory is that during the first attempt to output the String using println, it's sending an escape character of some sort, eating the rest of the string. During later prints, either Java or the network pipe are handling or removing it, since it previously got that escape sequence, perhaps changing the handling in some way.
As an unrelated nit, sb.toString() returns a new String, so it's unnecessary to call "new String(sb.toString())"
Same issue here, I'm going through this post again and again these days, hoping I can find some solution. Then I decide I should give up psexec and find some alternative. So this is the thing: PAExec. Works perfect for getting command output.
How are you executing PsExec? My suspicion is that this is some code within PsExec which is actually doing echo suppression, possibly for the purposes of protecting a password. One way to test this hypothesis would be to change this code:
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
to this:
System.out.println("Outputting it cleanly to standard out: ");
System.out.print(' ');
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
...thereby causing the output to be (if I'm right):
Outputting it cleanly to standard out:
Test
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
In particular, it's noticeable that the apparently-suppressed line is the first line which consists solely of Test - which is exactly the text you've just sent to the remote system. This sounds like PsExec attempting to suppress a remote system which is echoing its input in addition to producing its own output.
Is the password of the user on the remote machine perhaps Test? Are you using PsExec's -p parameter? Are you specifying -i?
I am dealing with this same issue and I am wondering if it has to do with how the cmd window and pipes in windows work while you don't have a true windowed session. The suppressed output happens when any new process is spawned. You would think that if you spawn a process that the stdout/stderr/stdin would be inherited from the process that spawned it; after all that is what happens if you spawn the process from a normal cmd window and the output from the new process is piped back to your own console. However if somewhere in the inheritance of the pipes something were to go wrong, like say not passing a WINDOW.GUI object because there is no physical window, windows doesn't let the stdin/stdout/stdin to be inherited. Can any one do some investigation or open a windows support ticket for this?
Seems no easy solution. My work-around in a recent project is using paexec.exe product. It captures output/error easily in JAVA(java-8), but hangs up upon completion of the remote command execution. When running this inside a server on the hosted machine, I have to spurn a new child JVM process to run paexec.exe and force kill it via its PID upon completion in order to release all the resources.
If anyone has better solution, please post it.