(This is a question my coworker posted elsewhere, but I thought I'd post it here to see if I could hit a different audience.)
Hello all,
I'm testing the possibility of writing a small java application the will use Psexec to kick off remote jobs. In the course of testing binding the stdin and stdout of a java program to psexec I came across an odd bug.
My test program is a basic echo program. It starts a thread to read from stdin and then pipes the read output directly back to stdout. When run on the local machine, not from psexec, it works beautifully. Exactly as it should.
However, when I call it from PsExec the first time the input is piped directly into stdout it is lost. What makes the bug really bizzare is that it is only the first time the input is piped directly into stdout that it is lost. If the input String is appended to another string it works fine. Either a String literal or a String variable. However, if the input String is sent directly to stdout it doesn't go through. The second time it is sent to stdout it goes through fine - and everytime there after.
I'm at a complete loss as to what's going on here. I've tried to test for every possible bug I can think of. I'm out of ideas. Did I miss one or is this just something inside psexec?
Here is the code in question, it's in three classes (one of which implements an interface which is a single function interace).
The Main class:
public class Main {
public static void main(String[] args) {
System.out.println("Starting up.");
CReader input = new CReader(new BufferedReader(
new InputStreamReader(System.in)));
CEcho echo = new CEcho();
input.addInputStreamListener(echo);
input.start();
System.out.println("Successfully started up. Awaiting input.");
}
}
The CReader class which is the thread that reads from stdin:
public class CReader extends Thread {
private ArrayList<InputStreamListener> listeners =
new ArrayList<InputStreamListener>();
private boolean exit = false;
private Reader in;
public CReader(Reader in) {
this.in = in;
}
public void addInputStreamListener(InputStreamListener listener) {
listeners.add(listener);
}
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(input);
}
#Override
public void run() {
StringBuilder sb = new StringBuilder();
int current = 0, last = 0;
while (!exit) {
try {
current = in.read();
}
catch (IOException e) {
System.out.println("Encountered IOException.");
}
if (current == -1) {
break;
}
else if (current == (int) '\r') {
if(sb.toString().length() == 0) {
// Extra \r, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else if(current == (int) '\n') {
if(sb.toString().length() == 0) {
// Extra \n, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else {
System.out.println("Recieved character: " + (char)current);
sb.append((char) current);
last = current;
}
}
}
}
The CEcho class, which is the class that pipes it back to stdout:
public class CEcho implements InputStreamListener {
public void inputRecieved(String input) {
System.out.println("\n\nSTART INPUT RECIEVED");
System.out.println("The input that has been recieved is: "+input);
System.out.println("It is a String, that has been copied from a " +
"StringBuilder's toString().");
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
System.out.println("Finished example outputs of input: "+input);
System.out.println("END INPUT RECIEVED\n\n");
}
}
And finally, here is the program output:
>psexec \\remotecomputer "C:\Program Files\Java\jre1.6.0_05\bin\java.exe" -jar "C:\Documents and Settings\testProram.jar"
PsExec v1.96 - Execute processes remotely
Copyright (C) 2001-2009 Mark Russinovich
Sysinternals - www.sysinternals.com
Starting up.
Successfully started up. Awaiting input.
Test
Recieved character: T
Recieved character: e
Recieved character: s
Recieved character: t
Input string has made it to fireInputRecieved: Test
START INPUT RECIEVED
The input that has been recieved is: Test
It is a String, that has been copied from a StringBuilder's toString().
Outputting it cleanly to standard out:
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
END INPUT RECIEVED
have you tried redirecting the output into a file ( java... >c:\output.txt )? this way you could doublecheck if everything is going into stdout and maybe just getting eaten by psexec
PsExec is eating the output. Next interesting thing might be where it's eating the output. You could check this by getting a copy of Wireshark and checking whether the output in question is traversing the network or not. If it's not, then it's being eaten on the remote side. If it is, it's being eaten locally.
Not that I'm really sure where to go from there, but collecting more information certainly seems like a good path to be following...
I was having the same issue and tried multiple combinations of redirects.
This is what worked:
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(Redirect.PIPE);
processBuilder.redirectInput(Redirect.INHERIT);
final Process process = processBuilder.start();
// Using Apache Commons IOUtils to get output in String
StringWriter writer = new StringWriter();
IOUtils.copy(process.getInputStream(), writer, StandardCharsets.UTF_8);
String result = writer.toString();
logger.info(result);
final int exitStatus = process.waitFor();
The Redirect.INHERIT for processBuilder.redirectInput got me the missing remote command output.
Is System.out not configured for autoflush? After the first print try System.out.flush() and see if the first line appears without more lines being printed.
(oh yeah, seriously, it is "RECEIVED", not "RECIEVED".)
OK, I've been thinking about this over the weekend and I since you are jumping from machine to machine I wonder if maybe there is a CharSet issue? Maybe it is eating the string the first time and dealing with a different code page or character set issue? Java is 16bit characters normally and windows is either 8bit with code pages or utf-8 these days.
Any chance the local and remote machines have different default character sets? If you are sending localized data over the net it might misbehave.
What I see when running psexec is that it spawns a child window to do the work but doesnt return that program's output to it's console window. I would suggest using WMI or some form of windows process API framework to gain a level of control you appear to lack with psexec. Surely java has an equivalent to .Net's System.Diagnotics.Process class.
Maybe you could try passing a copy of input to your listeners:
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
String inputCopy = new String(input);
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(inputCopy);
}
I had similar problems with listeners where a passed variable would end up empty unless I did pass an explicit copy of it.
I don't necessarily have an answer, but some comments may prove helpful.
The "pass a copy" idea shouldn't matter, since your output successfully prints the string twice before the failure, then succeeds again afterward.
auto-flush shouldn't matter either, as you've already mentioned
Niko's suggestion has some merit, for diagnostic purposes. Mixed with Mark's suggestion, it makes me wonder if there aren't some invisible control characters getting involved somewhere. What if you printed the characters byte values as a diagnostic step?
You know that the value is "Test" (at least in the output you gave us). What happens if you pass "Test" directly to the failing printLn statement?
In situations like this, you want to gain as much information as possible. Insert breakpoints and analyze characters. Send the bytes to files and open them in hex editors. Do whatever you can to trace things as accurately and as precisely as possible.
Come up with weird test scenarios and try them, even if they shouldn't possibly help. You never know what good idea you might have while analyzing the results of the hopeless idea.
I'd guess that there is a bogus byte in there prefacing the T. According to JavaDocs, an InputStreamReader will read one or more bytes, and decode them into characters.
You could have an escape sequence or spurious byte in there, masquerading as a multibyte character.
Quick check - see if "current" is ever > 128 or < 33.
What if you used a CharArrayReader to get individual bytes, without any charset translation?
The theory is that during the first attempt to output the String using println, it's sending an escape character of some sort, eating the rest of the string. During later prints, either Java or the network pipe are handling or removing it, since it previously got that escape sequence, perhaps changing the handling in some way.
As an unrelated nit, sb.toString() returns a new String, so it's unnecessary to call "new String(sb.toString())"
Same issue here, I'm going through this post again and again these days, hoping I can find some solution. Then I decide I should give up psexec and find some alternative. So this is the thing: PAExec. Works perfect for getting command output.
How are you executing PsExec? My suspicion is that this is some code within PsExec which is actually doing echo suppression, possibly for the purposes of protecting a password. One way to test this hypothesis would be to change this code:
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
to this:
System.out.println("Outputting it cleanly to standard out: ");
System.out.print(' ');
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
...thereby causing the output to be (if I'm right):
Outputting it cleanly to standard out:
Test
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
In particular, it's noticeable that the apparently-suppressed line is the first line which consists solely of Test - which is exactly the text you've just sent to the remote system. This sounds like PsExec attempting to suppress a remote system which is echoing its input in addition to producing its own output.
Is the password of the user on the remote machine perhaps Test? Are you using PsExec's -p parameter? Are you specifying -i?
I am dealing with this same issue and I am wondering if it has to do with how the cmd window and pipes in windows work while you don't have a true windowed session. The suppressed output happens when any new process is spawned. You would think that if you spawn a process that the stdout/stderr/stdin would be inherited from the process that spawned it; after all that is what happens if you spawn the process from a normal cmd window and the output from the new process is piped back to your own console. However if somewhere in the inheritance of the pipes something were to go wrong, like say not passing a WINDOW.GUI object because there is no physical window, windows doesn't let the stdin/stdout/stdin to be inherited. Can any one do some investigation or open a windows support ticket for this?
Seems no easy solution. My work-around in a recent project is using paexec.exe product. It captures output/error easily in JAVA(java-8), but hangs up upon completion of the remote command execution. When running this inside a server on the hosted machine, I have to spurn a new child JVM process to run paexec.exe and force kill it via its PID upon completion in order to release all the resources.
If anyone has better solution, please post it.
Related
I am a novice at coding but cannot understand why it runs fine on my machine, but when I upload my code I get a "NoSuchElementException" on line 19, "String command = keyboar.next();" I understand it has to do something with closing the scanner but I cannot figure out any other way to work it so it loops the print screen and input. Especially since it works fine when I run it on my machine.
Any insight is much appreciated here
import java.util.Scanner;
public class example1
{
public static void main(String[] args)
{
System.out.println("Enter an ending value");
Scanner keyboard = new Scanner(System.in);
int input;
input = keyboard.nextInt();
while(true){
System.out.println("Count up or down?");
String command = keyboard.next();
if (command.equalsIgnoreCase("up")) {
int one = 1;
int ten = 11;
int hund = 101;
while (one <= input) {
System.out.printf("%5d %4d %4d\n", one, ten, hund);
one++;
ten++;
hund++;
}
}
if (command.equalsIgnoreCase("down")) {
int neg = -input;
int one = -1;
int ten = 9;
int hund = 99;
while (one >= neg) {
System.out.printf("%5d %4d %4d\n", one, ten, hund);
one--;
ten--;
hund--;
}
}
}
}
}
You've created a scanner that reads from System.in. You don't close it anywhere, so I'm not sure why you wrote in your question that you feel it has something to do with that.
System.in does not represent the keyboard. It represents the java process's 'standard in' stream. If you just run java -jar foo.jar or whatnot on the command line (which is its own process, called the 'shell' - it'll be cmd.exe on windows, perhaps /bin/bash on linux. It's just an application, nothing special) - then that shell will decide that you intended to hook up the keyboard (technically, the 'terminal', which is usually virtualized, for example if you use ssh or other tools to remote your way onto another server, usually a physical keyboard isn't even connected to those things!).
But that's just because you started that process in a command line without explicitly specifying. If you double-click a jar on linux you probably won't get any terminal and nothing will be hooked up to standard in. If instead you start java -jar yourapp.jar <somefile.txt then bash will open the somefile.txt and set that up as the standard in.
The keyboard never runs out - you won't get a NoSuchElementException there.
But files run out. Given that you get this error when you 'upload' your application, clearly, something has been hooked up when whatever you uploaded it to runs your application that isn't the keyboard. It's probably a file, or at any rate, a limited stream.
You're asking for more tokens when there is nothing left to give.
Here's one obvious explanation:
This is homework or some coding exercise / coding competition.
You are uploading it to a grading server or competition testing server.
That server is (obviously - or you'd have to hire folks to type input data in over and over!) running your java app with the test data hooked up to System.in, and not an actual keyboard or even a virtualized one. Nobody is entering any keys to toss the test data at your app.
You have misunderstood the format of what the input is, so your application attempts to read more tokens than there actually are.
You can trivially reproduce this error yourself. First make a text file named 'test.txt', containing the string Hello and nothing more:
> cat test.txt
Hello
> cat Test.java
public class Test {
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
System.out.println(in.next());
System.out.println(in.next());
}
}
> javac Test.java
> java Test <test.txt
NoSuchElementException
After all, this code tries to read 2 tokens from the standard input, which is that test file, which doesn't have 2 tokens. The same thing is happening in your setup.
SOLUTION: Reread the exercise description, you've misunderstood the inputs. For example, I bet the description says that a blank line means you need to exit the app, or if a command quit or exit comes in, or whatnot. Your app runs forever, it's highly unlikely homework / a coding exercise requires this.
Is there an easy way to read a single char from the console as the user is typing it in Java? Is it possible? I've tried with these methods but they all wait for the user to press enter key:
char tmp = (char) System.in.read();
char tmp = (char) new InputStreamReader(System.in).read ();
char tmp = (char) System.console().reader().read(); // Java 6
I'm starting to think that System.in is not aware of the user input until enter is pressed.
What you want to do is put the console into "raw" mode (line editing bypassed and no enter key required) as opposed to "cooked" mode (line editing with enter key required.) On UNIX systems, the 'stty' command can change modes.
Now, with respect to Java... see Non blocking console input in Python and Java. Excerpt:
If your program must be console based,
you have to switch your terminal out
of line mode into character mode, and
remember to restore it before your
program quits. There is no portable
way to do this across operating
systems.
One of the suggestions is to use JNI. Again, that's not very portable. Another suggestion at the end of the thread, and in common with the post above, is to look at using jCurses.
You need to knock your console into raw mode. There is no built-in platform-independent way of getting there. jCurses might be interesting, though.
On a Unix system, this might work:
String[] cmd = {"/bin/sh", "-c", "stty raw </dev/tty"};
Runtime.getRuntime().exec(cmd).waitFor();
For example, if you want to take into account the time between keystrokes, here's sample code to get there.
I have written a Java class RawConsoleInput that uses JNA to call operating system functions of Windows and Unix/Linux.
On Windows it uses _kbhit() and _getwch() from msvcrt.dll.
On Unix it uses tcsetattr() to switch the console to non-canonical mode, System.in.available() to check whether data is available and System.in.read() to read bytes from the console. A CharsetDecoder is used to convert bytes to characters.
It supports non-blocking input and mixing raw mode and normal line mode input.
There is no portable way to read raw characters from a Java console.
Some platform-dependent workarounds have been presented above. But to be really portable, you'd have to abandon console mode and use a windowing mode, e.g. AWT or Swing.
Use jline3:
Example:
Terminal terminal = TerminalBuilder.builder()
.jna(true)
.system(true)
.build();
// raw mode means we get keypresses rather than line buffered input
terminal.enterRawMode();
reader = terminal .reader();
...
int read = reader.read();
....
reader.close();
terminal.close();
I' ve done it using jcurses...
import jcurses.system.InputChar;
import jcurses.system.Toolkit;
//(works best on the local machine when run through screen)
public class readchar3 {
public static void main (String[] args)
{
String st;
char ch;
int i;
st = "";
ch = ' ';
i = 0;
while (true)
{
InputChar c = Toolkit.readCharacter();
ch = c.getCharacter();
i = (int) ch;
System.out.print ("you typed " + ch + "(" + i + ")\n\r");
// break on '#'
if (ch == '#') break;
}
System.out.println ("Programm wird beendet. Verarbeitung kann beginnen.");
}
}
See This
It calls _getch() function from c to read a single char without hitting Enter
I have two projects in eclipse;
Project A is constantly streaming (currently to STDOUT) output (each new value streamed is newline delimited)
Project B I want to take in this, and do some stuff with it etc,
The only way I have found to do this is by writing to a file; I would prefer to avoid this as there is a ridiculous amount of data (at least for my resources), and most of it will end up being tossed.
Any advice appreciated, thanks!
There are several methods of approaching this:
Using TCP socket - not that difficult to achieve and gives you the ability to work remotely (run each program on a different computer).
Create a segmented file by program A and consume them with program B - The nuances are a bit tricky but overall a robust technique if it suits your needs
Use OS pipes - the easiest method given your current situation
I'll demonstrate option 3.
ProgramA.java:
for (int i=0; i<10; i++) {
System.out.println(i);
}
ProgramB.java:
java.io.BufferedReader br = new java.io.BufferedReader(new java.io.InputStreamReader(System.in));
String input;
while((input=br.readLine())!=null){
System.out.println("----" + input);
}
After which you can do this at the command line:
java -cp <program_A_classpath> ProgramA | java -cp <program_B_classpath> ProgramB
Output will be:
----0
----1
----2
----3
----4
----5
----6
----7
----8
----9
Is there an easy way to read a single char from the console as the user is typing it in Java? Is it possible? I've tried with these methods but they all wait for the user to press enter key:
char tmp = (char) System.in.read();
char tmp = (char) new InputStreamReader(System.in).read ();
char tmp = (char) System.console().reader().read(); // Java 6
I'm starting to think that System.in is not aware of the user input until enter is pressed.
What you want to do is put the console into "raw" mode (line editing bypassed and no enter key required) as opposed to "cooked" mode (line editing with enter key required.) On UNIX systems, the 'stty' command can change modes.
Now, with respect to Java... see Non blocking console input in Python and Java. Excerpt:
If your program must be console based,
you have to switch your terminal out
of line mode into character mode, and
remember to restore it before your
program quits. There is no portable
way to do this across operating
systems.
One of the suggestions is to use JNI. Again, that's not very portable. Another suggestion at the end of the thread, and in common with the post above, is to look at using jCurses.
You need to knock your console into raw mode. There is no built-in platform-independent way of getting there. jCurses might be interesting, though.
On a Unix system, this might work:
String[] cmd = {"/bin/sh", "-c", "stty raw </dev/tty"};
Runtime.getRuntime().exec(cmd).waitFor();
For example, if you want to take into account the time between keystrokes, here's sample code to get there.
I have written a Java class RawConsoleInput that uses JNA to call operating system functions of Windows and Unix/Linux.
On Windows it uses _kbhit() and _getwch() from msvcrt.dll.
On Unix it uses tcsetattr() to switch the console to non-canonical mode, System.in.available() to check whether data is available and System.in.read() to read bytes from the console. A CharsetDecoder is used to convert bytes to characters.
It supports non-blocking input and mixing raw mode and normal line mode input.
There is no portable way to read raw characters from a Java console.
Some platform-dependent workarounds have been presented above. But to be really portable, you'd have to abandon console mode and use a windowing mode, e.g. AWT or Swing.
Use jline3:
Example:
Terminal terminal = TerminalBuilder.builder()
.jna(true)
.system(true)
.build();
// raw mode means we get keypresses rather than line buffered input
terminal.enterRawMode();
reader = terminal .reader();
...
int read = reader.read();
....
reader.close();
terminal.close();
I' ve done it using jcurses...
import jcurses.system.InputChar;
import jcurses.system.Toolkit;
//(works best on the local machine when run through screen)
public class readchar3 {
public static void main (String[] args)
{
String st;
char ch;
int i;
st = "";
ch = ' ';
i = 0;
while (true)
{
InputChar c = Toolkit.readCharacter();
ch = c.getCharacter();
i = (int) ch;
System.out.print ("you typed " + ch + "(" + i + ")\n\r");
// break on '#'
if (ch == '#') break;
}
System.out.println ("Programm wird beendet. Verarbeitung kann beginnen.");
}
}
See This
It calls _getch() function from c to read a single char without hitting Enter
I have a program (in Java) that needs to use another program multiple times, with different arguments, during it's execution. It is multi-threaded, and also needs to do other things besides calling that program during it's execution, so I need to use Java to do that.
The problem is, all Runtime.exec() calls seem to be done in a synchronized way by Java, such that threads get bottlenecked not around the functions themselves, but in the Java call. Thus, we have a very slow running program, but that does not bottleneck at any system resource.
In order to fix that problem, I decided to not close the Process, and make all calls using this script:
#!/bin/bash
read choice
while [ "$choice" != "end" ]
do
$choice
read choice
done
And all the previous exec calls are substituted by this:
private Process ntpProc;
Initializer(){
try {
ntpProc = Runtime.getRuntime().exec("./runscript.sh");
} catch (Exception ex) {
//Error Processing
}
}
public String callFunction(String function) throws Exception e{
OutputStream os = ntpProc.getOutputStream();
String result = "";
os.write((function + "\n").getBytes());
os.flush();
BufferedReader bis = new BufferedReader(new InputStreamReader(ntpProc.getInputStream()));
int timeout = 5;
while(!bis.ready() && timeout > 0){
try{
sleep(1000);
timeout--;
}
catch (InterruptedException e) {}
}
if(bis.ready()){
while(bis.ready()) result += bis.readLine() + "\n";
String errorStream = "";
BufferedReader bes = new BufferedReader(new InputStreamReader(ntpProc.getErrorStream()));
while(bes.ready()) errorStream += bes.readLine() + "\n";
}
return result;
}
public void Destroyer() throws exception{
BufferedOutputStream os = (BufferedOutputStream) ntpProc.getOutputStream();
os.write(("end\n").getBytes());
os.close();
ntpProc.destroy();
}
That works very well, and actually managed to improve my program performance tenfold. SO, my question is: Is this correct? Or am I missing somethings about doing things this way that will make everything go terribly wrong eventually?
If you want to read from the process Error and Input streams ( aka stderr and stdout ), you need to do this job on dedicated threads.
The main problem is that you need to empty the buffers as they become filled up, and you can only do this on a separate thread.
What you did, you've managed to shorten the output, so it does not overflow these buffers, but the underlying problem is still there.
Also, from the past experience, calling external process from Java is extremely slow, so your approach may be better after all.
As long as you not calling Proccess.waitFor(), execution of process will not block. As Alex said - blocking in your case caused by those loops to read the output.
You can use commons-exec package, as it provides a nice way of running processes (sync or async), handling output, setting timeouts, etc.
Here is a link to a project:
http://commons.apache.org/exec/
The best example of using the api is test class they have:
http://svn.apache.org/viewvc/commons/proper/exec/trunk/src/test/java/org/apache/commons/exec/DefaultExecutorTest.java?view=markup