We Keep Coding

Why is my stdout not streaming in real time using zeroturnaround redirectOutput

I'm using the zeroturnaround zt-exec library (https://github.com/zeroturnaround/zt-exec), and I can't figure out why the output lines are not processed as they come in. It will wait until the script is done before spitting them all out at once. Is this something where they are being buffered? Seems like decreasing that would be pretty hacky.
Tried using 'start' instead of 'execute', and looked into buffer size but can't figure out what the issue is.
This answer seems to have what I'm looking for but I still get the issue
Read live output of process in java
Java code:
new ProcessExecutor().command("python", "printTest.py")
.redirectOutput(new LogOutputStream() {
#Override
protected void processLine(String line) {
logger.info("**** Line:" + line);
webSocketActorRef.tell(String.format("line:%s", line), self());
}
})
.execute();
python code:
print("**********INFO:MOCK SCRIPT STARTING **********")
print("STATUS: Starting the Mock simulation")
time.sleep(5)
print("STATUS: Message 1")
time.sleep(5)
print("STATUS: Message 2")
time.sleep(5)
print("STATUS: Message 3")

Error handling when reading from a serial port

I am working with an Arduino and making a small GUI to communicate with it on Processing. I am just printing out float values onto the serial port and reading it back using Processing. In most cases everything goes well and I am able to read the values. However, sometimes the serial read spits out fairly arbitrary values and I am not sure why. For example,
My Arduino code:
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
}
void loop() {
for (int i =1;i<5;i++)
{ Serial.println(float(i)+2.0);
delay(1);
// put your main code here, to run repeatedly:
}
}
The output according to the Serial monitor is 3.0,4.0,5.0,6.0 as expected.
Here's the processing code (Python Mode) that I use to read the data in
def connect2Arduino():
global arduinoPort
arduinoPort= Serial(this, 'COM32',9600)
def setup():
background(0)
connect2Arduino()
def draw():
global arduinoPort
if arduinoPort.available()>0:
dataIn = arduinoPort.readStringUntil(int(10))
if (dataIn != None or int(dataIn) != 13):
print dataIn
The output from processing looks like this
4.00
5.00
6.00
3.00
4.00
5.‚j
6.°3.00
4.00
5.00
6.00
When I try to change the type of dataIn, an error pops up when output is 5.‚j which is as expected.
First of, I have no idea why these errors exist. I would like to know why it's showing up in the first place. Secondly, what's a fix? I could go print instead of println on my arduino code and maybe fix it, but I am looking for something better.
Thank you

You may be filling the serial buffer. Try checking the value returned by println and verify the correct number of bytes have been processed. You can also try adding a Serial.flush() call after each print to ensure the data has been sent out.

Why static block are executed later?

P.S :
This question has been edited a few times as my previous code doesn't demonstrate the problem. There are some answers which may not make perfect sense against the edited question
I have a public class named Son.java
package com.t;
public class Son extends Father {
static int i;
static {
System.out.println("son - static");
i = 19;
}
{
System.out.println("son - init-block");
}
public static void main(String[] args) {
//Son s = new Son();
int a[] = new int[2];
System.out.println(a[5]);
}
}
class Father {
static {
System.out.println("f - static");
}
{
System.out.println("f - init-block");
}
}
When I run the program for the 1st time:
Output is:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
at com.t.Son.main(Son.java:19)
f - static
son - static
And later when I run this program (order of output is random)
Output is:
f - static
son - static
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
at com.t.Son.main(Son.java:19)
I have read that static blocks are executed as the classes are initalised.
But why does the exception has come first here and then static block is executed?
I am using Eclipse too to run my program.
Can somebody explain?

The exception doesn't happen first, you are just seeing the printout of the exception first.
Had the exception happened first, you would never have seen the rest of the output.
The reason for this is that you have output to both System.err (from your exception) and System.out in your program. The order in which these are printed to the screen is not defined, so therefore you can get them in different order.

Stack traces of uncaught exceptions are printed in System.err, which is an unbuffered stream. You print text to System.out which is a buffered stream and it is unpredictable whether it he buffer gets flushed before or after the stack trace is printed.
If you change all your print statements to System.err then the order of the output will become the order of printing, and it will always be the same order.

#Keppil's answer has nailed it.
I just want to point out something ... erm ... interesting.
The OP says this:
I am using Eclipse to run my program.
The knee jerk response would be to say "that isn't relevant" ... but in this case, I think it >>is<< relevant. I suspect that non-determinism in stdout/stderr timing is being is amplified by the fact that the output is going to a Eclipse "console" panel.
When an application is run from the command line, output to stderr and stdout probably gets merged into out stream somewhere in the OS kernel. And if not, the console program probably uses a select syscall to handle input from two sources ... and gives one stream priority over the other, 'cos that is the easy way to code it. As a result, you would expect the output to appear on the console in a mostly consistent order, even though order is non-deterministic.
But when the application writes to an Eclipse console, Eclipse probably uses a separate thread to read each stream. Assuming that both threads are blocked in read syscalls, and input arrives at roughly the same time on both streams, it will be up to the thread scheduler to decide which thread gets woken first. That is going to be far less predictable that the behaviour of a select ... or of stream merging in the kernel.
Either way, my observation is that reordered stdout / stderr output is more prevalent with an Eclipse console than when you are using a "native" console.

As the asker said main belongs to Son class and is extending Father. I modified the code a little, so I was able to compile.
class Father {
static{
System.out.println("f - static");
}
}
public class Son extends Father {
static {
System.out.println("son - static");
}
public static void main(String[] args) throws ArrayIndexOutOfBoundsException{
int a[] = new int[2];
System.out.println(a[3]);
}
}
And the output is:-
f - static
son - static
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 3
at kanwal.Son.main(Son.java:20)
Its working exactly, the way it is supposed to.
EDIT:-
This answer was made, before OP edited the question.

How to get the stacktrace in a mobile device?

I'm getting a NullPointerException in a Nokia S40.
I want to know what is causing this exception.
The device shows:
NullPointerException java/lang/NullPointerException
This error only occurs in the device, running in the emulator the application works fine.
I use microlog to debug my application. But the application works fine if the log is enabled.
Is there a way to get the stack trace when I get this NullPointerException? I don't need all details like the line number just which method cause this exception.
UPDATE: I installed the same application in another Nokia S40 and the same error didn't occur.
Nokia 2660 - error
Nokia 6131 - no error
UPDATE 2: Somehow I find what was causing the NullPointerException.
import javax.microedition.lcdui.Canvas;
import javax.microedition.lcdui.Graphics;
public class OuterClass extends Canvas {
private Config config;
public OuterClass() {
this.config = new Config();
}
public void paint(Graphics graphics) {
HelperClass helper = new HelperClass(this.config);
helper.doStuff();
}
public void dispose() {
this.config = null;
}
public class Config implements IConfig {
public int getSomething() {
// ...
}
}
}
public class HelperClass {
private IConfig config;
public HelperClass(IConfig) {
this.config = config;
}
public doStuff() {
config.getSomething(); // Here is thrown NullPointerException
}
}
In some situations a thread is started and call the OuterClass.dispose() before the helper.doStuff() causing the NPE. I think when I enabled the log it made the thread slower and helper.doStuff() was called when I expected it to be called.

You are not going to find any way to save a Throwable stack trace on a Nokia Series40 handset.
The usual brute force way of debugging JavaME application on Series40 is to modify your code to create a stack trace yourself in memory.
What I'm talking about is:
Each Thread that you can identify (including system callback threads) needs its own Stack object, containing strings. Obviously, this increases the memory footprint of your application somewhat but keeping it in memory should limit the impact on race conditions.
When your code enters a method, it adds the method signature to the current Thread Stack. When the method exits (and you better only have one exit point per method) it pops the top of the Stack.
You can add aditional debug info on the stack, like values of variables in different places of the code.
You don't necessarily need to add this to every single method in your code.
You can add try{}catch(Throwable){} to the entry point of every thread you identified and either dump the stack in a file or on the screen (in a Form).
Obviously, this is not the kind of change you want to manually add in a lot of places in a large existing codebase. You can however make it part of your organisation coding standards for the future and write a source code parsing script to automatically add it to existing code.

I had some trouble in the past trying to print the stack trace to somewhere else than the standard output. The standard exception class doesn't provide the printStackTrace method that receives the output stream, therefore it only prints to the standard output stream.
It's possible, at least in Java SE, to redirect the java output stream to somewhere else by simply saying that System.out = . The PrintStream class receives an OutputStream, which means that you could create your own ByteArrayOutputStream, initialize a PrintStream, sets System.out to that stream, and then call ex.printStackTrace(). I don't have a J2ME environment here but I believe that, as long as it won't break when you try to set System.out to something else (nothing says that it's readonly in the docs), you should be able to do it.
After you do that, I would recommend writing it to a RecordStore that you have specifically for that, and then upload the records of that RecordStore to your server so you can get it.
I know it's not very easy but it may work. I would first test the System.out thing. If it works, everything else should work too.
My answer was wrong. As pointed out, the System.out and System.err fields are declared final. If you can't get the stack trace, and if you can't get the error when running the application on your emulator, try creating trace bullets on your code (alerts, logs, whatever you can) to isolate the piece of code where the problem is happening. It has to be something that could change between the emulator and the real device - for example, something related to retrieving/ saving records in a RecordStore, opening a Connection, etc... What did you try to do when you had the problem?

You could use Microlog to send an e-mail when the exception occurs.
Microlog

You could try catching the exception in some high-level try/catch block and then emailing the trace to yourself.

Program output lost when passed through PsExec

(This is a question my coworker posted elsewhere, but I thought I'd post it here to see if I could hit a different audience.)
Hello all,
I'm testing the possibility of writing a small java application the will use Psexec to kick off remote jobs. In the course of testing binding the stdin and stdout of a java program to psexec I came across an odd bug.
My test program is a basic echo program. It starts a thread to read from stdin and then pipes the read output directly back to stdout. When run on the local machine, not from psexec, it works beautifully. Exactly as it should.
However, when I call it from PsExec the first time the input is piped directly into stdout it is lost. What makes the bug really bizzare is that it is only the first time the input is piped directly into stdout that it is lost. If the input String is appended to another string it works fine. Either a String literal or a String variable. However, if the input String is sent directly to stdout it doesn't go through. The second time it is sent to stdout it goes through fine - and everytime there after.
I'm at a complete loss as to what's going on here. I've tried to test for every possible bug I can think of. I'm out of ideas. Did I miss one or is this just something inside psexec?
Here is the code in question, it's in three classes (one of which implements an interface which is a single function interace).
The Main class:
public class Main {
public static void main(String[] args) {
System.out.println("Starting up.");
CReader input = new CReader(new BufferedReader(
new InputStreamReader(System.in)));
CEcho echo = new CEcho();
input.addInputStreamListener(echo);
input.start();
System.out.println("Successfully started up. Awaiting input.");
}
}
The CReader class which is the thread that reads from stdin:
public class CReader extends Thread {
private ArrayList<InputStreamListener> listeners =
new ArrayList<InputStreamListener>();
private boolean exit = false;
private Reader in;
public CReader(Reader in) {
this.in = in;
}
public void addInputStreamListener(InputStreamListener listener) {
listeners.add(listener);
}
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(input);
}
#Override
public void run() {
StringBuilder sb = new StringBuilder();
int current = 0, last = 0;
while (!exit) {
try {
current = in.read();
}
catch (IOException e) {
System.out.println("Encountered IOException.");
}
if (current == -1) {
break;
}
else if (current == (int) '\r') {
if(sb.toString().length() == 0) {
// Extra \r, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else if(current == (int) '\n') {
if(sb.toString().length() == 0) {
// Extra \n, don't return empty string.
continue;
}
fireInputRecieved(new String(sb.toString()));
sb = new StringBuilder();
}
else {
System.out.println("Recieved character: " + (char)current);
sb.append((char) current);
last = current;
}
}
}
}
The CEcho class, which is the class that pipes it back to stdout:
public class CEcho implements InputStreamListener {
public void inputRecieved(String input) {
System.out.println("\n\nSTART INPUT RECIEVED");
System.out.println("The input that has been recieved is: "+input);
System.out.println("It is a String, that has been copied from a " +
"StringBuilder's toString().");
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
System.out.println("Finished example outputs of input: "+input);
System.out.println("END INPUT RECIEVED\n\n");
}
}
And finally, here is the program output:
>psexec \\remotecomputer "C:\Program Files\Java\jre1.6.0_05\bin\java.exe" -jar "C:\Documents and Settings\testProram.jar"
PsExec v1.96 - Execute processes remotely
Copyright (C) 2001-2009 Mark Russinovich
Sysinternals - www.sysinternals.com
Starting up.
Successfully started up. Awaiting input.
Test
Recieved character: T
Recieved character: e
Recieved character: s
Recieved character: t
Input string has made it to fireInputRecieved: Test
START INPUT RECIEVED
The input that has been recieved is: Test
It is a String, that has been copied from a StringBuilder's toString().
Outputting it cleanly to standard out:
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
END INPUT RECIEVED

have you tried redirecting the output into a file ( java... >c:\output.txt )? this way you could doublecheck if everything is going into stdout and maybe just getting eaten by psexec

PsExec is eating the output. Next interesting thing might be where it's eating the output. You could check this by getting a copy of Wireshark and checking whether the output in question is traversing the network or not. If it's not, then it's being eaten on the remote side. If it is, it's being eaten locally.
Not that I'm really sure where to go from there, but collecting more information certainly seems like a good path to be following...

I was having the same issue and tried multiple combinations of redirects.
This is what worked:
processBuilder.redirectErrorStream(true);
processBuilder.redirectOutput(Redirect.PIPE);
processBuilder.redirectInput(Redirect.INHERIT);
final Process process = processBuilder.start();
// Using Apache Commons IOUtils to get output in String
StringWriter writer = new StringWriter();
IOUtils.copy(process.getInputStream(), writer, StandardCharsets.UTF_8);
String result = writer.toString();
logger.info(result);
final int exitStatus = process.waitFor();
The Redirect.INHERIT for processBuilder.redirectInput got me the missing remote command output.

Is System.out not configured for autoflush? After the first print try System.out.flush() and see if the first line appears without more lines being printed.
(oh yeah, seriously, it is "RECEIVED", not "RECIEVED".)

OK, I've been thinking about this over the weekend and I since you are jumping from machine to machine I wonder if maybe there is a CharSet issue? Maybe it is eating the string the first time and dealing with a different code page or character set issue? Java is 16bit characters normally and windows is either 8bit with code pages or utf-8 these days.
Any chance the local and remote machines have different default character sets? If you are sending localized data over the net it might misbehave.

What I see when running psexec is that it spawns a child window to do the work but doesnt return that program's output to it's console window. I would suggest using WMI or some form of windows process API framework to gain a level of control you appear to lack with psexec. Surely java has an equivalent to .Net's System.Diagnotics.Process class.

Maybe you could try passing a copy of input to your listeners:
public void fireInputRecieved(String input) {
if(input.equals("quit"))
exit = true;
String inputCopy = new String(input);
System.out.println("Input string has made it to fireInputRecieved: "
+ input);
for(int index = 0; index < listeners.size(); index++)
listeners.get(index).inputRecieved(inputCopy);
}
I had similar problems with listeners where a passed variable would end up empty unless I did pass an explicit copy of it.

I don't necessarily have an answer, but some comments may prove helpful.
The "pass a copy" idea shouldn't matter, since your output successfully prints the string twice before the failure, then succeeds again afterward.
auto-flush shouldn't matter either, as you've already mentioned
Niko's suggestion has some merit, for diagnostic purposes. Mixed with Mark's suggestion, it makes me wonder if there aren't some invisible control characters getting involved somewhere. What if you printed the characters byte values as a diagnostic step?
You know that the value is "Test" (at least in the output you gave us). What happens if you pass "Test" directly to the failing printLn statement?
In situations like this, you want to gain as much information as possible. Insert breakpoints and analyze characters. Send the bytes to files and open them in hex editors. Do whatever you can to trace things as accurately and as precisely as possible.
Come up with weird test scenarios and try them, even if they shouldn't possibly help. You never know what good idea you might have while analyzing the results of the hopeless idea.

I'd guess that there is a bogus byte in there prefacing the T. According to JavaDocs, an InputStreamReader will read one or more bytes, and decode them into characters.
You could have an escape sequence or spurious byte in there, masquerading as a multibyte character.
Quick check - see if "current" is ever > 128 or < 33.
What if you used a CharArrayReader to get individual bytes, without any charset translation?
The theory is that during the first attempt to output the String using println, it's sending an escape character of some sort, eating the rest of the string. During later prints, either Java or the network pipe are handling or removing it, since it previously got that escape sequence, perhaps changing the handling in some way.
As an unrelated nit, sb.toString() returns a new String, so it's unnecessary to call "new String(sb.toString())"

Same issue here, I'm going through this post again and again these days, hoping I can find some solution. Then I decide I should give up psexec and find some alternative. So this is the thing: PAExec. Works perfect for getting command output.

How are you executing PsExec? My suspicion is that this is some code within PsExec which is actually doing echo suppression, possibly for the purposes of protecting a password. One way to test this hypothesis would be to change this code:
System.out.println("Outputting it cleanly to standard out: ");
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
to this:
System.out.println("Outputting it cleanly to standard out: ");
System.out.print(' ');
System.out.println(input);
System.out.println("Outputting it cleanly to standard out again: ");
System.out.println(input);
...thereby causing the output to be (if I'm right):
Outputting it cleanly to standard out:
Test
Outputting it cleanly to standard out again:
Test
Finished example outputs of input: Test
In particular, it's noticeable that the apparently-suppressed line is the first line which consists solely of Test - which is exactly the text you've just sent to the remote system. This sounds like PsExec attempting to suppress a remote system which is echoing its input in addition to producing its own output.
Is the password of the user on the remote machine perhaps Test? Are you using PsExec's -p parameter? Are you specifying -i?

I am dealing with this same issue and I am wondering if it has to do with how the cmd window and pipes in windows work while you don't have a true windowed session. The suppressed output happens when any new process is spawned. You would think that if you spawn a process that the stdout/stderr/stdin would be inherited from the process that spawned it; after all that is what happens if you spawn the process from a normal cmd window and the output from the new process is piped back to your own console. However if somewhere in the inheritance of the pipes something were to go wrong, like say not passing a WINDOW.GUI object because there is no physical window, windows doesn't let the stdin/stdout/stdin to be inherited. Can any one do some investigation or open a windows support ticket for this?

Seems no easy solution. My work-around in a recent project is using paexec.exe product. It captures output/error easily in JAVA(java-8), but hangs up upon completion of the remote command execution. When running this inside a server on the hosted machine, I have to spurn a new child JVM process to run paexec.exe and force kill it via its PID upon completion in order to release all the resources.
If anyone has better solution, please post it.