We Keep Coding

Client-Thread mysteriously leaves out Java code, thus not answering

Every line of code is executed -> Event -> Important Java code lines skipped
# Client-Server # AssumeNoCodeOptimization # IDE:Processing (Processing.org by MIT)
There is a Thread "ConnectionHandler" that stays in a loop where it either sends or receives data. Before it receives the first data, every line of code is executed. After that, the program seems to pause at a System.out.println()!?
Then, when more data arrive, certain lines of Java Code are executed again, but the very same System.out.println() - in addition other things - I mentioned before is skipped.
The very problem is actually that a method sendOutput() is skipped. This is what really grinds my gears. Please help.
I am programming in an environment called Processing, developed by the MIT. As far as I know it only wraps more Java code arround your code.
Below the following code snippets I will explain why I think that code optimization by JIT or AOT is not the problem (At the end I guess I will be wrong. I hope the opposite because you can not pass System variables to Processing to suppress optimization)
private void runConnectionHandler() {
final BufferedReader inFromClient = getBufferedReader(socket);
final DataOutputStream dataOutputStream = getDataOutputStream(socket);
while (true) {
getInput(inFromClient);
sendOutput(dataOutputStream);
System.out.println("Cycle ends");
}
}
private void getInput(final BufferedReader input) {
System.out.println("Get");
try {
String clientSentence = null;
if (input.ready()) {
while ((clientSentence = input.readLine()) != null) {
inputQueue.add(new GameData(clientSentence));
System.out.println("Received se: \"" + clientSentence + "\"");
System.out.println("1");
}
System.out.println("2");
}
System.out.println("3");
}
catch (final Exception e) {
e.printStackTrace();
}
}
private void sendOutput(final DataOutputStream dataOutputStream) {
System.out.println("Send");
while (outputQueue.peek() != null) {
try {
String out = outputQueue.poll().toString();
dataOutputStream.writeBytes(out + "\r\n");
dataOutputStream.flush();
System.out.println("Sent \"" + out + "\"");
}
catch (final Exception e) {
e.printStackTrace();
}
}
}
OUTPUT: Before first data:
Having no idea about optimization I would think that it should also come to effect here, but it does not?
...
Get
3
Send
Cycle ends
...
OUTPUT: First data arrive
After the "1" the ouput stops. Why is System.out.println("2"); and so on not executed?! Why does it stop?!?!
...
Get
3
Send
Cycle ends
Get
Received se: "SET PLAY MODE"
1
OUTPUT: Manually sent more data
Gad dayium, I mean where went my System.out.println("Get"); and everything?!
...
Get
Received se: "SET PLAY MODE"
1
Received se: "Hey Ho1"
1
OUTPUT: Server shuts down
When I shut down the other side (Server), every line of code is executed again ?!?! Futhermore neither it stops, nor an execution is thrown (But maybe different issue).
...
Get
3
Send
Cycle ends
...
Also if my concept is deeply broken I would appreciate a lot of hints.

java println blocked after communicating with exe if i do not close the stream

I am currently interfacing a GUI to an UCI chess engine. For this purpose i am creating the engine process using:
try {
process = Runtime.getRuntime().exec(enginePath);
} catch (IOException e) {
System.err.println("ENGINE NOT FOUND");
e.printStackTrace();
}
and i am sure that i am able to open the engine.
When engine is opened, there is no significant stream outputted from engine. In order to initiate communication, I have to send specific commands to the engine. The engine will respond then... Therefore it is working in a command/response approach(not immediately streaming data when opened or talking without spoken to). In order to communicate i have a send message block. In this block write a message to the engine using its outputstream and get input using its standart input stream as in the following send method:
private String sendCommand(String command) {
stdin = new PrintWriter(new OutputStreamWriter(process.getOutputStream()));
BufferedReader inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedReader errReader = new BufferedReader(newInputStreamReader(process.getErrorStream()));
String answer = "";
stdin.println(command);
stdin.flush();
stdin.close();
try {
String line = "";
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
inputReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
try {
String line = "";
while ((line = errReader.readLine()) != null) {
System.err.println(line);
}
errReader.close();
} catch (IOException ioe) {
System.err.println("READ ERROR");
ioe.printStackTrace();
}
return answer;
}
By the way the whole engine is working in a single seperate thread other than the main GUI thread. Therefore no overloading for the GUI. However i didnot seperate the stdin stderr and output methods in individual threads since i suppose there will be only stdin and the one that i will write to the engine. For the time being i assume they will not collide.
With these codes i am able to communicate with the engine and i can see the outputs in the console. However due to stdin.close(); i can only use this method once (In the seconds time i have a STREAM CLOSED error). The chess engine is needed to be communicated in command/response approach many times when opened, without restarting the exe in each time. The problem is that if i remove the line : stdin.close(); yes the communication continues, but my console is blocked by this communication. I.e. i cannot println to the console for debugging purposes anymore. Which is very critical because my main debugging weapon is system.out.println. If i do not remove stdin.close(); i have to restart executable each time i want to send message and i do not want that. **The strange part is that i can see the messages coming from the engine in my console due to "System.out.println(line);" line in the send method; however i cannot print anything on the console once the code exits the method. **
EDIT: Actually after this block:
while ((line = inputReader.readLine()) != null) {
System.out.println(line);
answer = line;
}
System.out.println starts not to work.
EDIT: The problem seems not to be with println but any statement after while loop.
Why statements after while loop is not getting executed?
according to this, it seems that since the stream is never closed, "while loop" is stuck(?). Actually when printing inside while loop, at some point(after a stream is finished), the prints stop, which must indicate while loop is finished. Therefore if it is finished, it should continue on the next statements, shouldn't it? Anyway, the messages from the engine had a set of strings at end of each stream; therefore when i encounter one of them i am breaking the while loop.
Note: it is also interesting that : accumulating data as: "receivedString += line"; doesnot work,i.e. data is not accumulated. In order to fix it i luckily made it "receivedString = receivedString + line + "\n""; and it worked.. I dont know why.

You're reading the input until end of stream, so it won't stop reading until end of stream occurs. End of stream means that the peer has closed the connection in this case. That won't happen for a process's output or error stream until the process exits. You'll have to find some other loop termination condition, or else consume the stdout and stderr in separate threads.

Can't detect disconnect without extra readLine() loop

I am developing a program that uses sockets and currently I have a function in my code that checks for a heartbeat from the client every second.
private void userLoop() { // checks for incoming data from client
Timer t = new Timer();
t.schedule(new TimerTask() {
#Override
public void run() {
try {
socketIn.read(); // check for heartbeat from client
String userInput;
while ((userInput = br.readLine()) != null) {
}
} catch (Exception e) {
ControlPanel.model.removeElement(getUsername());
ControlPanel.append(getUsername() + " has disconnected.");
}
}
}, 1000);
}
When a client closes the game via the X button, shutting off their computer, logging out, whatever it may be, I get the message "'username' has disconnected". This is exactly what I want, however, it only works with the while loop in the code. The while loop essentially does nothing and I have no idea why it doesn't work with out.
If I remove the while loop and I disconnect using my client nothing gets printed out server sided.
String userInput;
while ((userInput = br.readLine()) != null) {
}
The above is essentially the dead code that does nothing but without it my program doesn't work the way it should..
Why is the code needed and how can I remove it and still make my program work correctly?

In this case, your while loop is essentially stalling your program until you no longer receive an input string. It's not dead code; it is just your way of installing a wait.
Otherwise, based on my understanding in the Timer class, it only waits one second, which might be too short of a timespan for what you're waiting to capture.

I fixed my problem by changing everything in the try block with
br.readLine();

There's a saying I've heard about exception handling: "Exceptions should only be used for exceptional situations." A client disconnecting from a server is not exceptional.
Now that I have that off my chest, let's move on. According to this other question,
socket.getInputSteam.read() does not throw when I close the socket from the client
it sounds like the read call won't throw if you're closing things properly on the client side.

The problem is that when the remote socket is closed, read() does not throw an Exception, it just returns -1 to signal the end of the stream.
The following should work without needing to call readLine():
try {
int ret = socketIn.read(); // check for heartbeat from client
if (ret == -1) {
// Remote side closed gracefully
clientDisconnected();
}
} catch (SocketTimeoutException e) {
// Timeout -- handle as required
handleTimeout();
} catch (IOException e) {
// Connection lost due to I/O error
clientDisconnected()
}

Problems with InputStream

Following is a part of the code snippet that I will be using for my project.
public String fetchFromStream()
{
try
{
int charVal;
StringBuffer sb = new StringBuffer();
while((charVal = inputStream.read()) > 0) {
sb.append((char)charVal);
}
return sb.toString();
} catch (Exception e)
{
m_log.error("readUntil(..) : " + e.getMessage());
return null;
} finally {
System.out.println("<<<<<<<<<<<<<<<<<<<<<< Called >>>>>>>>>>>>>>>>>>>>>>>>>>>");
}
}
Initially the while loop start working pretty fine. But after the probable last character is read from the stream I was expecting to get -1 return value. But this is where my problem starts. The code gets hanged, even the finally block is not executed.
I was debugging this code in Eclipse to see what is actually happening during the run-time. I set a pointer (debug) inside the while loop and was constantly monitoring the StringBuffer getting populated with char values one by one. But suddenly while checking the condition inside the while loop, the debugging control is getting lost and this is the point where the code goes to hangup state !! No exception is thrown as well !!
What is happening here ?
Edit::
This is how I'm getting my InputStream. Basically I'm using Apache Commons Net for Telnet.
private TelnetClient getTelnetSession(String hostname, int port)
{
TelnetClient tc = new TelnetClient();
try
{
tc.connect(hostname, port != 0 ? port : 23);
//These are instance variables
inputStream = tc.getInputStream();
outputStream = new PrintStream(tc.getOutputStream());
//More codes...
return tc;
} catch (SocketException se)
{
m_log.error("getTelnetSession(..) : " + se.getMessage());
return null;
} catch (IOException ioe)
{
m_log.error("getTelnetSession(..) : " + ioe.getMessage());
return null;
} catch (Exception e)
{
m_log.error("getTelnetSession(..) : " + e.getMessage());
return null;
}
}

Look at the JavaDocs:
Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.
In simple turns: if your stream ended (e.g. end of file), read() returns -1 immediately. However if the stream is still open but JVM is waiting for data (slow disk, socket connection), read() will block (not really hung).
Where are you getting the stream from? Check out the available() - but please do not call it in a loop exhausting CPU.
Finally: casting int/byte to char will only work for ASCII characters, consider using Reader on top of InputStream.

read the docs
read() will wait until there is more data on the InputStream if the InputStream is not closed.
I suspect you are doing this with sockets? This is the most common area where this comes up.
"Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown"

I have the same issue with the Apache Commons on Android ...
the read() command on the inputstream hangs forever for some reason. And no, it is not just blocking "until data is available" ...
my debugging information shows that there are several 100 chars available() ... yet it just randomly blocks at some read. However, whenever I send something to the telnet server the block is suddenly released and it will continue reading for several chars until it suddenly stops/blocks again at some arbitrary point!
I believe there is some bug within the Apache Commons library! This is really annoying because there isn't a lot that can be done ... no timeout for the read command or anything else ...
EDIT: I was able to get around it ... by setting the TelNetClient.setReaderThread(false) ... obviously there is a bug within the Library that exists as long as a thread handles the input data ... when dispabled it works just fine for me!

Java - Flushing the OutputStream of a process doesn't send the data immediately if it's too small

I'm firing up an external process from Java and grabbing its stdin, stdout and stderr via process.getInputStream() etc. My issue is: when I want to write data to my output stream (the proc's stdin) it's not getting sent until I actually call close() on the stream. I am explicitly calling flush().
I did some experimenting and noticed that if I increased the number of bytes I was sending, it would eventually go through. The magic number, on my system, is 4058 bytes.
To test I'm sending the data over to a perl script which reads like this:
#!/usr/bin/perl
use strict;
use warnings;
print "Perl starting";
while(<STDIN>) {
print "Perl here, printing this: $_"
}
Now, here's the java code:
import java.io.InputStream;
import java.io.IOException;
import java.io.OutputStream;
public class StreamsExecTest {
private static String readInputStream(InputStream is) throws IOException {
int guessSize = is.available();
byte[] bytes = new byte[guessSize];
is.read(bytes); // This call has side effect of filling the array
String output = new String(bytes);
return output;
}
public static void main(String[] args) {
System.out.println("Starting up streams test!");
ProcessBuilder pb;
pb = new ProcessBuilder("./test.pl");
// Run the proc and grab the streams
try {
Process p = pb.start();
InputStream pStdOut = p.getInputStream();
InputStream pStdErr = p.getErrorStream();
OutputStream pStdIn = p.getOutputStream();
int counter = 0;
while (true) {
String output = readInputStream(pStdOut);
if (!output.equals("")) {
System.out.println("<OUTPUT> " + output);
}
String errors = readInputStream(pStdErr);
if (!errors.equals("")) {
System.out.println("<ERRORS> " + errors);
}
if (counter == 50) {
// Write to the stdin of the execed proc. The \n should
// in turn trigger it to treat it as a line to process
System.out.println("About to send text to proc's stdin");
String message = "hello\n";
byte[] pInBytes = message.getBytes();
pStdIn.write(pInBytes);
pStdIn.flush();
System.out.println("Sent " + pInBytes.length + " bytes.");
}
if (counter == 100) {
break;
}
Thread.sleep(100);
counter++;
}
// Cleanup
pStdOut.close();
pStdErr.close();
pStdIn.close();
p.destroy();
} catch (Exception e) {
// Catch everything
System.out.println("Exception!");
e.printStackTrace();
System.exit(1);
}
}
}
So when I run this, I get effectively nothing back. If immediately after calling flush(), I call close() on pStdIn, it works as expected. This isn't what I want though; I want to be able to continually hold the stream open and write to it whenever it so pleases me. As mentioned before, if message is 4058 bytes or larger, this will work without the close().
Is the operating system (running on 64bit Linux, with a 64bit Sun JDK for what it's worth) buffering the data before sending it? I could see Java having no real control over that, once the JVM makes the system call to write to the pipe all it can do is wait. There's another puzzle though:
The Perl script prints line before going into the while loop. Since I check for any input from Perl's stdout on every iteration of my Java loop, I would expect to see it on the first run through the loop, see the attempt at sending data from Java->Perl and then nothing. But I actually only see the initial message from Perl (after that OUTPUT message) when the write to the output stream happens. Is something blocking that I'm not aware of?
Any help greatly appreciated!

You haven't told Perl to use unbuffered output. Look in perlvar and search for $| for different ways to set unbuffered mode. In essence, one of:
HANDLE->autoflush( EXPR )
$OUTPUT_AUTOFLUSH
$|

Perl may be buffering it before it starts printing anything.
is.read(bytes); // This call has side effect of filling the array
No it doesn't. It has the effect of reading between 1 and bytes.length-1 bytes into the array. See the Javadoc.

I don't see any obvious buffering in your code, so it may be on the Perl side. What happens if you put a newline \n at the end of your print statement?
Note also that you can't, in general, read the stdin and stderr on the main thread like that. You'll be subject to deadlock - e.g., if the child process prints lots of stderr, while the parent is reading stdin, the stderr buffer will fill and the child process will block, but the parent will stay blocked forever trying to read stdin.
You need to use separate threads to read stderr and stding (also separate from the main thread, which here is used to pump input to the process).