We Keep Coding

IOIO UART readback problems

I'm extending the BaseIOIOLooper to open up a UART device and send messages. I'm testing with a readback, where I send a packet over a line and receive that packet on another line and print it out. Because I don't want the InputStream.read() method to block, I am handling packet formation and input in a different thread. I have narrowed my problem down to the InputStream.read() method, which returns -1 (no bytes read, but no exception).
Here is what it looks like in the Looper thread:
#Override
protected void setup() throws ConnectionLostException, InterruptedException {
log_.write_log_line(log_header_ + "Beginning IOIO setup.");
// Initialize IOIO UART pins
// Input at pin 1, output at pin 2
try {
inQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
outQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
ioio_.waitForConnect();
uart_ = ioio_.openUart(1, 2, 38400, Uart.Parity.NONE, Uart.StopBits.ONE);
// Start InputHandler. Takes packets from ELKA on inQueue_
in_= new InputHandler(inQueue_, uart_.getInputStream());
in_.start();
// Start OutputHandler. Takes packets from subprocesses on outQueue_
out_= new OutputHandler(outQueue_);
out_.start();
// Get output stream
os_=uart_.getOutputStream();
// Set default target state
setTargetState(State.TRANSFERRING);
currInPacket_[0]=1; //Initial value to start transferring
log_.write_log_line(log_header_ + "IOIO setup complete.\n\t" +
"Input pin set to 1\n\tOutput pin set to 2\n\tBaud rate set to 38400\n\t" +
"Parity set to even\n\tStop bits set to 1");
} catch (IncompatibilityException e) {
log_.write_log_line(log_header_+e.toString());
} catch (ConnectionLostException e) {
log_.write_log_line(log_header_+e.toString());
} catch (Exception e) {
log_.write_log_line(log_header_+"mystery exception: "+e.toString());
}
}
And in the InputHandler thread:
#Override
public void run() {
boolean notRead;
byte i;
log_.write_log_line(log_header_+"Beginning InputHandler thread");
while (!stop) {
i = 0;
notRead = true;
nextInPacket = new byte[BUFFERSIZE];
readBytes = -1;
//StringBuilder s=new StringBuilder();
//TODO re-implement this with signals
while (i < READATTEMPTS && notRead) {
try {
// Make sure to adjust packet size. Done manually here for speed.
readBytes = is_.read(nextInPacket, 0, BUFFERSIZE);
/* Debugging
for (int j=0;j<nextInPacket.length;j++)
s.append(Byte.toString(nextInPacket[j]));
log_.write_log_line(log_header_+s.toString());
*/
if (readBytes != -1) {
notRead = false;
nextInPacket= new byte[]{1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0};
synchronized (q_) {
q_.add(nextInPacket);
}
//log_.write_log_line(log_header_ + "Incoming packet contains valid data.");
} else i++;
} catch (IOException e) {
log_.write_log_line(log_header_ + "mystery exception:\n\t" + e.toString());
}
}
if (i>=READATTEMPTS)
log_.write_log_line(log_header_+"Too many read attempts from input stream.");
/*
try {
sleep(100);
} catch (InterruptedException e) {
log_.write_log_line(log_header_+"fuck");
}
*/
}
}
On an oscilloscope, pins 1 and 2 both read an oscillating voltage, albeit at a very high amplitude, which is of some concern. Point is nothing is available to be read from the InputStream in the InputHandler class. Any ideas?

-1 returned from read() should only happen whenever the UART is closed. The closure can happen as result of explicitly calling close() on the Uart object or calling softReset() on the IOIO object.
The Android log might give you some clues about what's going on.
The reading you're seeing on the oscilloscope is suspicious: how high is "very high amplitude"? You should only ever see 0V or 3.3V on those pins, or floating in case the pins where not opened (or closed) for some reason.

Delays while sending data with Java NIO

I need your advice on a Java NIO package. I have an issue with delays while sending packets over network. The original code is actually my port of the SFML book source code to Java, but here I'll show you only a minimal working example, where the problem is reproduced. Though this code does contain some pieces from SFML library (actually creating a window and an event loop), I believe this has no impact on the issue.
Here I'll show only parts of the code, full version is available here.
So, the program has two entities: Server and Client. If you start an application in a server mode, then a Server is created, starts to listen for new connections, and a new Client is automatically created and tries to connect to the Server. In client mode only a Client is created and connects to the Server.
The application also creates a new basic GUI window and starts an event loop, where everything happens.
The Client sends packets to the Server. It handles them by just logging the fact of accepting. There are two types of packets the Client can send: periodical packet (with an incremental ID) and an event packet (application reacts to pressing SPACE or M buttons).
Client sends packets:
public void update(Time dt) throws IOException {
if (!isConnected) return;
if (tickClock.getElapsedTime().compareTo(Time.getSeconds(1.f / 20.f)) > 0) {
Packet intervalUpdatePacket = new Packet();
intervalUpdatePacket.append(PacketType.INTERVAL_UPDATE);
intervalUpdatePacket.append(intervalCounter++);
PacketReaderWriter.send(socketChannel, intervalUpdatePacket);
tickClock.restart();
}
}
public void handleEvent(Event event) throws IOException {
if (isConnected && (event.type == Event.Type.KEY_PRESSED)) {
KeyEvent keyEvent = event.asKeyEvent();
if (keyEvent.key == Keyboard.Key.SPACE) {
LOGGER.info("press SPACE");
Packet spacePacket = new Packet();
spacePacket.append(PacketType.SPACE_BUTTON);
PacketReaderWriter.send(socketChannel, spacePacket);
}
if (keyEvent.key == Keyboard.Key.M) {
LOGGER.info("press M");
Packet mPacket = new Packet();
mPacket.append(PacketType.M_BUTTON);
PacketReaderWriter.send(socketChannel, mPacket);
}
}
}
Server accepts packets:
private void handleIncomingPackets() throws IOException {
readSelector.selectNow();
Set<SelectionKey> readKeys = readSelector.selectedKeys();
Iterator<SelectionKey> it = readKeys.iterator();
while (it.hasNext()) {
SelectionKey key = it.next();
it.remove();
SocketChannel channel = (SocketChannel) key.channel();
Packet packet = null;
try {
packet = PacketReaderWriter.receive(channel);
} catch (NothingToReadException e) {
e.printStackTrace();
}
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
}
}
private void handleIncomingPacket(Packet packet, SocketChannel channel) {
PacketType packetType = (PacketType) packet.get();
switch (packetType) {
case INTERVAL_UPDATE:
int intervalId = (int) packet.get();
break;
case SPACE_BUTTON:
LOGGER.info("handling SPACE button");
break;
case M_BUTTON:
LOGGER.info("handling M button");
break;
}
}
Here is a PacketReaderWriter object:
package server;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
public class PacketReaderWriter {
private static final int PACKET_SIZE_LENGTH = 4;
private static final ByteBuffer packetSizeReadBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH);
private static ByteBuffer clientReadBuffer;
private static byte[] encode(Packet packet) throws IOException {
try (
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos)
) {
oos.writeObject(packet);
return baos.toByteArray();
}
}
private static Packet decode(byte[] encodedPacket) throws IOException, ClassNotFoundException {
try (ObjectInputStream oi = new ObjectInputStream(new ByteArrayInputStream(encodedPacket))) {
return (Packet) oi.readObject();
}
}
public static void send(SocketChannel channel, Packet packet) throws IOException {
byte[] encodedPacket = encode(packet);
ByteBuffer packetSizeBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH).putInt(encodedPacket.length);
packetSizeBuffer.flip();
// Send packet size
channel.write(packetSizeBuffer);
// Send packet content
ByteBuffer packetBuffer = ByteBuffer.wrap(encodedPacket);
channel.write(packetBuffer);
}
public static Packet receive(SocketChannel channel) throws IOException, NothingToReadException {
int bytesRead;
// Read packet size
packetSizeReadBuffer.clear();
bytesRead = channel.read(packetSizeReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
packetSizeReadBuffer.flip();
int packetSize = packetSizeReadBuffer.getInt();
// Read packet
clientReadBuffer = ByteBuffer.allocate(packetSize);
bytesRead = channel.read(clientReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
clientReadBuffer.flip();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.write(clientReadBuffer.array(), 0, bytesRead);
clientReadBuffer.clear();
try {
return decode(baos.toByteArray());
} catch (ClassNotFoundException e) {
e.printStackTrace();
return null;
}
}
}
And here is the problem: I have quite big delays between pressing a button (and sending a corresponding packet from the Client) and accepting this packet on the Server. If I start a new instance of the application in a client mode (just add a new Client in short), the delays become even bigger.
I don’t see any reason why these periodical packets create so much network load that other packets just cannot get through, but maybe I'm just missing something. Here I have to say that I’m not a Java expert, so don’t blame me too much for not seeing something obvious :)
Does anyone have any ideas?
Thanks!

I decided to take a look at the Github repo.
Your Server.run() looks like this.
public void run() {
while (isRunning) {
try {
handleIncomingConnections();
handleIncomingPackets();
} catch (IOException e) {
e.printStackTrace();
}
try {
// Sleep to prevent server from consuming 100% CPU
sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The sleep(100) will result in approximately 10 calls to handleIncomingPackets() per second. handleIncomingPackets() in turn will select a Client channel and call handleIncomingPacket() on a single received Packet. In total the server will be able to handle 10 Packets/second per Client if I understand it correctly.
The Client on the other hand tries to send 20 packets per second of the type PacketType.INTERVAL_UPDATE. Either the Client must send fewer packets per second or the Server needs to be able to handle more packets per second.
The current sleep(100) means that there will always be a latency of up to around 100ms before the server can respond to a single packet, even in a non-overloaded situation. This might be fine though if you make sure you really read all packets available on the channel instead of just a single one each time.
In summary: the smallest change you'd have to do to improve response times is to decrease the sleep() time. 10 ms would be fine. But I'd also suggest trying to check if there's more than one packet available in each iteration.
Update:
In the c++ file you linked my hunch is that it's reading more than one packet per iteration.
<snip>
while (peer->socket.receive(packet) == sf::Socket::Done)
{
// Interpret packet and react to it
handleIncomingPacket(packet, *peer, detectedTimeout);
</snip>
The while loop will read all available packets. Compared to your Java version where you read a single packet per client per server iteration.
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
You need to make sure that you read all available packets the Java version also.
If you just want to convince yourself that the client code sends more packets than the server code can handle it's quickly done by setting the sleep() to 10 ms temporarily.

Can't detect disconnect without extra readLine() loop

I am developing a program that uses sockets and currently I have a function in my code that checks for a heartbeat from the client every second.
private void userLoop() { // checks for incoming data from client
Timer t = new Timer();
t.schedule(new TimerTask() {
#Override
public void run() {
try {
socketIn.read(); // check for heartbeat from client
String userInput;
while ((userInput = br.readLine()) != null) {
}
} catch (Exception e) {
ControlPanel.model.removeElement(getUsername());
ControlPanel.append(getUsername() + " has disconnected.");
}
}
}, 1000);
}
When a client closes the game via the X button, shutting off their computer, logging out, whatever it may be, I get the message "'username' has disconnected". This is exactly what I want, however, it only works with the while loop in the code. The while loop essentially does nothing and I have no idea why it doesn't work with out.
If I remove the while loop and I disconnect using my client nothing gets printed out server sided.
String userInput;
while ((userInput = br.readLine()) != null) {
}
The above is essentially the dead code that does nothing but without it my program doesn't work the way it should..
Why is the code needed and how can I remove it and still make my program work correctly?

In this case, your while loop is essentially stalling your program until you no longer receive an input string. It's not dead code; it is just your way of installing a wait.
Otherwise, based on my understanding in the Timer class, it only waits one second, which might be too short of a timespan for what you're waiting to capture.

I fixed my problem by changing everything in the try block with
br.readLine();

There's a saying I've heard about exception handling: "Exceptions should only be used for exceptional situations." A client disconnecting from a server is not exceptional.
Now that I have that off my chest, let's move on. According to this other question,
socket.getInputSteam.read() does not throw when I close the socket from the client
it sounds like the read call won't throw if you're closing things properly on the client side.

The problem is that when the remote socket is closed, read() does not throw an Exception, it just returns -1 to signal the end of the stream.
The following should work without needing to call readLine():
try {
int ret = socketIn.read(); // check for heartbeat from client
if (ret == -1) {
// Remote side closed gracefully
clientDisconnected();
}
} catch (SocketTimeoutException e) {
// Timeout -- handle as required
handleTimeout();
} catch (IOException e) {
// Connection lost due to I/O error
clientDisconnected()
}

C++ TCP Winsock Server receiving the same packet many times

I have written a very simple C++ server which I am connecting to from a Java application. The C++ server uses winsock2. I am sending UTF8 encoded numbers to the server from my client and on receipt of these numbers I would like the server to perform an action. However my server seems to be receiving a series of numbers as one. At the moment I have the server listening every 1 millisecond for a new message.
This is my C++ server code which receives the message:
bool receive()
{
char buffer[1024];
int inDataLength=recv(Socket,buffer,sizeof(buffer),0);
if(buffer[0] != '\0')
{
std::cout<<"Client: ";
std::cout << buffer;
sendKey(string(buffer));
}
else if (inDataLength == 0) //Properly closed connection
{
std::cout<<"Connection lost..\r\n";
return false;
}
return true;
}
This is called within a loop like so:
while ( receive() )
{
Sleep(1);
}
This is my java client code to send a message where out is OutputStream = socket.getOutputStream():
public void send(String msg)
{
try {
out.write( msg.getBytes("UTF8") );
out.flush();
Thread.sleep(100);
} catch (SocketException e) {
Global.error("Connection error..");
//e.printStackTrace();
} catch (UnsupportedEncodingException e) {
} catch (IOException e) {
Global.error("Never connected..\r\n");
} catch (Exception e)
{ Global.error("Sending failed..\r\n"); }
}
What I am getting is the server receiving for example the number 1, then 2, then 12, then 121 etc.. in no specific pattern except once the server is receiving 2 numbers at once it will never start receiving only one again. This is the only place in my java code where anything is sent to the server and I flush the buffer after each message so I think the issue is on my server but I'm at a loss as to the problem.
Any help would be much appreciated.
Thanks.

You are forgetting the most important check:
int inDataLength=recv(Socket,buffer,sizeof(buffer),0);
if (inDataLength == -1 ) {
std::cerr << "receive error: " << GetLastError() << std::endl;
return false;
}
...
This actually might be the reason your loop took so much CPU time.

Given the sleeps it seems as the recv is not blocking.. Take a look at Winsock recv() does not block. You need to check for errors in the return value.

How to unblock InputStream.read() on Android?

I have a thread in which the read() method of an InputStream is called in a loop. When there are no more bytes to read, the stream will block until new data arrives.
If I call close() on the InputStream from a different thread, the stream gets closed, but the blocked read() call still remains blocked. I would assume that the read() method should now return with a value of -1 to indicate the end of the stream, but it does not. Instead, it stays blocked for several more minutes until a tcp timeout occurs.
How do I unblock the close() call?
Edit:
Apparently, the regular JRE will throw a SocketException immediately when the stream or socket the blocking read() call corresponds to is close()'d. The Android Java runtime which I am using, however, will not.
Any hints on a solution for the Android environment would be greatly appreciated.

Only call read() when there is data available.
Do something like that:
while( flagBlock )
{
if( stream.available() > 0 )
{
stream.read( byteArray );
}
}
set the flagBlock to stop the reading.

See Java Concurrency In Practice for a really good system to cancel a thread when working with sockets. It uses a special executor (CancellingExecutor) and a special Callable (SocketUsingTask).

When the other end closes the connection your stream will return -1 on a read(). If you cannot trigger the other end to close the connection e.g. by closing your output stream, you can close the socket which will cause an IOException in the blocking read() thread.
Can you provide a short example which reproduces your problem?
ServerSocket ss = new ServerSocket(0);
final Socket client = new Socket("localhost", ss.getLocalPort());
Socket server = ss.accept();
Thread t = new Thread(new Runnable() {
public void run() {
int ch;
try {
while ((ch = client.getInputStream().read()) != -1)
System.out.println(ch);
} catch (SocketException se) {
System.out.println(se);
} catch (IOException e) {
e.printStackTrace();
}
}
});
t.start();
server.getOutputStream().write("hi\n".getBytes());
Thread.sleep(100);
client.close();
t.join();
server.close();
ss.close();
prints
104
105
10
java.net.SocketException: Socket closed

We were having the same issue: no exception when switching network (e.g. switching from 3G to WiFi while downloading).
We are using the code from http://www.androidsnippets.com/download-an-http-file-to-sdcard-with-progress-notification, which is working perfectly except in some cases when the network connection was lost.
The solution was specifying a timeout value, this is set standard to 0 (meaning: wait infinitely).
HttpURLConnection c = (HttpURLConnection) u.openConnection();
c.setRequestMethod("GET");
c.setDoOutput(true);
c.setReadTimeout(1000);
c.connect();
Experiment with a timeout value appropriate for you.

I had such issue on Samsung 2.3. When switching from 3G to Wifi InputStream.read() method blocks. I tried all tips from this topic. Nothing helped. From my prospective this is device specific issue because it should throw IOException due to javadoc. My solution is to listen for android broadcast android.net.conn.CONNECTIVITY_CHANGE and close connection from another thread it will cause IOException in blocked thread.
Here is code example:
DownloadThread.java
private volatile boolean canceled;
private volatile InputStream in;
private boolean downloadFile(final File file, final URL url, long totalSize) {
OutputStream out = null;
try {
Log.v(Common.TAG, "DownloadThread: downloading to " + file);
in = (InputStream) url.getContent();
out = new FileOutputStream(file);
return copy(out, totalSize);
} catch (Exception e) {
Log.e(Common.TAG, "DownloadThread: Exception while downloading. Returning false ", e);
return false;
} finally {
closeStream(in);
closeStream(out);
}
}
public void cancelDownloading() {
Log.e(Common.TAG, "DownloadThread: cancelDownloading ");
canceled = true;
closeStream(in); //on my device this is the only way to unblock thread
}
private boolean copy(final OutputStream out, long totalSize) throws IOException {
final int BUFFER_LENGTH = 1024;
final byte[] buffer = new byte[BUFFER_LENGTH];
long totalRead = 0;
int progress = 0;
int read;
while (!canceled && (read = in.read(buffer)) != -1) {
out.write(buffer, 0, read);
totalRead += read;
}
return !canceled;
}

You could use java.nio package. NIO stands for Non-blocking IO. Here the calls (to say read & write) aren't blocked. This way you can close the stream.
There is a sample program you can look at here. Method: processRead