We Keep Coding

File transferred via TFTP has different size than on host

For quite long time now I'm struggling with handling TFTP protocol in my Android app. Its main feature is downloading files from custom designed device which hosts TFTP server.
I was browsing internet hoping to find some good, already written, implementation. First I've tried with TFTP library which is part of Apache Commons. Unfortunately no luck - constant timeouts or even complete freeze. After some further research I found some code on github - please take a look. I've adopted code to Android and after some tweaking I managed to finally receive some files.
Creator of the device stated, that block size should be exactly 1015 bytes. So I increased package size to 1015 and updated creating read request packet method:
DatagramPacket createReadRequestPacket(String strFileName) {
byte[] filename = strFileName.getBytes();
byte[] mode = currentMode.getBytes();
int len = rOpCode.length + filename.length + mode.length + 2;
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(len);
try {
outputStream.write(rOpCode);
outputStream.write(filename);
byte term = 0;
outputStream.write(term);
outputStream.write(mode); // "octet"
outputStream.write(term);
outputStream.write("blksize".getBytes());
outputStream.write(term);
outputStream.write("1015".getBytes());
outputStream.write(term);
} catch (IOException e) {
e.printStackTrace();
}
byte[] readPacketArray = outputStream.toByteArray();
return new DatagramPacket(readPacketArray, readPacketArray.length, serverAddr, port);
}
Chunks are being downloaded, but there is one major issue - files I'm downloading are in parts, 512kB each (except last one), and each part I receive on Android device is around 0,5kB larger. It seems like there is one byte more each time or one whole append more. Apparently I don't understand it completely and I'm missing something.
This is my method for file receiving:
byte previousBlockNumber = (byte) -1;
try {
PktFactory pktFactory;
DatagramSocket clientSocket;
byte[] buf;
DatagramPacket sendingPkt;
DatagramPacket receivedPkt;
System.out.print(ftpHandle);
if (isConnected) {
System.out.println("You're already connected to " + hostname.getCanonicalHostName());
}
try {
hostname = InetAddress.getByName(host);
if (!hostname.isReachable(4000)) {
System.out.println("Hostname you provided is not responding. Try again.");
return false;
}
} catch (UnknownHostException e) {
System.out.println("tftp: nodename nor servname provided, or not known");
return false;
}
clientSocket = new DatagramSocket();
pktFactory = new PktFactory(PKT_LENGTH + 4, hostname, TFTP_PORT);
System.out.println("Connecting " +
hostname.getCanonicalHostName() + " at the port number " + TFTP_PORT);
isConnected = true;
ftpHandle = "tftp#" + hostname.getCanonicalHostName() + "> ";
System.out.println("mode " + PktFactory.currentMode);
if (!isConnected) {
System.out.println("You must be connected first!");
}
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
buf = new byte[PKT_LENGTH + 4];
/* Sending the reading request with the filename to the server. **/
try {
/* Sending a RRQ with the filename. **/
System.out.println("Sending request to server.");
sendingPkt = pktFactory.createReadRequestPacket(filename);
clientSocket.setSoTimeout(4500);
clientSocket.send(sendingPkt);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Connection with server failed");
}
boolean receivingMessage = true;
while (true) {
try {
receivedPkt = new DatagramPacket(buf, buf.length);
clientSocket.setSoTimeout(10000);
clientSocket.receive(receivedPkt);
byte[] dPkt = receivedPkt.getData();
byte[] ropCode = pktFactory.getOpCode(dPkt);
/* rPkt either a DATA or an ERROR pkt. If an error then print the error message and
* terminate the program finish get command. **/
if (ropCode[1] == 5) {
String errorMsg = pktFactory.getErrorMessage(dPkt);
System.out.println(errorMsg);
return false;
}
if (receivedPkt.getLength() < PKT_LENGTH + 4 && ropCode[1] == 3) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
outputStream.write(fileDataBytes);
if (isListFile) {
listBytes = outputStream.toByteArray();
} else {
FileOutputStream fstream = new FileOutputStream(Constants.EEG_DATA_PATH.concat("file.bin"), true);
// Let's get the last data pkt for the current transfering file.
fstream.write(outputStream.toByteArray());
fstream.close();
}
// It's time to send the last ACK message before Normal termination.
byte[] bNum = pktFactory.getBlockNum(dPkt);
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
disconnect();
return true;
}
if (ropCode[1] == 3) {
if (receivingMessage) {
System.out.println("Receiving the file now..");
receivingMessage = false;
}
byte[] bNum = pktFactory.getBlockNum(dPkt);
//I've added this if and it reduces file size a little (it was more than 0,5kB bigger)
if (previousBlockNumber != bNum[1]) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
previousBlockNumber = bNum[1];
outputStream.write(fileDataBytes);
}
/* For each received DATA pkt we need to send ACK pkt back. **/
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
}
} catch (SocketTimeoutException e) {
disconnect();
System.out.println("Server didn't respond and timeout occured.");
return false;
}
}
} catch (Exception e) {
System.out.println(e.getMessage());
return false;
}

I know what was wrong. That strange behavior was result of this line when last packet was received:
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
Returned array size was always equal to specified packet length, even if received data was smaller. In my case last packet was 0 bytes (+4 bytes for tftp), but even then extra 512 bytes was added to output stream.
To resolve this I overload mentioned method with extra parameter - actual size of received packet when received data size is higher than 4 bytes and lower than specified packet size (512 bytes). This change resulted with getting correct size of array for last packet, so received file has correct size at the end of the operation.

IOIO UART readback problems

I'm extending the BaseIOIOLooper to open up a UART device and send messages. I'm testing with a readback, where I send a packet over a line and receive that packet on another line and print it out. Because I don't want the InputStream.read() method to block, I am handling packet formation and input in a different thread. I have narrowed my problem down to the InputStream.read() method, which returns -1 (no bytes read, but no exception).
Here is what it looks like in the Looper thread:
#Override
protected void setup() throws ConnectionLostException, InterruptedException {
log_.write_log_line(log_header_ + "Beginning IOIO setup.");
// Initialize IOIO UART pins
// Input at pin 1, output at pin 2
try {
inQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
outQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
ioio_.waitForConnect();
uart_ = ioio_.openUart(1, 2, 38400, Uart.Parity.NONE, Uart.StopBits.ONE);
// Start InputHandler. Takes packets from ELKA on inQueue_
in_= new InputHandler(inQueue_, uart_.getInputStream());
in_.start();
// Start OutputHandler. Takes packets from subprocesses on outQueue_
out_= new OutputHandler(outQueue_);
out_.start();
// Get output stream
os_=uart_.getOutputStream();
// Set default target state
setTargetState(State.TRANSFERRING);
currInPacket_[0]=1; //Initial value to start transferring
log_.write_log_line(log_header_ + "IOIO setup complete.\n\t" +
"Input pin set to 1\n\tOutput pin set to 2\n\tBaud rate set to 38400\n\t" +
"Parity set to even\n\tStop bits set to 1");
} catch (IncompatibilityException e) {
log_.write_log_line(log_header_+e.toString());
} catch (ConnectionLostException e) {
log_.write_log_line(log_header_+e.toString());
} catch (Exception e) {
log_.write_log_line(log_header_+"mystery exception: "+e.toString());
}
}
And in the InputHandler thread:
#Override
public void run() {
boolean notRead;
byte i;
log_.write_log_line(log_header_+"Beginning InputHandler thread");
while (!stop) {
i = 0;
notRead = true;
nextInPacket = new byte[BUFFERSIZE];
readBytes = -1;
//StringBuilder s=new StringBuilder();
//TODO re-implement this with signals
while (i < READATTEMPTS && notRead) {
try {
// Make sure to adjust packet size. Done manually here for speed.
readBytes = is_.read(nextInPacket, 0, BUFFERSIZE);
/* Debugging
for (int j=0;j<nextInPacket.length;j++)
s.append(Byte.toString(nextInPacket[j]));
log_.write_log_line(log_header_+s.toString());
*/
if (readBytes != -1) {
notRead = false;
nextInPacket= new byte[]{1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0};
synchronized (q_) {
q_.add(nextInPacket);
}
//log_.write_log_line(log_header_ + "Incoming packet contains valid data.");
} else i++;
} catch (IOException e) {
log_.write_log_line(log_header_ + "mystery exception:\n\t" + e.toString());
}
}
if (i>=READATTEMPTS)
log_.write_log_line(log_header_+"Too many read attempts from input stream.");
/*
try {
sleep(100);
} catch (InterruptedException e) {
log_.write_log_line(log_header_+"fuck");
}
*/
}
}
On an oscilloscope, pins 1 and 2 both read an oscillating voltage, albeit at a very high amplitude, which is of some concern. Point is nothing is available to be read from the InputStream in the InputHandler class. Any ideas?

-1 returned from read() should only happen whenever the UART is closed. The closure can happen as result of explicitly calling close() on the Uart object or calling softReset() on the IOIO object.
The Android log might give you some clues about what's going on.
The reading you're seeing on the oscilloscope is suspicious: how high is "very high amplitude"? You should only ever see 0V or 3.3V on those pins, or floating in case the pins where not opened (or closed) for some reason.

Delays while sending data with Java NIO

I need your advice on a Java NIO package. I have an issue with delays while sending packets over network. The original code is actually my port of the SFML book source code to Java, but here I'll show you only a minimal working example, where the problem is reproduced. Though this code does contain some pieces from SFML library (actually creating a window and an event loop), I believe this has no impact on the issue.
Here I'll show only parts of the code, full version is available here.
So, the program has two entities: Server and Client. If you start an application in a server mode, then a Server is created, starts to listen for new connections, and a new Client is automatically created and tries to connect to the Server. In client mode only a Client is created and connects to the Server.
The application also creates a new basic GUI window and starts an event loop, where everything happens.
The Client sends packets to the Server. It handles them by just logging the fact of accepting. There are two types of packets the Client can send: periodical packet (with an incremental ID) and an event packet (application reacts to pressing SPACE or M buttons).
Client sends packets:
public void update(Time dt) throws IOException {
if (!isConnected) return;
if (tickClock.getElapsedTime().compareTo(Time.getSeconds(1.f / 20.f)) > 0) {
Packet intervalUpdatePacket = new Packet();
intervalUpdatePacket.append(PacketType.INTERVAL_UPDATE);
intervalUpdatePacket.append(intervalCounter++);
PacketReaderWriter.send(socketChannel, intervalUpdatePacket);
tickClock.restart();
}
}
public void handleEvent(Event event) throws IOException {
if (isConnected && (event.type == Event.Type.KEY_PRESSED)) {
KeyEvent keyEvent = event.asKeyEvent();
if (keyEvent.key == Keyboard.Key.SPACE) {
LOGGER.info("press SPACE");
Packet spacePacket = new Packet();
spacePacket.append(PacketType.SPACE_BUTTON);
PacketReaderWriter.send(socketChannel, spacePacket);
}
if (keyEvent.key == Keyboard.Key.M) {
LOGGER.info("press M");
Packet mPacket = new Packet();
mPacket.append(PacketType.M_BUTTON);
PacketReaderWriter.send(socketChannel, mPacket);
}
}
}
Server accepts packets:
private void handleIncomingPackets() throws IOException {
readSelector.selectNow();
Set<SelectionKey> readKeys = readSelector.selectedKeys();
Iterator<SelectionKey> it = readKeys.iterator();
while (it.hasNext()) {
SelectionKey key = it.next();
it.remove();
SocketChannel channel = (SocketChannel) key.channel();
Packet packet = null;
try {
packet = PacketReaderWriter.receive(channel);
} catch (NothingToReadException e) {
e.printStackTrace();
}
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
}
}
private void handleIncomingPacket(Packet packet, SocketChannel channel) {
PacketType packetType = (PacketType) packet.get();
switch (packetType) {
case INTERVAL_UPDATE:
int intervalId = (int) packet.get();
break;
case SPACE_BUTTON:
LOGGER.info("handling SPACE button");
break;
case M_BUTTON:
LOGGER.info("handling M button");
break;
}
}
Here is a PacketReaderWriter object:
package server;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
public class PacketReaderWriter {
private static final int PACKET_SIZE_LENGTH = 4;
private static final ByteBuffer packetSizeReadBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH);
private static ByteBuffer clientReadBuffer;
private static byte[] encode(Packet packet) throws IOException {
try (
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos)
) {
oos.writeObject(packet);
return baos.toByteArray();
}
}
private static Packet decode(byte[] encodedPacket) throws IOException, ClassNotFoundException {
try (ObjectInputStream oi = new ObjectInputStream(new ByteArrayInputStream(encodedPacket))) {
return (Packet) oi.readObject();
}
}
public static void send(SocketChannel channel, Packet packet) throws IOException {
byte[] encodedPacket = encode(packet);
ByteBuffer packetSizeBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH).putInt(encodedPacket.length);
packetSizeBuffer.flip();
// Send packet size
channel.write(packetSizeBuffer);
// Send packet content
ByteBuffer packetBuffer = ByteBuffer.wrap(encodedPacket);
channel.write(packetBuffer);
}
public static Packet receive(SocketChannel channel) throws IOException, NothingToReadException {
int bytesRead;
// Read packet size
packetSizeReadBuffer.clear();
bytesRead = channel.read(packetSizeReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
packetSizeReadBuffer.flip();
int packetSize = packetSizeReadBuffer.getInt();
// Read packet
clientReadBuffer = ByteBuffer.allocate(packetSize);
bytesRead = channel.read(clientReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
clientReadBuffer.flip();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.write(clientReadBuffer.array(), 0, bytesRead);
clientReadBuffer.clear();
try {
return decode(baos.toByteArray());
} catch (ClassNotFoundException e) {
e.printStackTrace();
return null;
}
}
}
And here is the problem: I have quite big delays between pressing a button (and sending a corresponding packet from the Client) and accepting this packet on the Server. If I start a new instance of the application in a client mode (just add a new Client in short), the delays become even bigger.
I don’t see any reason why these periodical packets create so much network load that other packets just cannot get through, but maybe I'm just missing something. Here I have to say that I’m not a Java expert, so don’t blame me too much for not seeing something obvious :)
Does anyone have any ideas?
Thanks!

I decided to take a look at the Github repo.
Your Server.run() looks like this.
public void run() {
while (isRunning) {
try {
handleIncomingConnections();
handleIncomingPackets();
} catch (IOException e) {
e.printStackTrace();
}
try {
// Sleep to prevent server from consuming 100% CPU
sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The sleep(100) will result in approximately 10 calls to handleIncomingPackets() per second. handleIncomingPackets() in turn will select a Client channel and call handleIncomingPacket() on a single received Packet. In total the server will be able to handle 10 Packets/second per Client if I understand it correctly.
The Client on the other hand tries to send 20 packets per second of the type PacketType.INTERVAL_UPDATE. Either the Client must send fewer packets per second or the Server needs to be able to handle more packets per second.
The current sleep(100) means that there will always be a latency of up to around 100ms before the server can respond to a single packet, even in a non-overloaded situation. This might be fine though if you make sure you really read all packets available on the channel instead of just a single one each time.
In summary: the smallest change you'd have to do to improve response times is to decrease the sleep() time. 10 ms would be fine. But I'd also suggest trying to check if there's more than one packet available in each iteration.
Update:
In the c++ file you linked my hunch is that it's reading more than one packet per iteration.
<snip>
while (peer->socket.receive(packet) == sf::Socket::Done)
{
// Interpret packet and react to it
handleIncomingPacket(packet, *peer, detectedTimeout);
</snip>
The while loop will read all available packets. Compared to your Java version where you read a single packet per client per server iteration.
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
You need to make sure that you read all available packets the Java version also.
If you just want to convince yourself that the client code sends more packets than the server code can handle it's quickly done by setting the sleep() to 10 ms temporarily.

How to manage lots of incoming packets

I have a socketserver set up with a remote client, and it is functional. Upon opening the client and logging in, I noticed that sometimes, there is an error that seems to be due to the client reading an int when it shouldn't be.
Upon logging on, the server sends a series of messages/packets to the client, and these are anything from string messages to information used to load variables on the client's side.
Occasionally, while logging in, an error gets thrown showing that the client has read a packet of size 0 or a very large size. Upon converting the large-sized number into ascii I once found that it was a bit of a string "sk." (I located this string in my code so it's not entirely random).
Looking at my code, I'm not sure why this is happening. Is it possible that the client is reading an int at the wrong time? If so, how can I fix this?
InetAddress address = InetAddress.getByName(host);
connection = new Socket(address, port);
in = new DataInputStream(connection.getInputStream());
out = new DataOutputStream(connection.getOutputStream());
String process;
System.out.println("Connecting to server on "+ host + " port " + port +" at " + timestamp);
process = "Connection: "+host + ","+port+","+timestamp + ". Version: "+version;
write(0, process);
out.flush();
while (true) {
int len = in.readInt();
if (len < 2 || len > 2000) {
throw new Exception("Invalid Packet, length: "+len+".");
}
byte[] data = new byte[len];
in.readFully(data);
for (Byte b : data) {
System.out.printf("0x%02X ",b);
}
try {
reader.handlePackets(data);
} catch (Exception e) {
e.printStackTrace();
//connection.close();
//System.exit(0);
//System.out.println("Exiting");
}
}
//Here is code for my write function (Server sided):
public static void write(Client c, Packet pkt) {
for (Client client : clients) {
if (c.equals(client)) {
try {
out.writeInt(pkt.size());
out.write(pkt.getBytes());
out.flush();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
So looking at the write function, I don't really see how it could be confusing the client and making it read for the size of the packet twice for one packet (at least that's what I think is happening).
If you need more information please ask me.

The client side code looks fine, and the server side code looks fine too.
The most likely issue is that this is some kind of issue with multi-threading and (improper) synchronization. For example, maybe two server-side threads are trying to write a packet to the same client at the same time.
It is also possible that your Packet class has inconsistent implementations of size() and getBytes() ... or that one thread is modifying a Packet objects while a second one is sending it.

File transfer from C++ client to Java server

I have a c++ client which needs to send a file to a c++ server. I'm splitting the file to chunks of PACKET_SIZE (=1024) bytes and send them over a TCP socket. At the server side I read at most PACKET_SIZE bytes to a buffer. When the client sends files which are less than PACKET_SIZE, the server receives more bytes than sent. Even when I limit the number of bytes to be exactly the size of the file, the files differ. I know the problem does not have to do with the client because I've tested it with a c++ server and it works flawlessly.
Thanks.
Server:
public void run() {
DataInputStream input = null;
PrintWriter output = null;
try {
input = new DataInputStream (_client.getInputStream());
}
catch (Exception e) {/* Error handling code */}
FileHeader fh = recvHeader(input);
size = fh._size;
filename = fh._name;
try {
output = new PrintWriter(_client.getOutputStream(), true);
}
catch (Exception e) {/* Error handling code */}
output.write(HEADER_ACK);
output.flush();
FileOutputStream file = null;
try {
file = new FileOutputStream(filename);
}
catch (FileNotFoundException fnfe) {/* Error handling code */}
int total_bytes_rcvd = 0, bytes_rcvd = 0, packets_rcvd = 0;
byte [] buf = new byte [PACKET_DATA_SIZE];
try {
int max = (size > PACKET_DATA_SIZE)? PACKET_DATA_SIZE: size;
bytes_rcvd = input.read(buf,0, max);
while (total_bytes_rcvd < size) {
if (-1 == bytes_rcvd) {...}
++packets_rcvd;
total_bytes_rcvd += bytes_rcvd;
file.write (buf,0, bytes_rcvd);
if (total_bytes_rcvd < size)
bytes_rcvd = input.read(buf);
}
file.close();
}
catch (Exception e) {/* Error handling code */}
}
Client:
char packet [PACKET_SIZE] ;
file.open (filename, ios::in | ios::binary);//fopen (file_path , "rb");
int max = 0;
if (file.is_open()) {
if (size > PACKET_SIZE)
max = PACKET_SIZE;
else
max = size;
file.read (packet , max);
}
else {...}
int sent_packets = 0;
while (sent_packets < (int) ceil (((float)size)/PACKET_SIZE) ) {
_write=send(_sd , packet, max,0);
if (_write <0) {...}
else {
++sent_packets;
if (size > PACKET_SIZE* sent_packets) {
if (size - PACKET_SIZE* sent_packets >= PACKET_SIZE)
max = PACKET_SIZE;
else
max = size - PACKET_SIZE* sent_packets;
file.read (packet , max);
}
}
}

Is the sending socket closed at the end of the file, or is the next file streamed over the same socket? If more than one file is streamed, you could pick up data from the next file if you have the endedness wrong for the file size in recvHeader(), i.e. you send a file of length 0x0102 and try to read one of length 0x0201.
Other question, why do you provide a max for the first read, but not for the following reads on the same file?

One issue I see is that it appears that you assume that if the send returns a non-error, that it sent the entire chunk you requested it to send. This is not necessarily true, especially with stream sockets. How large are the packets you are sending, and how many? The most likely reason this could occur would be if the sndbuf for the socket filled, and your socket _sd is set to non-blocking. I'm not positive (depends on stack implementation), but I believe it could also likely occur if the TCP transmit window was full for your connection, and tcp couldn't enqueue your entire packet.
You should probably loop on the send until max is sent.
Thusly:
int send_ct=0;
while( (_write = send(_sd, packet + send_ct, max-send_ct, 0)) > 0) {
send_ct += _write;
if(send_ct >= max) {
break;
} else {
// Had to do another send
}
}

the code is not complete. E.g. you have omitted the sending of the filename and the filesize, as well as the parsing of those values. Are those values correct? If not first ensure that these values are the right ones before investigating further.