We Keep Coding

File transferred via TFTP has different size than on host

For quite long time now I'm struggling with handling TFTP protocol in my Android app. Its main feature is downloading files from custom designed device which hosts TFTP server.
I was browsing internet hoping to find some good, already written, implementation. First I've tried with TFTP library which is part of Apache Commons. Unfortunately no luck - constant timeouts or even complete freeze. After some further research I found some code on github - please take a look. I've adopted code to Android and after some tweaking I managed to finally receive some files.
Creator of the device stated, that block size should be exactly 1015 bytes. So I increased package size to 1015 and updated creating read request packet method:
DatagramPacket createReadRequestPacket(String strFileName) {
byte[] filename = strFileName.getBytes();
byte[] mode = currentMode.getBytes();
int len = rOpCode.length + filename.length + mode.length + 2;
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(len);
try {
outputStream.write(rOpCode);
outputStream.write(filename);
byte term = 0;
outputStream.write(term);
outputStream.write(mode); // "octet"
outputStream.write(term);
outputStream.write("blksize".getBytes());
outputStream.write(term);
outputStream.write("1015".getBytes());
outputStream.write(term);
} catch (IOException e) {
e.printStackTrace();
}
byte[] readPacketArray = outputStream.toByteArray();
return new DatagramPacket(readPacketArray, readPacketArray.length, serverAddr, port);
}
Chunks are being downloaded, but there is one major issue - files I'm downloading are in parts, 512kB each (except last one), and each part I receive on Android device is around 0,5kB larger. It seems like there is one byte more each time or one whole append more. Apparently I don't understand it completely and I'm missing something.
This is my method for file receiving:
byte previousBlockNumber = (byte) -1;
try {
PktFactory pktFactory;
DatagramSocket clientSocket;
byte[] buf;
DatagramPacket sendingPkt;
DatagramPacket receivedPkt;
System.out.print(ftpHandle);
if (isConnected) {
System.out.println("You're already connected to " + hostname.getCanonicalHostName());
}
try {
hostname = InetAddress.getByName(host);
if (!hostname.isReachable(4000)) {
System.out.println("Hostname you provided is not responding. Try again.");
return false;
}
} catch (UnknownHostException e) {
System.out.println("tftp: nodename nor servname provided, or not known");
return false;
}
clientSocket = new DatagramSocket();
pktFactory = new PktFactory(PKT_LENGTH + 4, hostname, TFTP_PORT);
System.out.println("Connecting " +
hostname.getCanonicalHostName() + " at the port number " + TFTP_PORT);
isConnected = true;
ftpHandle = "tftp#" + hostname.getCanonicalHostName() + "> ";
System.out.println("mode " + PktFactory.currentMode);
if (!isConnected) {
System.out.println("You must be connected first!");
}
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
buf = new byte[PKT_LENGTH + 4];
/* Sending the reading request with the filename to the server. **/
try {
/* Sending a RRQ with the filename. **/
System.out.println("Sending request to server.");
sendingPkt = pktFactory.createReadRequestPacket(filename);
clientSocket.setSoTimeout(4500);
clientSocket.send(sendingPkt);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Connection with server failed");
}
boolean receivingMessage = true;
while (true) {
try {
receivedPkt = new DatagramPacket(buf, buf.length);
clientSocket.setSoTimeout(10000);
clientSocket.receive(receivedPkt);
byte[] dPkt = receivedPkt.getData();
byte[] ropCode = pktFactory.getOpCode(dPkt);
/* rPkt either a DATA or an ERROR pkt. If an error then print the error message and
* terminate the program finish get command. **/
if (ropCode[1] == 5) {
String errorMsg = pktFactory.getErrorMessage(dPkt);
System.out.println(errorMsg);
return false;
}
if (receivedPkt.getLength() < PKT_LENGTH + 4 && ropCode[1] == 3) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
outputStream.write(fileDataBytes);
if (isListFile) {
listBytes = outputStream.toByteArray();
} else {
FileOutputStream fstream = new FileOutputStream(Constants.EEG_DATA_PATH.concat("file.bin"), true);
// Let's get the last data pkt for the current transfering file.
fstream.write(outputStream.toByteArray());
fstream.close();
}
// It's time to send the last ACK message before Normal termination.
byte[] bNum = pktFactory.getBlockNum(dPkt);
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
disconnect();
return true;
}
if (ropCode[1] == 3) {
if (receivingMessage) {
System.out.println("Receiving the file now..");
receivingMessage = false;
}
byte[] bNum = pktFactory.getBlockNum(dPkt);
//I've added this if and it reduces file size a little (it was more than 0,5kB bigger)
if (previousBlockNumber != bNum[1]) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
previousBlockNumber = bNum[1];
outputStream.write(fileDataBytes);
}
/* For each received DATA pkt we need to send ACK pkt back. **/
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
}
} catch (SocketTimeoutException e) {
disconnect();
System.out.println("Server didn't respond and timeout occured.");
return false;
}
}
} catch (Exception e) {
System.out.println(e.getMessage());
return false;
}

I know what was wrong. That strange behavior was result of this line when last packet was received:
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
Returned array size was always equal to specified packet length, even if received data was smaller. In my case last packet was 0 bytes (+4 bytes for tftp), but even then extra 512 bytes was added to output stream.
To resolve this I overload mentioned method with extra parameter - actual size of received packet when received data size is higher than 4 bytes and lower than specified packet size (512 bytes). This change resulted with getting correct size of array for last packet, so received file has correct size at the end of the operation.

ServerSocket while not breaks on image data inputstream

I tried to send an image from One device to other Device using Bluetooth.For that I take Android Bluetooth chat application source code and it works fine when I send String.But If i send image as byte array the while loop not breaks or EOF not reached when read from Inputstream.
Model:1
It receives image properly.But here I need to pass resultByteArray length.But I dont know the length.How to know the length of byte array in inputstream? inputstream.available() returns 0.
while(true)
{
byte[] resultByteArray = new byte[150827];
DataInputStream dataInputStream = new DataInputStream(mmInStream);
dataInputStream.readFully(resultByteArray);
mHandler.obtainMessage(AppConstants.MESSAGE_READ, dataInputStream.available(),-1, resultByteArray).sendToTarget();
}
Model:2
In this code while loop not breaks,
ByteArrayOutputStream bao = new ByteArrayOutputStream();
byte[] resultByteArray = new byte[1024];
int bytesRead;
while ((bytesRead = mmInStream.read(resultByteArray)) != -1) {
Log.i("BTTest1", "bytesRead=>"+bytesRead);
bao.write(resultByteArray,0,bytesRead);
}
final byte[] data = bao.toByteArray();
Also tried byte[] resultByteArray = IOUtils.toByteArray(mmInStream);but it also not works.I followed Bluetooth chat sample.
How to solve this issue?

As noted in the comment, the server needs to put the length of image at front of the actual image data. And the length of the image length information should be fixed like 4 bytes.
Then in the while loop, you need to get 4 bytes first to figure out the length of the image. After that, read bytes of the exact length from the input stream. That is the actual image.
The while loop doesn't need to break during the connection is alive. Actually it needs to wait another image data in the same while loop. The InputStream.read() is a blocking function and the thread will be sleeping until it receives enough data from the input stream.
And then you can expect another 4 bytes right after the previous image data as a start of another image.
while(true) {
try {
// Get the length first
byte[] bytesLengthOfImage = new byte[4];
mmInStream.read(bytesLengthOfImage);
int lengthOfImage = 0;
{
ByteBuffer buffer = ByteBuffer.wrap(bytesLengthOfImage);
buffer.order(ByteOrder.BIG_ENDIAN); // Assume it is network byte order.
lengthOfImage = buffer.getInt();
}
byte[] actualImage = new byte[lengthOfImage]; // Mind the memory allocation.
mmInStream.read(actualImage);
mHandler.obtainMessage(AppConstants.MESSAGE_READ, lengthOfImage,-1, actualImage).sendToTarget();
} catch (Exception e) {
if(e instanceof IOException) {
// If the connection is closed, break the loop.
break;
}
else {
// Handle errors
break;
}
}
}
This is a kind of simplified communication protocol. There is an open source framework for easy protocol implementation, called NFCommunicator.
https://github.com/Neofect/NFCommunicator
It might be an over specificiation for a simple project, but is worth a look.

How to read (all available) data from serial connection when using JSSC?

I'm trying to work with JSSC.
I built my app according to this link:
https://code.google.com/p/java-simple-serial-connector/wiki/jSSC_examples
My event handler looks like:
static class SerialPortReader implements SerialPortEventListener {
public void serialEvent(SerialPortEvent event) {
if(event.isRXCHAR()){//If data is available
try {
byte buffer[] = serialPort.readBytes();
}
catch (SerialPortException ex) {
System.out.println(ex);
}
}
}
}
}
The problem is that I'm always not getting the incoming data in one piece. (I the message has a length of 100 bytes, Im getting 48 and 52 bytes in 2 separates calls)
- The other side send me messages in different lengths.
- In the ICD Im working with, there is a field which tell us the length of the message. (from byte #10 to byte #13)
- I cant read 14 bytes:
(serialPort.readBytes(14);,
parse the message length and read the rest of the message:
(serialPort.readBytes(messageLength-14);
But if I will do it, I will not have the message in once piece (I will have 2 separates byte[] and I need it in one piece (byte[]) without the work of copy function.
Is it possible ?
When working with Ethernet (SocketChannel) we can read data using ByteBuffer. But with JSSC we cant.
Is there a good alternative to JSSC ?
Thanks

You can't rely on any library to give you all the content you need at once because :
the library dont know how many data you need
the library will give you data as it comes and also depending on buffers, hardware, etc
You must develop your own business logic to handle your packets reception. It will of course depend on how your packets are defined : are they always the same length, are they separated with same ending character, etc.
Here is an example that should work with your system (note you should take this as a start, not a full solution, it doesn't include timeout for example) :
static class SerialPortReader implements SerialPortEventListener
{
private int m_nReceptionPosition = 0;
private boolean m_bReceptionActive = false;
private byte[] m_aReceptionBuffer = new byte[2048];
#Override
public void serialEvent(SerialPortEvent p_oEvent)
{
byte[] aReceiveBuffer = new byte[2048];
int nLength = 0;
int nByte = 0;
switch(p_oEvent.getEventType())
{
case SerialPortEvent.RXCHAR:
try
{
aReceiveBuffer = serialPort.readBytes();
for(nByte = 0;nByte < aReceiveBuffer.length;nByte++)
{
//System.out.print(String.format("%02X ",aReceiveBuffer[nByte]));
m_aReceptionBuffer[m_nReceptionPosition] = aReceiveBuffer[nByte];
// Buffer overflow protection
if(m_nReceptionPosition >= 2047)
{
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
else if(m_bReceptionActive)
{
m_nReceptionPosition++;
// Receive at least the start of the packet including the length
if(m_nReceptionPosition >= 14)
{
nLength = (short)((short)m_aReceptionBuffer[10] & 0x000000FF);
nLength |= ((short)m_aReceptionBuffer[11] << 8) & 0x0000FF00;
nLength |= ((short)m_aReceptionBuffer[12] << 16) & 0x00FF0000;
nLength |= ((short)m_aReceptionBuffer[13] << 24) & 0xFF000000;
//nLength += ..; // Depending if the length in the packet include ALL bytes from the packet or only the content part
if(m_nReceptionPosition >= nLength)
{
// You received at least all the content
// Reset for next packet
m_bReceptionActive = false;
m_nReceptionPosition = 0;
}
}
}
// Start receiving only if this is a Start Of Header
else if(m_aReceptionBuffer[0] == '\0')
{
m_bReceptionActive = true;
m_nReceptionPosition = 1;
}
}
}
catch(Exception e)
{
e.printStackTrace();
}
break;
default:
break;
}
}
}

After writing data to serial port it need to be flushed. Check the timing and pay attention to the fact that read should occur only after other end has written. read size is just an indication to read system call and is not guaranteed. The data may have arrived and is buffered in serial port hardware buffer but may not have been transferred to operating system buffer hence not to application. Consider using scm library, it flushes data after each write http://www.embeddedunveiled.com/

Try this:
Write your data to the serial port (using serialPort.writeBytes()) and if you are expecting a response, use this:
byte[] getData() throws SerialPortException, IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] b;
try {
while ((b = serialPort.readBytes(1, 100)) != null) {
baos.write(b);
// System.out.println ("Wrote: " + b.length + " bytes");
}
// System.out.println("Returning: " + Arrays.toString(baos.toByteArray()));
} catch (SerialPortTimeoutException ex) {
; //don't want to catch it, it just means there is no more data to read
}
return baos.toByteArray();
}
Do what you want with the returned byte array; in my case I just display it for testing.
I found it works just fine if you read one byte at a time, using a 100ms timeout, and when it does time out, you've read all data in the buffer.
Source: trying to talk to an Epson serial printer using jssc and ESC/POS.

Bluetooth transfer App stops after using InputStream.read() with no error

I am trying to make a file transfer Bluetooth app work using these sources:
http://developer.android.com/guide/topics/connectivity/bluetooth.html
https://android.googlesource.com/platform/development/+/25b6aed7b2e01ce7bdc0dfa1a79eaf009ad178fe/samples/BluetoothChat/
When I attempt to get the InputStream bytes using InputStream.read() method this way:
public class ConnectedThread extends Thread {
...(some code here)
public void run(){
byte[] buffer = new byte[1024];
int bytes = -1;
//Keep listening to the InputStream while connected
while (true){
try {
bytes = this.mmInStream.read(buffer);
//* this part is not reached
if (bytes==-1){
Log.d("NoData:","-1");
}
}
catch(Exception e){
Log.d("inStream exception:",e.getMessage());
break;
}
}
}
...(some code here)
}
The next part of the code ("if" part in this case) is never reached, nor a Log.D debug output or whatever else I put in following. I just get this message from LogCat:
BluetoothSocket read in: android.net.LocalStocketImpl$SocketInputStream#f7e
b08 len: 1024
To transfer the data from the Client to the Server I am doing this:
public class MainActivity extends Activity {
...(some code here)
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
clientConnect();
//serverConnect();
}
...(some code here)
public void clientConnect(){
Set<BluetoothDevice> devices;
devices = bConfig.getPairedDevices();
if (devices == null){
return;
}
if (devices.size() > 0) {
BluetoothDevice device = devices.iterator().next();
ConnectThread connectTransmit = new ConnectThread(device,bConfig.getBluetoothAdapter(),BluetoothConfig.mUUID);
connectTransmit.start();
Toast.makeText(this, "connected", Toast.LENGTH_SHORT).show();
socket = connectTransmit.mmSocket;
ConnectedThread connectedThread = new ConnectedThread(socket);
//write file bytes to the connected thread, so the thread can receive its own input written bytes later
File file_to_transfer = new File(Environment.getExternalStorageDirectory() + "/txtTransfer.txt");
//get bytes from our File
int size = (int) file_to_transfer.length();
byte[] bytes = new byte[size];
try {
//14b are read succesfully, the whole text file
BufferedInputStream buf = new BufferedInputStream(new FileInputStream(file_to_transfer));
buf.read(bytes,0,bytes.length);
buf.close();
}catch (FileNotFoundException e){
Log.d("FileNotFoundException:",e.getMessage());
}catch (IOException e){
Log.d("IOException:",e.getMessage());
}
//send the data to the server
connectedThread.start();
connectedThread.write(bytes);
//connectedThread.cancel();
}
}
...(some code here)
}
The AcceptThread (Server part of the implementation) works, because when I run the client part to connect and then transfer the data, while debuging in the device the LogCat on the Server part activates and reaches the run method of the thread, where I call the ConnectedThread implementation but then after it "apparently" reads the bytes but it gets stuck on LogCat with no error.
Please let me know what can I do to finish reading the bytes to move to the next part of the flow.
Thank you

You're blocked waiting for more input.
The part labelled ... (some code here) should be inside the read loop, after the test for end of stream. NB If read() returns -1 it doesn't mean 'no data', it means end of stream, and you should close the socket and break out of the read loop. Otherwise you should then go on to process the data you've just read. At present you just read and ignore all input until end of stream, which is pointless. At best you can only process the last partial buffer, and you won't know how long it was.

In my opinion You should verify if something is in buffer before reading.
reading from stream is blocking operation so aplication will hang until somehing data appear.
How can I check if an InputStream is empty without reading from it?

In your client code you should probably keep the connectedThread object alive a while longer. Might be that once the if clause closes and it goes out of scope (not quite sure what happens with GC and all) the write just doesn't happen and your connection is not closed but not used either.
Calling flush() on the mmOutStream inside the connectedThread after the write might help also.
Like #EJP suggested, you should put something inside your read loop.
Edit: For the sake of debugging you could add this.wait(1000); immediately after your write in the client code.

Try changing your run method to this:
public void run(){
byte[] buffer = new byte[1024];
int bytesRead = 0;
final int shortSleepTime = 1000;
final int longSleepTime = 5000;
int emptyReadCounter = 0;
int sleepCounter = 0;
int currentSleepTime = shortSleepTime;
//Keep listening to the InputStream while connected
while (bytesRead >= 0){
try {
// if available() returns 0, there is nothing to read yet
if (this.mmInStream.available() != 0){
bytesRead = this.mmInStream.read(buffer);
// Check if we need to reset the sleep counters
if (emptyReadCounter != 0){
emptyReadCounter = 0;
sleepCounter = 0;
currentSleepTime = shortSleepTime;
// We can also do anything else dependent on just waking up
// from a sleep cycle in this block
}
// Do something with my now full buffer
// Remember not to process more than
// 'bytesRead' bytes from my buffer because the
// rest could be filled with crap left over from
// the last iteration
} else {
// Three consecutive empty reads means sleep
if (emptyReadCounter++ >= 3){
if (currentSleepTime != longSleepTime && sleepCounter++ >= 3){
currentSleepTime = longSleepTime;
}
Thread.sleep(currentSleepTime);
}
}
}
catch(Exception e){
Log.d("inStream exception:",e.getMessage());
break;
}
}
}

Android bluetooth communication with Arduino

I am currently looking at Google's Bluetooth Chat example. The goal is to get communication between android and and Arduino working based on this example.
While communication from the smartphone to the Arduino is working great, the other direction does not:
When sending bytes from the Arduino to the smartphone, the following code is used for receiving:
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(MainActivity.MESSAGE_READ, bytes, -1, buffer).sendToTarget();
This has the following problems:
In my main activity, I get a handed a byte array that's always 1024 bytes long. No matter what the incoming byte length was. It would be really nice, if I had an idication how many bytes were received.
The bytes seem not to get read all at once. E.g. the code above is called multiple times, but the buffer NEVER contains all the bytes I sent from the Arduino. SOmetimes there is only the first bytes, then later only the last bytes.
Although this code is calles multiple times, my main activity gets only notified once. How can that be?
What is the right way to do this. Should one implement a mechanism that collects and concatenates the bytes? Or am I using this code the wrong way?

I always had trouble reading a byte buffer greater than one at a time. This is because there is no way to guarantee that you received all the bytes correctly. My work around was to call read repeatedly one byte at a time and fill out my buffer. That way if any of my bytes aren't read ill catch that in the I/O catch part of my connectedThread and can choose to deal with it however I want.
Sample connectedThread
private class ConnectedThread extends Thread {
private final BluetoothSocket mmSocket;
private final InputStream mmInStream;
private final OutputStream mmOutStream;
public ConnectedThread(BluetoothSocket socket) {
mmSocket = socket;
InputStream tmpIn = null;
OutputStream tmpOut = null;
// Get the input and output streams, using temp objects because
// member streams are final
try {
tmpIn = socket.getInputStream();
tmpOut = socket.getOutputStream();
} catch (IOException e) { }
mmInStream = tmpIn;
mmOutStream = tmpOut;
}
public void run() {
byte[] buffer; // buffer store for the stream
int bytes; // bytes returned from read()
// Keep listening to the InputStream until an exception occurs
while (true) {
try {
// Read from the InputStream
// You can define this buffer to be 1024 or anything you like
buffer = new byte[3];
mmOutStream.write(253);
bytes = mmInStream.read(buffer,0,1);
mmOutStream.write(254);
bytes = mmInStream.read(buffer,1,1);
mmOutStream.write(255);
bytes = mmInStream.read(buffer,2,1);
mHandler.obtainMessage(MESSAGE_READ, buffer).sendToTarget();
}
catch (IOException e) {
break;
}
}
}
/* Call this from the main activity to send data to the remote device */
public void write(byte[] bytes) {
try {
mmOutStream.write(bytes);
} catch (IOException e) { }
}
}
In this case I used a unsigned byte array to represent integers from 0-255. Furthermore I used values 255-253 as commands to tell my Arduino to send me certain types of information. You do not have to set any value to represent a command to arduino, instead you can just tell the arduino to loop through values it needs to send each time it receives a request for information. I found out this is one of the only ways to can confirm the amounts of bytes you received(i.e the size of your byte[] buffer).Although in this case I did not put anything in my catch statement for the connectedThread you could put a read command in there to confirm you receive a byte.
Message Handler
Here is how I dealt with the readBuffer...
/*
* Bluetooth Handler Method
*/
ConnectedThread connectedThread;
Handler mHandler = new Handler(){
public void handleMessage(Message msg){
super.handleMessage(msg);
switch(msg.what){
case SUCCESS_CONNECT:
// Do Something;
Toast.makeText(getActivity(),"CONNECTED",Toast.LENGTH_SHORT).show();
connectedThread = new ConnectedThread((BluetoothSocket)msg.obj);
listView.setVisibility(View.GONE);
connectedThread.start();
break;
case MESSAGE_READ:
byte[] readBuf = (byte[])msg.obj;
int tempInt = byteToInt(readBuf[0]);
int speedInt = byteToInt(readBuf[1]);
int cadenceInt = byteToInt(readBuf[2]);
EditText temperatureData = (EditText)getActivity().findViewById(R.id.temperatureData);
temperatureData.setText(Integer.toString(tempInt) + " C" );
EditText cadenceData = (EditText)getActivity().findViewById(R.id.cadence);
cadenceData.setText(Integer.toString(cadenceInt) + " rpm");
EditText speedData = (EditText)getActivity().findViewById(R.id.speed_data);
speedData.setText(Integer.toString(speedInt) + " kph");
}
}
};
In this case I was displaying live sensor data on my phone. But you can do anything really.
Hope that helped.