We Keep Coding

File transferred via TFTP has different size than on host

For quite long time now I'm struggling with handling TFTP protocol in my Android app. Its main feature is downloading files from custom designed device which hosts TFTP server.
I was browsing internet hoping to find some good, already written, implementation. First I've tried with TFTP library which is part of Apache Commons. Unfortunately no luck - constant timeouts or even complete freeze. After some further research I found some code on github - please take a look. I've adopted code to Android and after some tweaking I managed to finally receive some files.
Creator of the device stated, that block size should be exactly 1015 bytes. So I increased package size to 1015 and updated creating read request packet method:
DatagramPacket createReadRequestPacket(String strFileName) {
byte[] filename = strFileName.getBytes();
byte[] mode = currentMode.getBytes();
int len = rOpCode.length + filename.length + mode.length + 2;
ByteArrayOutputStream outputStream = new ByteArrayOutputStream(len);
try {
outputStream.write(rOpCode);
outputStream.write(filename);
byte term = 0;
outputStream.write(term);
outputStream.write(mode); // "octet"
outputStream.write(term);
outputStream.write("blksize".getBytes());
outputStream.write(term);
outputStream.write("1015".getBytes());
outputStream.write(term);
} catch (IOException e) {
e.printStackTrace();
}
byte[] readPacketArray = outputStream.toByteArray();
return new DatagramPacket(readPacketArray, readPacketArray.length, serverAddr, port);
}
Chunks are being downloaded, but there is one major issue - files I'm downloading are in parts, 512kB each (except last one), and each part I receive on Android device is around 0,5kB larger. It seems like there is one byte more each time or one whole append more. Apparently I don't understand it completely and I'm missing something.
This is my method for file receiving:
byte previousBlockNumber = (byte) -1;
try {
PktFactory pktFactory;
DatagramSocket clientSocket;
byte[] buf;
DatagramPacket sendingPkt;
DatagramPacket receivedPkt;
System.out.print(ftpHandle);
if (isConnected) {
System.out.println("You're already connected to " + hostname.getCanonicalHostName());
}
try {
hostname = InetAddress.getByName(host);
if (!hostname.isReachable(4000)) {
System.out.println("Hostname you provided is not responding. Try again.");
return false;
}
} catch (UnknownHostException e) {
System.out.println("tftp: nodename nor servname provided, or not known");
return false;
}
clientSocket = new DatagramSocket();
pktFactory = new PktFactory(PKT_LENGTH + 4, hostname, TFTP_PORT);
System.out.println("Connecting " +
hostname.getCanonicalHostName() + " at the port number " + TFTP_PORT);
isConnected = true;
ftpHandle = "tftp#" + hostname.getCanonicalHostName() + "> ";
System.out.println("mode " + PktFactory.currentMode);
if (!isConnected) {
System.out.println("You must be connected first!");
}
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
buf = new byte[PKT_LENGTH + 4];
/* Sending the reading request with the filename to the server. **/
try {
/* Sending a RRQ with the filename. **/
System.out.println("Sending request to server.");
sendingPkt = pktFactory.createReadRequestPacket(filename);
clientSocket.setSoTimeout(4500);
clientSocket.send(sendingPkt);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Connection with server failed");
}
boolean receivingMessage = true;
while (true) {
try {
receivedPkt = new DatagramPacket(buf, buf.length);
clientSocket.setSoTimeout(10000);
clientSocket.receive(receivedPkt);
byte[] dPkt = receivedPkt.getData();
byte[] ropCode = pktFactory.getOpCode(dPkt);
/* rPkt either a DATA or an ERROR pkt. If an error then print the error message and
* terminate the program finish get command. **/
if (ropCode[1] == 5) {
String errorMsg = pktFactory.getErrorMessage(dPkt);
System.out.println(errorMsg);
return false;
}
if (receivedPkt.getLength() < PKT_LENGTH + 4 && ropCode[1] == 3) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
outputStream.write(fileDataBytes);
if (isListFile) {
listBytes = outputStream.toByteArray();
} else {
FileOutputStream fstream = new FileOutputStream(Constants.EEG_DATA_PATH.concat("file.bin"), true);
// Let's get the last data pkt for the current transfering file.
fstream.write(outputStream.toByteArray());
fstream.close();
}
// It's time to send the last ACK message before Normal termination.
byte[] bNum = pktFactory.getBlockNum(dPkt);
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
disconnect();
return true;
}
if (ropCode[1] == 3) {
if (receivingMessage) {
System.out.println("Receiving the file now..");
receivingMessage = false;
}
byte[] bNum = pktFactory.getBlockNum(dPkt);
//I've added this if and it reduces file size a little (it was more than 0,5kB bigger)
if (previousBlockNumber != bNum[1]) {
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
previousBlockNumber = bNum[1];
outputStream.write(fileDataBytes);
}
/* For each received DATA pkt we need to send ACK pkt back. **/
DatagramPacket sPkt = pktFactory.createAckPacket(bNum, receivedPkt.getPort());
clientSocket.send(sPkt);
}
} catch (SocketTimeoutException e) {
disconnect();
System.out.println("Server didn't respond and timeout occured.");
return false;
}
}
} catch (Exception e) {
System.out.println(e.getMessage());
return false;
}

I know what was wrong. That strange behavior was result of this line when last packet was received:
byte[] fileDataBytes = pktFactory.getDataBytes(dPkt);
Returned array size was always equal to specified packet length, even if received data was smaller. In my case last packet was 0 bytes (+4 bytes for tftp), but even then extra 512 bytes was added to output stream.
To resolve this I overload mentioned method with extra parameter - actual size of received packet when received data size is higher than 4 bytes and lower than specified packet size (512 bytes). This change resulted with getting correct size of array for last packet, so received file has correct size at the end of the operation.

Netty HttpStaticFileServerHandler problem

I use netty offical example HttpStaticFileServerHandler as file server, but when I download file from server I met a problem, the mp4 file I download from server is not complete and can't display.
https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/http/file/HttpStaticFileServerHandler.java
And here is my client code:
FileOutputStream fos = null;
try {
URL website = new URL("http://localhost:8090/export/App/***.mp4");
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
fos = new FileOutputStream("/Users/mine/***.mp4");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
} catch (Exception e) {
System.out.println("error msg:\n" + e);
} finally {
try {
if (fos != null) {
fos.close();
}
} catch (IOException ioe) {
System.out.println("fos close fail:\n" + ioe);
}
}

Make sure you close your FileOutputStream using fos.close().
Failing to do so means that only part of the data will be written, and that other programs may experience problems when accessing the file.
Another thing you could check is viewing the filesize of the file, those should match at both sides, if the file is way too small, open it with an text editor, and view the contents to check for clues.

I solved this problem. I find the RandomAccessFile is not closed in correct time.
Here is the change:
RandomAccessFile raf;
try {
raf = new RandomAccessFile(file, "r");
} catch (FileNotFoundException ignore) {
sendError(ctx, NOT_FOUND);
return;
}
...
sendFileFuture.addListener(new ChannelProgressiveFutureListener() {
#Override
public void operationProgressed(ChannelProgressiveFuture future, long progress, long total) {
if (total < 0) { // total unknown
System.err.println(future.channel() + " Transfer progress: " + progress);
} else {
System.err.println(future.channel() + " Transfer progress: " + progress + " / " + total);
}
}
#Override
public void operationComplete(ChannelProgressiveFuture future) {
System.err.println(future.channel() + " Transfer complete.");
raf.close();// close raf when transfer completed
}
});

Java server can't send bytes below 0 to PHP website

I'm running a Java server that uses the Twitter API and collects search results about any given keyword. My goal is to send the results to my website in PHP. However some Tweets have text with bytes less than 0, these appear to be unicode characters or similar. I've had to replace all of those characters with a space for the packet to be sent at all. If a byte less than 0 is sent the PHP script just reads "null". I need to be able to send bytes of any value, even if they're below 0.
Java: Replace bytes below value 0 with a space
// Get the Tweet text
String text = content.getData(2);
// Get the bytes
byte [] bytes = text.getBytes();
// Replace any byte below 0 with a space
for(int a = 0; a < bytes.length; ++a) {
if(bytes[a] < 0) {
bytes[a] = " ".getBytes()[0];
}
}
// Put the bytes back into a String
text = new String(bytes);
Java: Server that listens to commands and replies with output
ServerSocket socket = null;
InputStreamReader inputStream = null;
BufferedReader input = null;
try {
socket = new ServerSocket(port);
Logger.log("Server running on port " + port);
while(running) {
connection = socket.accept();
inputStream = new InputStreamReader(connection.getInputStream(), StandardCharsets.UTF_8);
input = new BufferedReader(inputStream);
// Run the command we're given, in this case the command will request Twitter search results
String reply = runCommand(input.readLine());
// Reply(String) will reply with the results
reply(reply);
}
} catch(Exception e) {
e.printStackTrace();
} finally {
try {
connection.close();
response.close();
if(inputStream != null) {
inputStream.close();
}
if(input != null) {
input.close();
}
if(socket != null) {
socket.close();
}
} catch(IOException e) {
e.printStackTrace();
}
}
Java: Reply method to send the results of the command execution (in this case it'll send the Twitter search results)
private void reply(String reply) {
try {
response = new DataOutputStream(connection.getOutputStream());
response.writeUTF(reply);
response.flush();
} catch(IOException e) {
e.printStackTrace();
}
}
PHP: Send commands (Twitter search query) via sockets and get the reply (Tweet data)
$socket = socket_create(AF_INET, SOCK_STREAM, getprotobyname('tcp'));
try {
socket_connect($socket, $address, $port);
// Encode the message in UTF-8, is this correct do to?
$message = utf8_encode($message);
// Get the result of sending this message to my Java server
$status = socket_sendto($socket, $message, strlen($message), MSG_EOF, $address, $port);
// Decode and return the results, is this the correct way to do this?
if($status != false) {
if($next = utf8_decode(socket_read($socket, $port))) {
return substr($next, 2);
}
}
} catch(Exception $e) {
}
// Even when I console.log the return of this method and it comes out "null" it is never this. I've tried changing this to return "-1" and other values and it still always returned "null" as if it was returning the string "null" in the above if statements.
return null;
I believe it may be important to note that bottom comment in PHP. I've tried looking over Google for a while now about this and I'm not sure if I'm doing things wrong or if I'm searching for the wrong thing.
How would I send bytes that are less than 0 through this system?

Android internal Storage error

I am really new to android and writing a simple game for android.
In the game, like in most games you have score,
now, I want the score to be saved in Internal Storage, and for some reason, I manage to save the score, but not load it back.
here is the code:
final TextView best = (TextView) findViewById(R.id.best);
public int read = -1;
public StringBuffer buffer = new StringBuffer();
public String scoreTxt = buffer.substring(0, buffer.indexOf(" ") + 1);
public int score = 0;
// Save
try {
fileOutputStream = openFileOutput("record.txt", Context.MODE_PRIVATE);
fileOutputStream.write(scoreString.getBytes());
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}finally {
if (fileOutputStream != null) {
try {
fileOutputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Toast.makeText(MainActivity.this, "Save() works fine", Toast.LENGTH_SHORT).show();
// load
try {
FileInputStream fileInputStream = openFileInput("record.txt");
while ((read = fileInputStream.read())!= -1){
buffer.append((char)read);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
best.setText("Best: " + scoreTxt);
Toast.makeText(MainActivity.this, "load() is good too " + scoreTxt, Toast.LENGTH_SHORT).show();
when I run the app, there is no crash or anything special in the logcat, but when ever I use scoreTxt the output is nothing, just " ".
can somebody help me solve this problem? Thanks

You never assign scoreTxt a value in your code.

You need to parse buffer after it has being populated.Now when scoreTxt is initialized buffer is null
You need to replace this
best.setText("Best: " + scoreTxt);
with
scoreTxt = buffer.substring(0, buffer.indexOf(" ") + 1);
best.setText("Best: " + scoreTxt);

In addition I wouldn't store game score into a file because score changes frequently and you want to avoid disk access a lot. Store it in SharedPreferences and from time to time flush it to a file on a background thread.

Cannot read file just written to

I am trying to read from a file to which some user credentials were written. I want to write the file to an internal storage location. The code:
private void writeSendDetails(String name, String number, String emailID) {
//This function writes details to userCredentials.txt and also sends it to server.
String text = "Name: " + userName + "\n" + "Number: " + userNumber + "\n" + "Email ID:" + userEmailID;
FileOutputStream fos = null;
try {
fos = openFileOutput(userCredFile, MODE_PRIVATE);
Log.v(this.toString(), fos.toString());
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
if(fos != null) {
OutputStreamWriter osw = new OutputStreamWriter(fos);
try {
osw.write(text);
} catch (IOException e) {
// TODO Auto-generated catch block
Log.v(this.toString(), "IOException caught in osw.write");
e.printStackTrace();
}
try {
osw.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
osw.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Log.v(this.toString(), "Written everything to userCredentials.txt");
readUserCredentials();
//code to send to server should begin here.
}
private void readUserCredentials() {
//function to read name, number and email ID from userCredentials.txt
/* Steps:
* 1. Check if file exists.
* 2. If it does, read all relevant credentials.
*/
File f = new File(userCredFile);
if(f.canRead()) {
Log.v(this.toString(), "Can open userCredentials for reading from.");
}
try {
FileReader fis = new FileReader(f);
Log.v(this.toString(), "Wrapping a buffered reader around file reader.");
BufferedReader bufRead = new BufferedReader(fis, 100);
String line;
try {
while((line = bufRead.readLine()) != null) {
Log.v(this.toString(), "Line read = " + line);
line = bufRead.readLine();
if(line.indexOf("Name: ") != -1) {
Log.v(this.toString(), "Found name in the string.");
userName = line.substring(6);
} else if(line.indexOf("Number: ") != -1) {
Log.v(this.toString(), "Found number in the string.");
userNumber = line.substring(8);
} else if(line.indexOf("Email ID: ") != -1) {
Log.v(this.toString(), "Found email in the string.");
userEmailID = line.substring(10);
}
}
Log.v(this.toString(), "User credentials = " + userName + " " + userNumber + " " + userEmailID);
} catch (IOException e) {
Log.v(this.toString(), "IOException caught.");
}
} catch (FileNotFoundException e1) {
Log.v(this.toString(), "File not found for reading.");
}
}
The LogCat output shows:
04-14 15:20:43.789: V/com.sriram.htmldisplay.htmlDisplay#44c0c660(675): Written everything to userCredentials.txt
04-14 15:20:43.789: V/com.sriram.htmldisplay.htmlDisplay#44c0c660(675): File not found for reading.
04-14 15:20:43.889: V/com.sriram.htmldisplay.fireflymenu#44c401e0(675): File not found for reading.
My question(s):
1. I need to write the file to internal storage. Am I doing it correctly?
2. Why is the file just written not being read?

Some things for your code:
#Oren is correct, you should use Log.e(TAG, "message", e) instead of the auto-created stuff from eclipse!
you should simply merge the 3 try/catch to one. No need to make it 3 times...
you should use Log.e() as said above for your FileNotFoundException too, so see the stacktrace to check the real reason (which currently covers the hint to solve your issue)
If you would have done at least number 3, you would have seen that the file you try to read could not be found. Thats why your log doesn't show the Can open userCredentials for reading from. output from your if statement.
The reason for that is pretty simple: You create the file by using openFileOutput(userCredFile, MODE_PRIVATE);. If you read the documentation of openFileOutput you will stumble upon:
The name of the file to open; can not contain path separators.
That means that userCredFile can only be something like test.txt. Also this method creates a file in a directory that can't be easily access from "outside".
When you now try to read the file via FileReader(userCredFile) it should be obvious, that android will try to open it in the root directory: /text.txt and it will, of course, fail. No non-root app can write/read in the root directory.
The main question, and also the answer to your issue: Why don't you use the corresponding openFileInput(userCredFile) method to read the file?