We Keep Coding

Missing one byte when transferring an image over Socket Java

I have a problem transferring a file over socket.
I Wrote a simple client / server app and the client takes a screenshot and send it to server.
The problem is the file is not completed whatever i do, It's always missing the first byte from the array which makes the photo damaged.
When I open the photo in any hex editor and compare the original photo with the one that the client sent, I can see the missing byte, as if I add it, the photo opens without the problem. The size of the sent file missing just one byte !
Here is a photo for the problem :
Original photo
sent photo
Here is the code :
Server ( Receiver ) :
byte[] buf;
InputStream inp;
try (BufferedOutputStream out1 = new BufferedOutputStream(new FileOutputStream(new File("final.jpeg")))) {
buf = new byte[s.getReceiveBufferSize()];
inp = new DataInputStream(s.getInputStream());
Thread.sleep(200);
int len = 0;
while ((len = inp.read(buf)) >0){
out1.write(buf,0,len);
}
out1.flush();
inp.close();
out1.close();
}
Client ( Sender ):
BufferedImage screenshot = new Robot().createScreenCapture(new Rectangle(Toolkit.getDefaultToolkit().getScreenSize()));
ByteArrayOutputStream os = new ByteArrayOutputStream();
ImageIO.write(screenshot, "jpeg", os);
ImageIO.write(screenshot, "jpeg", new File("test.jpeg"));
OutputStream out = new BufferedOutputStream( connection.getOutputStream());
out.write(os.toByteArray());
out.close();
I have tried to send the array with the same way I receive it but no lock. I have tried with, and without buffered, I have tried flush in both sides, I tried to turn off Nod antivirus, Tried a sleep when sending length,
I almost tried everything without success .
I have tried on both, My pc and a virtual machine windows 7 !
Any help will be appreciated.
Edit :
First 10 bytes from the original file :
first 10 bytes from the sent file :

The code you posted does not lose data. Somewhere prior to executing the server code you posted, you have executed a single InputStream.read() of one byte, possibly in a misguided attempt to test for end of stream.
The sleep is just literally a waste of time. Remove it. You don't need the DataInput/OutputStreams either.

Please keep in mind that DataInputStream signals end of stream by returning value -1 from read() therefore your server reading loop should look like this:
while ((len = inp.read(buf)) != -1){
out1.write(buf,0,len);
}
Perhaps this helps.

The client code looks fine. Must be the server. You only posted the part when "some" input stream is written to a file. What happens before? Anyone doing a read() on the input stream?
Sorry for writing this in the "answer" section. Apparently, I cannot comment yet.

Ok it was my fault ! I was looking for something wrong in server side but the fault was in client side ! I opened a DataInputStream to read the order coming from server without closing it and that was the problem.

Using TCP for File Transfer Between Java (Android client) and C (PC server - Linux)

I know similar questions exist but I haven't found any of them suitable for my problem. I have an Android device (API 15 - Android version 4.0.4) and a Linux machine running Arch Linux. The idea was to create a connection between the server (C program) and client (Android app) so I could exchange files. Also, the server supports parallel connections and requires authentication. The android app has to create 3 connections to the server (using 3 different ports, so that means 3 AsyncTask-s running multi-threaded ) .. Two of them are for parallel background processes, and 1 is for the actual file transfer. I have created a code which worked well on the Emulator (Android KitKat OS), but when testing on my own phone it doesn't work. I will post my code, and I would like some advice from you, if that is possible. Thanks.
This is the code running on Android devices... BUFFSIZE
is 1024 and it is a global variable.
I've tried setting it to many values and none of them worked for me. filesize is set earlier in the code and always has the correct value so don't worry about it :)
InputStream is = socket.getInputStream();
FileOutputStream fs = new FileOutputStream(target);
int u;
byte[] jj = new byte[BUFFSIZE];
long overall = 0, percent = 0;
try {
while (overall < filesize && mRun) {
u = is.read(jj, 0, BUFFSIZE);
if (u == -1) break;
fs.write(jj, 0, u);
overall += u;
percent = overall*100/filesize;
}
fs.flush();
fs.close();
is.close();
socket.close();
} catch (IOException ex) {
// There were no exceptions while testing
// There is some code here that deals with the UI
// which is not important
}
And this is the C code...
for (;;)
{
/* First read file in chunks of BUF_SIZE bytes */
unsigned char buff[BUFFER]={0};
int nread = fread(buffer,1,BUFFER, input);
printf("Bytes read %d \n", nread);
/* If read was success, send data. */
if(nread > 0)
{
printf("Sending \n");
write(sockfd, buffer, nread);
}
/*
* There is something tricky going on with read ..
* Either there was error, or we reached end of file.
*/
if (nread < BUFFER)
{
if (feof(input))
printf("End of file\n");
if (ferror(input))
printf("Error reading\n");
break;
}
}
I have tested this code many times, even using telnet and it worked quite well. But I am not sure about the Java code.
So, why doesn't it work? Well, what I know so far is that some files are damaged. Let's just say that if I transfer an mp3 file with the size of 4MB, 3.99 would be sent and the resting 0.01 would be lost somewhere in the middle of the file, for no reason! When you play the damaged mp3, you can realize that some parts (like, every 10 seconds) you go "off-beat".. Like if there was a small noise that is then skipped. The resulting file is shorter for around 10 000 bytes than the original (but that depends on the actual file size.. You always lose some small percentage of the file, and that means that the while loop never finishes - the download process never finishes because the sockets are blocking and the client ends up waiting for more bytes which are never received) .. What I believe that happens is that, from a 1024-byte-long buffer, around 1000 bytes are used sometimes, instead of the full 1024 buffer size, which leads to the loss of 24 bytes. I am not saying that these are the actual numbers, but, that is just something going on in my head; I am likely wrong about this. I couldn't share the whole code with you because it's really long, so I decided to use the functions that deal with the download process, instead.

It is totally fine for the read method not to fill the whole buffer. That method returns the number of bytes read. You even assign that value to a variable:
u = is.read(jj, 0, BUFFSIZE);
But then you only check if u is -1 in order to find out when to stop reading. From the documentation:
The number of bytes actually read is returned as an integer.
Which means that your byte array has a maximum length of 1024 bytes, but not all of those will be filled at each read. And of course it won't be, otherwise this would only work if your input stream contains an exakt multiple of 1024 bytes.
Also: a thing called debugging exists. It might be hard to inspect binary data such as mp3 files, so try debugging while transmitting a text file.

Java transfer files via UDP. Compressed files are damaged?

i'm trying to transfer Files with a DatagrammSocket in Java. I'm reading the files into 4096 Byte pieces. We are using ACK, so all pieces are in the right order, we tried pdf, exe, jpg and lot more stuff successfully, but iso, zip and 7z are not working. They have exactly the same size afterwards. Do you have any idea?
Reading the Parts:
byte[] b = new byte[FileTransferClient.PACKAGE_SIZE - 32];
FileInputStream read = new FileInputStream(file);
read.skip((part - 1) * (FileTransferClient.PACKAGE_SIZE - 32));
read.read(b);
content = b;
Writing the Parts:
stream = new FileOutputStream(new File(this.filePath));
stream.write(output);
...
stream.write(output);
stream.close();
(Sorry for great grammar, i'm German)

Your write() method calls are assuming that the entire buffer was filled by receive(). You must use the length provided with the DatagramPacket:
datagramSocket.receive(packet);
stream.write(packet.getData(), packet.getOffset(), packet.getLength());
If there is overhead in the packet, e.g. a sequence number, which there should be, you will need to adjust the offset and length accordingly.
NB TCP will ensure 'everything gets transferred and is not damaged'.

Why does my untar not contain the last bytes

I've written a rest resource that serves a .tar.gz file. It's working OK. I've tried requesting it, saving the data, unpacking it (with tar xzvf [filename]) and I get the correct data.
However, I'm trying to use java.util.zip.GZIPInputStream and org.apache.tools.tar.TarInputStream to unzip and untar a .tar.gz that I'm serving in a JUnit test, to verify that it's working automatically. This is the code in my unit test with some details removed:
HttpResponse response = <make request code here>
byte[] receivedBytes = FileHelper.copyInputStreamToByteArray(response.getEntity().getContent(), true);
GZIPInputStream gzipInputStream = new GZIPInputStream(new ByteArrayInputStream(receivedBytes));
TarInputStream tarInputStream = new TarInputStream(gzipInputStream);
TarEntry tarEntry = tarInputStream.getNextEntry();
ByteArrayOutputStream byteArrayOutputStream = null;
System.out.println("Record size: " + tarInputStream.getRecordSize());
while (tarEntry != null) // It only goes in here once
{
byteArrayOutputStream = new ByteArrayOutputStream();
tarInputStream.copyEntryContents(byteArrayOutputStream);
tarEntry = tarInputStream.getNextEntry();
}
byteArrayOutputStream.flush();
byteArrayOutputStream.close();
byte[] archivedBytes = byteArrayOutputStream.toByteArray();
byte[] actualBytes = <get actual bytes>
Assert.assertArrayEquals(actualBytes, archivedBytes);
The final assert fails with a difference at byte X = (n * 512) + 1, where n is the greatest natural number such that n * 512 <= l and l is the length of the data. That is, I get the the biggest possible multiple of 512 bytes of data correctly, but debugging the test I can see that all the remaining bytes are zero. So, if the total amount of data is 1000 bytes, the first 512 bytes in archivedBytes are correct, but the last 488 are all zero / unset, and if the total data is 262272 bytes I get the first 262144 (512*512) bytes correctly, but the remaining bytes are all zero again.
Also, the tarInputStream.getRecordSize() System out above prints Record size: 512, so I presume that this is somehow related. However, since the archive works if I download it, I guess the data must be there, and there's just something I'm missing.
Stepping into the tarInputStream.copyEntryContents(byteArrayOutputStream) with the 1000 byte data, in
int numRead = read(buf, 0, buf.length);
the numRead is 100, but looking at the buffer, only the first 512 bytes are non-zero. Maybe I shouldn't be using that method to get the data out of the TarInputStream?
If anyone knows how it's supposed to work, I'd be very grateful for any advice or help.

You can specify the output block size to be used when you create a tar archive. Thus the size of the archive will be a multiple of the block size. As the archive size doesn't normally fit in a whole number of blocks, zeros are added to the last block of data to make it of the right size.

It turned out that I was wrong in my original question, and the error was in the resource code. I wasn't closing the entry on the TarOutputStream when writing to it. I guess this was not causing any problems when requesting it manually from the server, maybe because the entry was closed with the connection or something, but working differently when being requested from a Unit test... though I must admit that doesn't make a whole lot of sense to be :P
Looking at the fragment of my writing code below, I was missing line 3.
1: tarOutputStream.putNextEntry(tarEntry);
2: tarOutputStream.write(fileRawBytes);
3: tarOutputStream.closeEntry();
4: tarOutputStream.close();
I didn't even know there was such a thing as a "closeEntry" on the TarOutputStream... I do now! :P

Can I write multiple byte arrays to an HttpClient without client-side buffering?

The Problem
I would like to upload very large files (up to 5 or 6 GB) to a web server using the HttpClient class (4.1.2) from Apache. Before sending these files, I break them into smaller chunks (100 MB, for example). Unfortunately, all of the examples I see for doing a multi-part POST using HttpClient appear to buffer the file contents before sending them (typically, a small file size is assumed). Here is such an example:
HttpClient httpclient = new DefaultHttpClient();
HttpPost post = new HttpPost("http://www.example.com/upload.php");
MultipartEntity mpe = new MultipartEntity();
// Here are some plain-text fields as a part of our multi-part upload
mpe.addPart("chunkIndex", new StringBody(Integer.toString(chunkIndex)));
mpe.addPart("fileName", new StringBody(somefile.getName()));
// Now for a file to include; looks like we're including the whole thing!
FileBody bin = new FileBody(new File("/path/to/myfile.bin"));
mpe.addPart("myFile", bin);
post.setEntity(mpe);
HttpResponse response = httpclient.execute(post);
In this example, it looks like we create a new FileBody object and add it to the MultipartEntity. In my case, where the file could be 100 MB in size, I'd rather not buffer all of that data at once. I'd like to be able to write out that data in smaller chunks (4 MB at a time, for example), eventually writing all 100 MB. I'm able to do this using the HTTPURLConnection class from Java (by writing directly to the output stream), but that class has its own set of problems, which is why I'm trying to use the Apache offerings.
My Question
Is it possible to write 100 MB of data to an HttpClient, but in smaller, iterative chunks? I don't want the client to have to buffer up to 100 MB of data before actually doing the POST. None of the examples I see seem to allow you to write directly to the output stream; they all appear to pre-package things before the execute() call.
Any tips would be appreciated!
--- Update ---
For clarification, here's what I did previously with the HTTPURLConnection class. I'm trying to figure out how to do something similar in HttpClient:
// Get the connection's output stream
out = new DataOutputStream(conn.getOutputStream());
// Write some plain-text multi-part data
out.writeBytes(fieldBuffer.toString());
// Figure out how many loops we'll need to write the 100 MB chunk
int bufferLoops = (dataLength + (bufferSize - 1)) / bufferSize;
// Open the local file (~5 GB in size) to read the data chunk (100 MB)
raf = new RandomAccessFile(file, "r");
raf.seek(startingOffset); // Position the pointer to the beginning of the chunk
// Keep track of how many bytes we have left to read for this chunk
int bytesLeftToRead = dataLength;
// Write the file data block to the output stream
for(int i=0; i<bufferLoops; i++)
{
// Create an appropriately sized mini-buffer (max 4 MB) for the pieces
// of this chunk we have yet to read
byte[] buffer = (bytesLeftToRead < bufferSize) ?
new byte[bytesLeftToRead] : new byte[bufferSize];
int bytes_read = raf.read(buffer); // Read ~4 MB from the local file
out.write(buffer, 0, bytes_read); // Write that bit to the stream
bytesLeftToRead -= bytes_read;
}
// Write the final boundary
out.writeBytes(finalBoundary);
out.flush();

If I'm understanding your question correctly, your concern is loading the whole file into memory (right?). If That is the case, you should employ Streams (such as a FileInputStream). That way, the whole file doesn't get pulled into memory at once.
If that doesn't help, and you still want to divide the file up into chunks, you could code the server to deal with multiple POSTS, concatenating the data as it gets them, and then manually split up the bytes of the file.
Personally, I prefer my first answer, but either way (or neither way if these don't help), Good luck!

Streams are definitely the way to go, I remember doing something similar a while back with some bigger files and it worked perfectly.

All you need is to wrap your custom content generation logic into HttpEntity implementation. This will give you a complete control over the process of content generation and content streaming.
And for the record: MultipartEntity shipped with HttpClient does not buffer file parts in memory prior to writing them out to the connection socket.