We Keep Coding

How do I prevent a file from corrupting when I transfer it over a local network using sockets?

I am working on a school project where I want to make a personal storage server. At the moment, what I am trying to achieve is being able to transfer a file from the client machine to the server. However, when testing this with an image, the file partially sends before it corrupts.
Please bare in mind that I am a reasonably new programmer and that my technical knowledge may be some-what limited.
I am using a byte array through a DataOutputStream to transfer the file. I want to use this method as it should work for any file type. I've tried to set the buffer size to the exact size of the file and larger but neither have worked.
Server:
public void run() {
try {
System.out.println("ip: " + clientSocket.getInetAddress().getHostAddress());
out = new DataOutputStream(clientSocket.getOutputStream());
in = new DataInputStream(clientSocket.getInputStream());
in.read(buffer, 0, buffer.length);
fileOut = new FileOutputStream("X:\\My Documents\\My
Pictures\\gradient.jpg");
fileOut.write(buffer, 0, buffer.length);
in.close();
out.close();
clientSocket.close();
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
Client:
public void startConnection(String ip, int port) {
try {
clientSocket = new Socket(ip, port);
out = new DataOutputStream(clientSocket.getOutputStream());
in = new DataInputStream(clientSocket.getInputStream());
x = false;
Path filePath = Paths.get("C:\\Users\\georg\\Documents\\gradient.jpg");
buffer = Files.readAllBytes(filePath);
Thread.sleep(3000);
//Files.write(filePath, buffer);
//out.write(buffer,0,buffer.length);
x = true;
sendMessage(buffer);
} catch (IOException ex) {
System.out.println(ex.getMessage());
} catch (InterruptedException ex) {
Logger.getLogger(PCS_Client.class.getName()).log(Level.SEVERE, null, ex);
}
}
public byte[] sendMessage(byte[] buffer) {
if (x==true){
try {
out.write(buffer,0,buffer.length);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
return null;
}
Here is a comparison of the files I've tried to send vs the files I receive:
https://imgur.com/gallery/T7nUUJT
Curiously, sending a single colour image produces a single colour image on the server. I believe the issue here may have to be in the timing of code execution however I am not sure and do not know how to go about fixing it.

The issue is in your server code, at this line:
in.read(buffer, 0, buffer.length);
You expect to read all the data at once, but if you read the doc you will find this:
public final int read(byte[] b,
int off,
int len)
throws IOException
Reads up to len bytes of data from the contained input stream into an
array of bytes. An attempt is made to read as many as len bytes, but a
smaller number may be read, possibly zero. The number of bytes
actually read is returned as an integer.
The important part is Reads up to len bytes of data.
You must use the return value of read and call it read repeatedly until the is nothing more to read.

How to add string to a file without closing the output stream objects every time

I will receive a large chunk of data (say 1000 data/second, each data has minimum size of 15 bytes). My earlier approach was to create a new outputstream object every time, specify the path and add the values to the file, all this is done in a separate thread. How ever I am still facing a performance hit. I taught instead of writing the data to a file as
File dir = new
File(android.os.Environment.getExternalStorageDirectory().getAbsolutePath() + DEBUG_FILE_PATH);
boolean b = dir.mkdirs();
try
{
fileOutputStream = new FileOutputStream(new File(dir, FILE_NAME),
true);
outputStreamWriter = new OutputStreamWriter(fileOutputStream);
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
outputstreamwriter.append("some data").close();
I want to maintain the outputstreamwriter and other objects and use them to add the data to the outputstreamwriter buffer, and at the end of my application(when i close the app, may at onDestroy() method of activity). I need to write the data to the file and then close all the open stream.
This approach works for me, but the buffer size for outputstreamwriter is 8kb only. Which is less compared to the amount of data that I am receiving.
How can i solve this ?

The vast majority of your performance hit is most probably in opening the file every single time you want to write a few bytes to it.
So, if you just eliminate the opening and closing of the file all the time, you should be fine.
Just open the file once, keep writing data to it as the data arrives, and then close the file when your application closes.
This way, using a buffered OutputStreamWriter will give you a performance benefit without having to worry about the size of its buffer: when its buffer is full, it will flush itself, transparently to you. You don't need to know anything about how it works and how large (or small) its buffer is.

This solved my problem.
By this approach I have opened the file once when the app starts and, I am adding the 'n' values which I receive from the service into the file. I am flushing the buffer(this writes the data to file). Now even if I receive large data not more than 8kb(buffer max size) I can write it to file which is already opened. Finally I am closing the streams when the app is closed.
//Util class
public static File dir;
public static FileOutputStream fileOutputStream;
public static OutputStreamWriter outputStreamWriter;
//Different class, you can initialize it on your Application class or home activity
private void initializeFileWriteObjects()
{
dir = new File(android.os.Environment.getExternalStorageDirectory().getAbsolutePath() + DEBUG_FILE_PATH);
boolean b = dir.mkdirs();
try
{
fileOutputStream = new FileOutputStream(new File(dir, FILE_NAME), true);
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
outputStreamWriter = new OutputStreamWriter(fileOutputStream);
}
//Util File
private static boolean writeToFile(final byte[] stream)
{
//Convert String to hex, as I want this to be in hex
final String stringData = byteArrayToHexString(stream);
try
{
outputStreamWriter.append(stringData);
outputStreamWriter.flush();
}
catch (IOException e)
{
e.printStackTrace();
return false;
}
return true;
}
//When the app is closed.
#Override
protected void onDestroy()
{
super.onDestroy();
closeFileStream();
}
//This method is in same Util class, but called at onDestroy()
public static void closeFileStream()
{
try
{
outputStreamWriter.close();
fileOutputStream.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}

How to only get the first 8 bytes from a file from a DataInputStream java

Currently writing a java swing program. which gets all files from a drive and checks the binary signature. But i only want the first 2-8 bytes of the file to speed to program up. Tried most solutions already available but none work with what i have already coded.
Current Code:
public void getBinary() {
try {
// get the file as a DataInputStream
DataInputStream input = new DataInputStream(new FileInputStream(file));
try {
// if files are avaliable countinue looping and get bytes / hex
while (input.available() > 0) {
// build a hex string from the bytes and change to uppercase
sb.append(Integer.toHexString(input.readByte()).toUpperCase());
// need to get the first couple of (8) bytes
}
} catch (IOException e) {
}
// print the hex out to command
// System.out.println(sb.toString());
} catch (FileNotFoundException e2) {
}
}

works perfect for me, and may help others too!!!
https://i.stack.imgur.com/8SU5A.png

Java OutOfMemoryError while merge large file parts from chunked files

I have a problem when the user upload large files (> 1 GB) (I'm using flow.js library), it creates hundred of thousand small chunked files (e.g 100KB each) inside temporary directory but failed to merge into single file, due to MemoryOutOfException. This is not happened when the file is under 1 GB. I know it sound tedious and you probably suggest me to increase the XmX in my container-but I want to have another angle besides that.
Here is my code
private void mergeFile(String identifier, int totalFile, String outputFile) throws AppException{
File[] fileDatas = new File[totalFile]; //we know the size of file here and create specific amount of the array
byte fileContents[] = null;
int totalFileSize = 0;
int filePartUploadSize = 0;
int tempFileSize = 0;
//I'm creating array of file and append the length
for (int i = 0; i < totalFile; i++) {
fileDatas[i] = new File(identifier + "." + (i + 1)); //indentifier is the name of the file
totalFileSize += fileDatas[i].length();
}
try {
fileContents = new byte[totalFileSize];
InputStream inStream;
for (int j = 0; j < totalFile; j++) {
inStream = new BufferedInputStream(new FileInputStream(fileDatas[j]));
filePartUploadSize = (int) fileDatas[j].length();
inStream.read(fileContents, tempFileSize, filePartUploadSize);
tempFileSize += filePartUploadSize;
inStream.close();
}
} catch (FileNotFoundException ex) {
throw new AppException(AppExceptionCode.FILE_NOT_FOUND);
} catch (IOException ex) {
throw new AppException(AppExceptionCode.ERROR_ON_MERGE_FILE);
} finally {
write(fileContents, outputFile);
for (int l = 0; l < totalFile; l++) {
fileDatas[l].delete();
}
}
}
Please show the "inefficient" of this method, once again... only large files that cannot be merge using this method, smaller one ( < 1 GB) no problem at all....
I appreciate if you do not suggest me to increase the heap memory instead show me the fundamental error of this method... thanks...
Thanks

It's unnecessary to allocate the entire file size in memory by declaring a byte array of the entire size. Building the concatenated file in memory in general is totally unnecessary.
Just open up an outputstream for your target file, and then for each file that you are combining to make it, just read each one as an input stream and write the bytes to outputstream, closing each one as you finish. Then when you're done with them all, close the output file. Total memory use will be a few thousand bytes for the buffer.
Also, don't do I/O operations in finally block (except closing and stuff).
Here is a rough example you can play with.
ArrayList<File> files = new ArrayList<>();// put your files here
File output = new File("yourfilename");
BufferedOutputStream boss = null;
try
{
boss = new BufferedOutputStream(new FileOutputStream(output));
for (File file : files)
{
BufferedInputStream bis = null;
try
{
bis = new BufferedInputStream(new FileInputStream(file));
boolean done = false;
while (!done)
{
int data = bis.read();
boss.write(data);
done = data < 0;
}
}
catch (Exception e)
{
//do error handling stuff, log it maybe?
}
finally
{
try
{
bis.close();//do this in a try catch just in case
}
catch (Exception e)
{
//handle this
}
}
}
} catch (Exception e)
{
//handle this
}
finally
{
try
{
boss.close();
}
catch (Exception e) {
//handle this
}
}

... show me the fundamental error of this method
The implementation flaw is that you are creating a byte array (fileContents) whose size is the total file size. If the total file size is too big, that will cause an OOME. Inevitably.
Solution - don't do that! Instead "stream" the file by reading from the "chunk" files and writing to the final file using a modest sized buffer.
There are other problems with your code too. For instance, it could leak file descriptors because you are not ensure that inStream is closed under all circumstances. Read up on the "try-with-resources" construct.

Incomplete file using RandomAccessFile in java

I made a small program to download data and write it to a file.
Here is the code:
public void run()
{
byte[] bytes = new byte[1024];
int bytes_read;
URLConnection urlc = null;
RandomAccessFile raf = null;
InputStream i = null;
try
{
raf = new RandomAccessFile("file1", "rw");
}
catch(Exception e)
{
e.printStackTrace();
return;
}
try
{
urlc = new URL(link).openConnection();
i = urlc.getInputStream();
}
catch(Exception e)
{
e.printStackTrace();
return;
}
while(canDownload())
{
try
{
bytes_read = i.read(bytes);
}
catch(Exception e)
{
e.printStackTrace();
return;
}
if(bytes_read != -1)
{
try
{
raf.write(bytes, 0, bytes_read);
}
catch(Exception e)
{
e.printStackTrace();
return;
}
}
else
{
try
{
i.close();
raf.close();
return;
}
catch(Exception e)
{
e.printStackTrace();
return;
}
}
}
}
The problem is that when I download big files, I get few bytes missing in the end of the file.
I tried to change the byte array size to 2K, and the problem was solved. But when I downloaded a bigger file (500 MB) , I got few bytes missing again.
I said "Ok, let's try with 4K size". And I changed the byte array size to 4K. It worked!
Nice, but then I downloaded a 4 GB file, I got bytes missing in the end again!
I said "Cool, let's try with 8K size". And then I changed the byte array size to 8K. Worked.
My first question is: Why this happens? (when I change buffer size, the file doesn't get corrupted).
Ok, in theory, the file corrupted problem can be solved changing the byte array size to bigger values.
But there's another problem: how can I measure the download speed (in one second interval) with big byte array sizes?
For example: Let's say that my download speed is 2 KB/s. And the byte array size is 4 K.
My second question is: How can I measure the speed (in one second interval) if the thread will have to wait the byte array to be full? My answer should be: change the byte array size to a smaller value. But the file will get corrupted xD.
After trying to solve the problem by myself, I spent 2 days searching over the internet for a solution. And nothing.
Please, can you guys answer my two questions? Thanks =D
Edit
Code for canDownload():
synchronized private boolean canDownload()
{
return can_download;
}

My advice is to use a proven library such as Apache Commons IO instead of trying to roll your own code. For your particular problem, take a look at the copyURLToFile(URL, File) method.

I would:
Change the RandomAccessFile to a FileOutputStream.
Get rid of canDownload(), whatever it's for, and set a read timeout on the connection instead.
Simplify the copy loop to this:
while ((bytes_read = i.read(bytes)) > 0)
{
out.write(bytes, 0, bytes_read);
}
out.close();
i.close();
with all the exception handling outside this loop.

I think you will find the problem is that you closed the underlying InputStream while the RandomAccessFile still had data in its write buffers. This will be why you are occasionally missing the last few bytes of data.
The race condition is between the JVM flushing the final write, and your call to i.close().
Removing the i.close() should fix the problem; it isn't necessary as the raf.close() closes the underlying stream anyway, but this way you give the RAF a chance to flush any outstanding buffers before it does so.