We Keep Coding

How do I prevent a file from corrupting when I transfer it over a local network using sockets?

I am working on a school project where I want to make a personal storage server. At the moment, what I am trying to achieve is being able to transfer a file from the client machine to the server. However, when testing this with an image, the file partially sends before it corrupts.
Please bare in mind that I am a reasonably new programmer and that my technical knowledge may be some-what limited.
I am using a byte array through a DataOutputStream to transfer the file. I want to use this method as it should work for any file type. I've tried to set the buffer size to the exact size of the file and larger but neither have worked.
Server:
public void run() {
try {
System.out.println("ip: " + clientSocket.getInetAddress().getHostAddress());
out = new DataOutputStream(clientSocket.getOutputStream());
in = new DataInputStream(clientSocket.getInputStream());
in.read(buffer, 0, buffer.length);
fileOut = new FileOutputStream("X:\\My Documents\\My
Pictures\\gradient.jpg");
fileOut.write(buffer, 0, buffer.length);
in.close();
out.close();
clientSocket.close();
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
Client:
public void startConnection(String ip, int port) {
try {
clientSocket = new Socket(ip, port);
out = new DataOutputStream(clientSocket.getOutputStream());
in = new DataInputStream(clientSocket.getInputStream());
x = false;
Path filePath = Paths.get("C:\\Users\\georg\\Documents\\gradient.jpg");
buffer = Files.readAllBytes(filePath);
Thread.sleep(3000);
//Files.write(filePath, buffer);
//out.write(buffer,0,buffer.length);
x = true;
sendMessage(buffer);
} catch (IOException ex) {
System.out.println(ex.getMessage());
} catch (InterruptedException ex) {
Logger.getLogger(PCS_Client.class.getName()).log(Level.SEVERE, null, ex);
}
}
public byte[] sendMessage(byte[] buffer) {
if (x==true){
try {
out.write(buffer,0,buffer.length);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
return null;
}
Here is a comparison of the files I've tried to send vs the files I receive:
https://imgur.com/gallery/T7nUUJT
Curiously, sending a single colour image produces a single colour image on the server. I believe the issue here may have to be in the timing of code execution however I am not sure and do not know how to go about fixing it.

The issue is in your server code, at this line:
in.read(buffer, 0, buffer.length);
You expect to read all the data at once, but if you read the doc you will find this:
public final int read(byte[] b,
int off,
int len)
throws IOException
Reads up to len bytes of data from the contained input stream into an
array of bytes. An attempt is made to read as many as len bytes, but a
smaller number may be read, possibly zero. The number of bytes
actually read is returned as an integer.
The important part is Reads up to len bytes of data.
You must use the return value of read and call it read repeatedly until the is nothing more to read.

Thread blocking sometimes while unzipping png files

I am developing an android app that requires downloading a zip file (around 1,5 MB max) with a small amount of logos (png files of 20-30KB average size) from a webserver.
I have encapsulated the process of downloading and unzipping the files into android internal storage in an AsyncTask's doInbackground() method.
The issue I have is that the unZipIntoInternalStorage() method I have developed (pasted down), sometimes runs forever. Usually it takes around 900ms seconds to unzip and save the logos into internal storage, but for some unknown reason around 1 of 4 executions blocks during the loop (and stays there "for ever" taking more than 2 or 3 mins to decompress all png files):
while ((count = zipInputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, count);
}
Edited: After doing some logging and debugging I found out that the line slowing down so much the execution is : zipInputStream.read(buffer) inside the while condition. Any ideas why sometimes it runs extremely fast and some others extremely slow?
Here is my complete method to unzip the downloaded files and save them into android internal storage. I also add the method where the zipInputStream is initialized, from the zip file downloaded (both methods executed inside doInBackground() ):
private void unZipIntoInternalStorage(ZipInputStream zipInputStream) {
long start = System.currentTimeMillis();
Log.i(LOG_TAG, "Unzipping started ");
try {
File iconsDir = context.getDir("icons", Context.MODE_PRIVATE);
ZipEntry zipEntry;
byte[] buffer = new byte[1024];
int count;
FileOutputStream outputStream;
while ((zipEntry = zipInputStream.getNextEntry()) != null) {
File icon = new File(iconsDir, zipEntry.getName());
outputStream = new FileOutputStream(icon);
while ((count = zipInputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, count);
}
zipInputStream.closeEntry();
outputStream.close();
}
zipInputStream.close();
} catch (Exception e) {
Log.e(LOG_TAG + " Decompress", "unzip error ", e);
e.printStackTrace();
}
Log.i(LOG_TAG, "Unzipping completed time required: " + (System.currentTimeMillis() - start) + " ms");
}
private ZipInputStream httpDownloadIconsZip(String zipUrl) {
URLConnection urlConnection;
try {
URL finalUrl = new URL(zipUrl);
urlConnection = finalUrl.openConnection();
return new ZipInputStream(urlConnection.getInputStream());
} catch (IOException e) {
Log.e(LOG_TAG, Log.getStackTraceString(e));
return null;
}
}
To clarify, after testing this method several times and debugging, the blocking for ever always happens in the nested while loop I described previously. But I can't find the reason (see edited clarification)
Also I have already tried this method using BufferedOutputStream class and with the same results: nested while loop running forever sometimes and others unzipping successfully in less than a second.
Hope I have been as clear as possible, since I have spent long hours looking for posible causes to the issue in several post regarding unzipping files or java I/O methods with no success.
Any help appreciated. Thanks

I would suspect the InputStream rather than the output to be the issue.
Try :
return new ZipInputStream(new BufferedInputStream(urlConnection.getInputStream()));
You can add an argument for setting buffer size, but default settings should be fine for your use case.
The problem is typically caused by small packet size, leading to one read forcing several IO operations.
Ideally, you do want to use also a BufferedOutputStream, since the read could read much less than 1kB, but you still do pay a full I/O for each write.
As a general rule, remember I/O is 100 times slower than anything else you could do, and often leads to the scheduler putting your task on Wait. So just use BufferedStream anywhere the stream is not in memory (i.e. always except for StringBufferXXXStream basically).
In your case, due to zip protocol, your read could lead to any number of smaller reads on the actual network socket, as Zip parses and interprets headers and contents of the compressed file.

I know this error. If your zip file has been damaged, when you try to unzip it by ZipInputStream, it will be a infinite loop, because the file has no EOF.
But if your unzip it by ZipFile, you can catch that Exception!
public static boolean unZipByFilePath(String fileName, String unZipDir) {
long startUnZipTime = System.currentTimeMillis();
try {
File f = new File(unZipDir);
if (!f.exists()) {
f.mkdirs();
}
BufferedOutputStream dest = null;
BufferedInputStream is = null;
ZipEntry entry;
ZipFile zipfile = new ZipFile(fileName);
Enumeration e = zipfile.entries();
while (e.hasMoreElements()) {
entry = (ZipEntry) e.nextElement();
is = new BufferedInputStream(zipfile.getInputStream(entry));
int count = 0;
byte data[] = new byte[BUFFER];
String destFilePath = unZipDir + "/" + entry.getName();
File desFile = new File(destFilePath);
if (entry.isDirectory()) {
desFile.mkdirs();
} else if (!desFile.exists()) {
desFile.getParentFile().mkdirs();
desFile.createNewFile();
}
FileOutputStream fos = new FileOutputStream(destFilePath);
dest = new BufferedOutputStream(fos, BUFFER);
while ((count = is.read(data, 0, BUFFER)) != -1) {
dest.write(data, 0, count);
}
dest.flush();
dest.close();
is.close();
}
zipfile.close();
} catch (Exception e) {
Log.e(TAG, "unZipByFilePath failed : " + e.getMessage());
return false;
}
return true;
}

Unable to Read Local Resource

I have a project (in Eclipse, but that doesn't matter) with a hierarchy as follows:
-src
---Start.java
---resources
-----media
-------intro.wav
-----textures
-------logo.png
-------tiles.abotm
In Start.java, I'm trying to get tiles.abotm as an InputStream using Class.getResourceAsStream(String) as such:
public class Start
{
public static void main(String[] args)
{
try
{
InputStream in = Start.class.getResourceAsStream(
"/resources/textures/tiles.abotm");
}catch(Exception e){
e.printStackTrace();
}
}
}
Simple enough, right? Unfortunately, no. The InputStream is completely empty with a size of 0. I've also tried opening a FileInputStream directly to the absolute location of tiles.abotm, but I get the same thing! I know the file is not empty. In fact, it has 2,257 bytes, according to Windows, Eclipse, and the File object used to create the FileInputStream mentioned previously. Also according to the File object, it is readable, writable, it exists, it is not a directory, and the name of it is tiles.abotm. So, if the File object can read it, why can't it be opened up in an InputStream??
--EDIT--
I forgot to mention that I have another file in the textures directory called logo.png which I am able to open and read in the exact same manner with no problem at all. It is only this file.
--In reply to fge, this is the actual code:
Loader.loadTextureMap("/resources/textures/tiles.abotm");//This is called in a separate method in a separate class.
public class Loader{
public static TextureMap loadTextureMap(String texMap){
DataInputStream dis = new DataInputStream(
Start.class.getResourceAsStream(texMap));
//It then goes on to read it, but I've determined that at this point,
there is nothing in this DataInputStream.
}
}

After a lot of discussion, the code that works for the OP:
final byte[] buf = new byte[1024]; // or other
final URL url = Start.class.getResource("whatever");
// check for url == null
InputStream in;
ByteArrayOutputStream out;
// I really wish this syntax was something else, it sucks
try (
in = url.openStream();
out = new ByteArrayOutputStream();
) {
int count;
while ((count = in.read(buf)) != -1)
out.write(buf, 0, count);
out.flush();
} catch (IOException e) {
// handle e here
}
final ByteBuffer buffer = ByteBuffer.wrap(out.toByteArray());
// use the buffer

Why do I get exception error while trying to reset Reader to 0 position?

I'm trying to read a webpage using following code :
URL url = new URL("somewebsitecomeshere");
URLConnection c = url.openConnection();
if(getHttpResponseCode(c) == 200)
{
if (isContentValid(c))//accept html/xml only!
{
InputStream is = c.getInputStream();
Reader r = new InputStreamReader(is);
System.out.println(r.toString());
//after commenting this everything works great!
setHTMLString(getStringFromReader(r));
System.out.println(getHTMLString());
ParserDelegator parser = new ParserDelegator();
parser.parse(r, new Parser(url), true);
r.close();
is.close();
try {
Thread.sleep(500);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else
log("content is not valid!");
}
else
{
System.out.println("ERROR" + c.getContentType() + c.getURL());
}
//---------------------------------------------------
private String getStringFromReader(Reader reader) throws IOException {
char[] arr = new char[8*1024]; // 8K at a time
StringBuffer buf = new StringBuffer();
int numChars;
while ((numChars = reader.read(arr, 0, arr.length)) > 0) {
buf.append(arr, 0, numChars);
}
//Reset position to 0
reader.reset();
return buf.toString();
}
if try to read string using getStringFromReader() the rest of the code will be ignored due to changing position of Reader to EOF so I tried to reset the position to 0 but I got the following error :
java.io.IOException: reset() not supported
at java.io.Reader.reset(Unknown Source)
at sample.getStringFromReader(Spider.java:248)
at default(sample.java:286)
at default.main(sample.java:130)
How can I reset the Reader position to 0?

Short answer, your stream doesn't support reset or mark methods. Check the result of:
is.markSupported()
Long answer, an InputStream is a flow of bytes. Bytes can come from a file, a network resource, a string, etc. So basically, there are streams that don't support resetting the reader position to the start of the stream, while others do (random access file).
A stream from a web site will normally use underlying network connection to provide the data. It means that it's up to the underlying network protocol (TCP/IP for example) to support or not resetting the stream, and normally they don't.
In order to reset any stream you would have to know the entire flow, from start to end. Network communications send a bunch of packages (which may be in order or not) to transfer data. Packages may get lost or even be duplicated, so normally information is buffered and interpreted as it is received. It would be very expensive to reconstruct all messages at network level. So that is normally up to the receiver, if it wants to do that.
In your case If what you want is print the input stream I would recommend creating a custom InputStream, which receives the original InputStream and whenever it is read it prints the read value and returns it at the same time. For example:
class MyInputStream extends InputStream {
InputStream original = null;
public MyInputStream(InputStream original) {
this.original = original;
}
#Override
public int read() throws IOException {
int c = original.read();
System.out.printf("%c", c);
return c;
}
}
Then wrap your original InputStream with that:
.
.
.
InputStream myIs = new MyInputStream(is);
Reader r = new InputStreamReader(myIs);
.
.
.
Hope it helps.

InputStreamReader does not support reset(). Also, you did not call mark(0) before.
What you could do is wrap your reader in a BufferedReader of a sufficient size so that reset is supported. If you cannot do that, then you should try to open a new connection to your URL.

java nio and FileInputStream

Basically I have this code to decompress some string that stores in a file:
public static String decompressRawText(File inFile) {
InputStream in = null;
InputStreamReader isr = null;
StringBuilder sb = new StringBuilder(STRING_SIZE);
try {
in = new FileInputStream(inFile);
in = new BufferedInputStream(in, BUFFER_SIZE);
in = new GZIPInputStream(in, BUFFER_SIZE);
isr = new InputStreamReader(in);
int length = 0;
while ((length = isr.read(cbuf)) != -1) {
sb.append(cbuf, 0, length);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
in.close();
} catch (Exception e1) {
e1.printStackTrace();
}
}
return sb.toString();
}
Since physical IO is quite time consuming, and since my compressed version of files are all quite small(around 2K from 2M of text), is it possible for me to still do the above, but on a file that is already mapped to memory? possibly using java NIO? Thanks

It won't make any difference, at least not much. Mapped files are about 20% faster in I/O last time I looked. You still have to actually do the I/O: mapping just saves some data copying. I would look at increasing BUFFER_SIZE to at least 32k. Also the size of cbuf, which should be a local variable in this method, not a member variable, so it will be thread-safe. It might be worth not compressing the files under a certain size threshold, say 10k.
Also you should be closing isr here, not in.
It might be worth trying putting another BufferedInputStream on top of the GZIPInputStream, as well as the one underneath it. Get it to do more at once.