I'm downloading a zip file from a server but keep getting corrupted file. I have a slow connection and I know that the server keeps updating the file frequently. Is this why I get corrupted files? I would assume the network protocol should be smart enough to avoid this kind of situations.
private void downloadFile(String urlString, String fileName)
throws MalformedURLException, IOException {
InputStream input = new URL(urlString).openConnection().getInputStream();
FileOutputStream output = new FileOutputStream(fileName);
int bufferSize = 153600;
byte[] buffer = new byte[bufferSize];
int totalBytesRead = 0;
int bytesRead = 0;
while ((bytesRead = input.read(buffer)) > 0) {
output.write(buffer, 0, bytesRead);
buffer = new byte[bufferSize];
totalBytesRead += bytesRead;
}
output.close();
input.close();
}
Thanks!
It's nothing to do with the protocol, and everything to do with the server software you're using at the other end of your URL. Your code can only read what the server sends you. The server code needs to ensure that it either maintains a write lock on the file while it's streaming it out to you, or otherwise ensures you receive a valid copy of the (unmodified) file.
Related
I'm trying to transfer some image files from an android phone, over a socket, to a server. The only way I've found to do this on android so far is using a FileInputStream to read the image as a byte array and send this over the socket to be reconstructed on the server side. This works well, unfortunately Android (or java?) does not allow Metadata, in my case exif data, to be included in a FileInputStream. This means that my exif data is missing once the images are on the server.
I've tried to solve this issue using both ExifInterface, which doesn't seem to be able to read a lot of the exif data I need, and the Metadata library. The metadata library does seem to get all the exif data I want but I can't figure out how to write it out as bytes that can be sent over my stream, it only has a toString which gets rid of some of the data that needs to be transferred.
Ideally I'd love a way to transfer the file with it's metadata, however I'd be happy with a way to turn Metadata tags into bytes which I can add to my socket's output stream.
Here is the code which uploads files over the socket
FileInputStream in = new FileInputStream(lastSavedPath);
byte[] buffer = new byte[1024];
int length = 0;
while ((length = in.read(buffer, 0, buffer.length)) != -1){
outputStream.write(buffer, 0, length);
}
ExifInterface exifInterface = new ExifInterface(lastSavedPath);
Metadata metadata = ImageMetadataReader.readMetadata(new File(lastSavedPath));
for (Directory directory : metadata.getDirectories()){
for (Tag tag : directory.getTags()){
Log.d("Socket Listener", tag.toString());
if (tag.toString().indexOf("Exif")>=0)
Log.d("Socket exif", "Data"+exifInterface.getAttribute(tag.getTagName()));
}
}
outputStream.flush();
Log.d("Socket Listener", "Data has been sent");
in.close();
socket.close();
The issue here wasn't with android at all. I had read in another thread that the android FileInputStream did not include metadata but that was not the case. I believe now the issue was in my server side code. I've fixed the issue with the following code:
Server side (Needs to be in a try catch):
socket = new Socket(args[0], 8888);
dataOutputStream = new DataOutputStream(socket.getOutputStream());
dataInputStream = new DataInputStream(socket.getInputStream());
dataOutputStream.writeUTF(args[1]);
System.out.println("Saving image");
FileOutputStream fileout = new FileOutputStream("/home/jamie/Documents/UMDSummer16/Thermal/TemporalAnalysisSensor/SocketTest/"+args[2]);
byte[] bytes = new byte[1024];
int count;
while ((count = dataInputStream.read(bytes)) > 0){
fileout.write(bytes);
}
fileout.close();
dataInputStream.close();
Android side (also in a try catch):
FileInputStream in = new FileInputStream(lastSavedPath);
byte[] buffer = new byte[1024];
int length = 0;
while ((length = in.read(buffer, 0, buffer.length)) != -1) {
outputStream.write(buffer, 0, length);
}
outputStream.flush();
Log.d("Socket Listener", "Data has been sent");
in.close();
socket.close();
When try to convert InputStream into byte array to know size of the file being uploaded. I am able to get size, but InputStream.read() becomes -1. How to check file size before writing it to server?
My current code gives me size , but InputStream reaches the end.
private static byte[] readFully(InputStream input) throws IOException
{
byte[] buffer = new byte[8192];
int bytesRead;
byte []bytes=null;
ByteArrayOutputStream output = new ByteArrayOutputStream();
while ((bytesRead = input.read(buffer)) != -1)
{
System.out.println("Buffer is "+input.read(buffer));
output.write(buffer, 0, bytesRead);
}
bytes=output.toByteArray();
output.close();
return bytes;
}
If you are implementing a web-server in Java, please take a look at the following link:
http://www.prasannatech.net/2008/11/http-web-server-java-post-file-upload.html
You must try to read the incomming information until you found its boundary.
You can't use read() method now, because the InputStream may not be ready to be read yet.
I have a problem to send a file(not necessarily a txt file) over a socket.I have 2 classes:Server,Client.When I read from a socket output stream and want to write the bytes in a file,it looks working but when i open the file it has nothing.(corrupted showing the size=0 kb).I also want it to transfer all kind of file over a socket.I don't want to use appache commons net.
Here is my code
Server class
FileOutputStream toFile1 = new FileOutputStream(f);
BufferedOutputStream toFile= new BufferedOutputStream(toFile1);
BufferedInputStream bis=new BufferedInputStream(incoming.getInputStream());
byte[]buffer=new byte[2048];
int bytesRead=0;
while((bytesRead = bis.read(buffer)) >= 0)
{
toFile.write(buffer, 0, bytesRead);
}
toFile.close();
toFile1.close();
bis.close();
out.println("226 Connection Closed");
out.flush();
}
Client class
BufferedOutputStream output = new BufferedOutputStream(socket.getOutputStream());
byte[] buffer = new byte[60*2024];
int bytesRead = 0;
while ((bytesRead = input.read(buffer,0,60*1024)) != -1) {
output.write(buffer, 0, bytesRead);
}
The only way that can happen with that code is if you are sending a zero length file, or maybe reading from a file input stream that is already positioned at EOF, or else you are looking at the wrong file afterwards.
I got working over socket file sender, it worked perfectly, but I couldn't send large files with it. Always got heap error. Then I changed the code of client, so it would send file in chunks. Now I can send big files, but there is new problem. Now I recieve small files empty and larger files for example videos can't be played. Here is the code of client that sends file:
public void send(File file) throws UnknownHostException, IOException {
// Create socket
hostIP = "localhost";
socket = new Socket(hostIP, 22333);
//Send file
FileInputStream fis = new FileInputStream(file);
BufferedInputStream bis = new BufferedInputStream(fis);
DataInputStream dis = new DataInputStream(bis);
OutputStream os = socket.getOutputStream();
//Sending size of file.
DataOutputStream dos = new DataOutputStream(os);
dos.writeUTF(file.getName() + ":" + userName);
byte[] arr = new byte[1024];
try {
int len = 0;
while ((len = dis.read(arr)) != -1) {
dos.write(arr, 0, len);
}
} catch (IOException ex) {
ex.printStackTrace();
}
dos.flush();
socket.close();
}
and here is the server code:
void start() throws IOException {
// Starts server on port.
serverSocket = new ServerSocket(port);
int bytesRead;
while (true) {
connection = serverSocket.accept();
in = connection.getInputStream();
clientData = new DataInputStream(in);
String[] data = clientData.readUTF().split(":");
String fileName = data[0];
String userName = data[1];
output = new FileOutputStream("C:/" + fileName);
long size = clientData.readLong();
byte[] buffer = new byte[1024];
// Build new file
while (size > 0 && (bytesRead = clientData.read(buffer, 0, (int) Math.min(buffer.length, size))) != -1) {
output.write(buffer, 0, bytesRead);
size -= bytesRead;
}
output.close();
}
}
You failed to write out the length of the file to the stream in the client:
long size = clientData.readLong();
So that call in the server is reading the first 8 bytes of the actual file and who knows what that quantity is. You don't have to read the length from the stream since you only wrote a single file. After reading the filename, and username (not very secure is it?) you can just read the stream until EOF. If you ever wanted to send multiple files over the same open socket then you'd need to know the length before reading the file.
Also your buffers for reading are way to small. You should be at a minimum of 8192 instead of 1024. And you'll want to put all .close() in a finally block to make sure your server and clients shutdown appropriately if there is an exception ever.
When I am trying to download a large file which is of 260MB from server, I get this error: java.lang.OutOfMemoryError: Java heap space. I am sure my heap size is less than 252MB. Is there any way I can download large files without increasing heap size?
How I can download large files without getting this issue? My code is given below:
String path= "C:/temp.zip";
response.addHeader("Content-Disposition", "attachment; filename=\"test.zip\"");
byte[] buf = new byte[1024];
try {
File file = new File(path);
long length = file.length();
BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
ServletOutputStream out = response.getOutputStream();
while ((in != null) && ((length = in.read(buf)) != -1)) {
out.write(buf, 0, (int) length);
}
in.close();
out.close();
There are 2 places where I can see you could potentially be building up memory usage:
In the buffer reading your input file.
In the buffer writing to your output stream (HTTPOutputStream?)
For #1 I would suggest reading directly from the file via FileInputStream without the BufferedInputStream. Try this first and see if it resolves your issue. ie:
FileInputStream in = new FileInputStream(file);
instead of:
BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
If #1 does not resolve the issue, you could try periodically flushing the output stream after so much data is written (decrease chunk size if necessary):
ie:
try
{
FileInputStream fileInputStream = new FileInputStream(file);
byte[] buf=new byte[8192];
int bytesread = 0, bytesBuffered = 0;
while( (bytesread = fileInputStream.read( buf )) > -1 ) {
out.write( buf, 0, bytesread );
bytesBuffered += bytesread;
if (bytesBuffered > 1024 * 1024) { //flush after 1MB
bytesBuffered = 0;
out.flush();
}
}
}
finally {
if (out != null) {
out.flush();
}
}
Unfortunately you have not mentioned what type out is. If you have memory issues I guess it is ByteArrayOutpoutStream. So, replace it by FileOutputStream and write the byte you are downloading directly to file.
BTW, do not use read() method that reads byte-by-byte. Use read(byte[] arr) instead. This is much faster.
First you can remove the (in != null) from your while statement, it's unnecessary. Second, try removing the BufferedInputStream and just do:
FileInputStream in = new FileInputStream(file);
There's nothing wrong (in regard to memory usage) with the code you're show. Either the servlet container is configured to buffer the entire response (look at the web.xml configuration), or the memory is being leaked elsewhere.