URL url = new URL("http://download.thinkbroadband.com/20MB.zip");
URLConnection connection = url.openConnection();
File fileThatExists = new File(path);
OutputStream output = new FileOutputStream(path, true);
connection.setRequestProperty("Range", "bytes=" + fileThatExists.length() + "-");
connection.connect();
int lenghtOfFile = connection.getContentLength();
InputStream input = new BufferedInputStream(url.openStream());
byte data[] = new byte[1024];
long total = 0;
while ((count = input.read(data)) != -1) {
total += count;
output.write(data, 0 , count);
}
in this code I try to resume download. Target file is 20MB. But when I stop download on 10mb, then contunue, I get file with filesize 30MB. It seems that it continue writing to file, but cant partly download from server. Wget -c works great with this file. How can I resume file download?
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if(ISSUE_DOWNLOAD_STATUS.intValue()==ECMConstant.ECM_DOWNLOADING){
File file=new File(DESTINATION_PATH);
if(file.exists()){
downloaded = (int) file.length();
connection.setRequestProperty("Range", "bytes="+(file.length())+"-");
}
}else{
connection.setRequestProperty("Range", "bytes=" + downloaded + "-");
}
connection.setDoInput(true);
connection.setDoOutput(true);
progressBar.setMax(connection.getContentLength());
in = new BufferedInputStream(connection.getInputStream());
fos=(downloaded==0)? new FileOutputStream(DESTINATION_PATH): new FileOutputStream(DESTINATION_PATH,true);
bout = new BufferedOutputStream(fos, 1024);
byte[] data = new byte[1024];
int x = 0;
while ((x = in.read(data, 0, 1024)) >= 0) {
bout.write(data, 0, x);
downloaded += x;
progressBar.setProgress(downloaded);
}
This is not my code, but it works.
I guess the problem you are facing is calling url.openStream() after url.openConnection().
url.openStream() is equivalent to url.openConnection().getInputStream(). Hence, you are requesting the url twice. Particularly the second time, it is not specifying the range property. Therefore download always starts at the beginning.
You should replace url.openStream() with connection.getInputStream().
This is what I am using to download the file in chunk Updating the UI with progress.
/*
* #param callback = To update the UI with appropriate action
* #param fileName = Name of the file by which downloaded file will be saved.
* #param downloadURL = File downloading URL
* #param filePath = Path where file will be saved
* #param object = Any object you want in return after download is completed to do certain operations like insert in DB or show toast
*/
public void startDownload(final IDownloadCallback callback, String fileName, String downloadURL, String filePath, Object object) {
callback.onPreExecute(); // Callback to tell that the downloading is going to start
int count = 0;
File outputFile = null; // Path where file will be downloaded
try {
File file = new File(filePath);
file.mkdirs();
long range = 0;
outputFile = new File(file, fileName);
/**
* Check whether the file exists or not
* If file doesn't exists then create the new file and range will be zero.
* But if file exists then get the length of file which will be the starting range,
* from where the file will be downloaded
*/
if (!outputFile.exists()) {
outputFile.createNewFile();
range = 0;
} else {
range = outputFile.length();
}
//Open the Connection
URL url = new URL(downloadURL);
URLConnection con = url.openConnection();
// Set the range parameter in header and give the range from where you want to start the downloading
con.setRequestProperty("Range", "bytes=" + range + "-");
/**
* The total length of file will be the total content length given by the server + range.
* Example: Suppose you have a file whose size is 1MB and you had already downloaded 500KB of it.
* Then you will pass in Header as "Range":"bytes=500000".
* Now the con.getContentLength() will be 500KB and range will be 500KB.
* So by adding the two you will get the total length of file which will be 1 MB
*/
final long lenghtOfFile = (int) con.getContentLength() + range;
FileOutputStream fileOutputStream = new FileOutputStream(outputFile, true);
InputStream inputStream = con.getInputStream();
byte[] buffer = new byte[1024];
long total = range;
/**
* Download the save the content into file
*/
while ((count = inputStream.read(buffer)) != -1) {
total += count;
int progress = (int) (total * 100 / lenghtOfFile);
EntityDownloadProgress entityDownloadProgress = new EntityDownloadProgress();
entityDownloadProgress.setProgress(progress);
entityDownloadProgress.setDownloadedSize(total);
entityDownloadProgress.setFileSize(lenghtOfFile);
callback.showProgress(entityDownloadProgress);
fileOutputStream.write(buffer, 0, count);
}
//Close the outputstream
fileOutputStream.close();
// Disconnect the Connection
if (con instanceof HttpsURLConnection) {
((HttpsURLConnection) con).disconnect();
} else if (con instanceof HttpURLConnection) {
((HttpURLConnection) con).disconnect();
}
inputStream.close();
/**
* If file size is equal then return callback as success with downlaoded filepath and the object
* else return failure
*/
if (lenghtOfFile == outputFile.length()) {
callback.onSuccess(outputFile.getAbsolutePath(), object);
} else {
callback.onFailure(object);
}
} catch (Exception e) {
e.printStackTrace();
callback.onFailure(object);
}
}
interface IDownloadCallback {
void onPreExecute(); // Callback to tell that the downloading is going to start
void onFailure(Object o); // Failed to download file
void onSuccess(String path, Object o); // Downloaded file successfully with downloaded path
void showProgress(EntityDownloadProgress entityDownloadProgress); // Show progress
}
public class EntityDownloadProgress {
int progress; // range from 1-100
long fileSize;// Total size of file to be downlaoded
long downloadedSize; // Size of the downlaoded file
public void setProgress(int progress) {this.progress = progress;}
public void setFileSize(long fileSize) {this.fileSize = fileSize;}
public void setDownloadedSize(long downloadedSize) {this.downloadedSize = downloadedSize;}
}
Check out this thread which has a problem similar to yours. If wget is working, then the server clearly supports resuming downloads. It looks like you're not setting the If-Range header as mentioned in the accepted answer of the above link. ie. add:
// Initial download.
String lastModified = connection.getHeaderField("Last-Modified");
// ...
// Resume download.
connection.setRequestProperty("If-Range", lastModified);
How about this?
public static void download(DownloadObject object) throws IOException{
String downloadUrl = object.getDownloadUrl();
String downloadPath = object.getDownloadPath();
long downloadedLength = 0;
File file = new File(downloadPath);
URL url = new URL(downloadUrl);
BufferedInputStream inputStream = null;
BufferedOutputStream outputStream = null;
URLConnection connection = url.openConnection();
if(file.exists()){
downloadedLength = file.length();
connection.setRequestProperty("Range", "bytes=" + downloadedLength + "-");
outputStream = new BufferedOutputStream(new FileOutputStream(file, true));
}else{
outputStream = new BufferedOutputStream(new FileOutputStream(file));
}
connection.connect();
inputStream = new BufferedInputStream(connection.getInputStream());
byte[] buffer = new byte[1024*8];
int byteCount;
while ((byteCount = inputStream.read(buffer)) != -1){
outputStream.write(buffer, 0, byteCount);
break;
}
inputStream.close();
outputStream.flush();
outputStream.close();
}
Used break; to test the code.. ;)
I have a way for your code to work.
First, check if the file exits or not
If the file exists, set the connection:
connection.setRequestProperty("Range", "bytes=" + bytedownloaded + "-");
If file does not exist, do the same download in a new file.
Since the question is tagged with Android:
Have you tried using DownloadManager.
It handles all this stuff nicely for you.
Related
I'm trying to download a single file from a web server (http or https) using as few third party libraries as possible.
The method I've come up with is as follows:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
byte[] buffer;
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) { // TODO fix endless loop if server timeout
buffer = new byte[BUFFER_SIZE];
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}
However, it does by no means work as intended. I tried to download https://upload.wikimedia.org/wikipedia/commons/6/6d/Rubber_Duck_Florentijn_Hofman_Hong_Kong_2013d.jpg for example, the result was horrifying:
For some reason I was able to view the picture in IrfanView but not in any other viewer, so this is a re saved version.
I tried messing with the buffer size or downloading other images but the results are more or less the same.
If I look at the file, there are entire parts of the content simply replaced with dots:
I'm really lost on this one so thanks for any help :)
The problem occurs when there aren't 8 bytes of data to read. This leaves part of the array filled with zeros, which is why you're seeing so many in your hex editor. The solution is simple: replace out.write(buffer); with out.write(buffer, 0, n);. This tells the FileOutputStream to only read the bytes between indexes 0 and n.
Fixed code:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
// We can move the buffer declaration outside the loop
byte[] buffer = new byte[BUFFER_SIZE];
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) {
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer, 0, n);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}
Try something like this to download pictures
public static byte[] download(String param) throws IOException {
InputStream in = null;
ByteArrayOutputStream out = null;
try {
URL url = new URL(param);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setConnectTimeout(120000);
con.setReadTimeout(120000);
con.setRequestMethod("GET");
con.connect();
in = new BufferedInputStream(con.getInputStream());
out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
return out.toByteArray();
} finally {
try {
out.close();
} catch (Exception e1) {
}
try {
in.close();
} catch (Exception e2) {
}
}
}
I just tried to download a file from a Website, i checked the URL and everything seems to be fine there, but when i try to do it like described it prints out that my update size is -1 Bytes, don't know why this is happening.
If anyone has a solution, i would be glad to here it.
My Code:
private void downloadFile(String link) throws MalformedURLException, IOException
{
URL url = new URL(link);
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();
long max = conn.getContentLength();
outText.setText(outText.getText()+"\n"+"Downloding file...\nUpdate Size(compressed): "+max+" Bytes");
System.out.println("Update Size --> " + max + " Bytes");
BufferedOutputStream fOut = new BufferedOutputStream(new FileOutputStream(new File("update.zip")));
byte[] buffer = new byte[32 * 1024];
int bytesRead = 0;
int in = 0;
while ((bytesRead = is.read(buffer)) != -1) {
in += bytesRead;
fOut.write(buffer, 0, bytesRead);
}
fOut.flush();
fOut.close();
is.close();
outText.setText(outText.getText()+"\nDownload Complete!");
}
Big Thanks already :)
The javadoc is your friend:
Returns:
the content length of the resource that this connection's URL references, -1 if the content length is not known, or if the content length is greater than Integer.MAX_VALUE.
(emphasis mine)
I am trying to create a program that will download resources from a webpage into a file. I created an mkir function that creates a directory whose name is a hexadecimal version of the hash of a given String. I then, created a saveResource function that saves a resource in a file, as well as in a byte array. However, when I try saving the resource into the file I get an error message stating: java.io.FileNotFoundException: 648451a1 (Is a directory)
Here are the functions:
public static File mkdir(String s) throws IOException
{
String dirname = s;
s = Integer.toHexString(dirname.hashCode());
File directory = new File(s);
if (!directory.exists() && !directory.mkdir())
throw new IOException("can't make directory for " + s);
return directory;
}
public static byte[] saveResource(File dir, String urlString,
String argURLString)
throws IOException, URISyntaxException
{
URL u = new URL(urlString);
URLConnection uc = u.openConnection();
urlString = uc.getContentType();
int contentLength = uc.getContentLength();
try (InputStream raw = uc.getInputStream()) {
InputStream in = new BufferedInputStream(raw);
byte[] data = new byte[contentLength];
int offset = 0;
while (offset < contentLength) {
int bytesRead = in.read(data, offset, data.length - offset);
if (bytesRead == -1) break;
offset += bytesRead;
}
if (offset != contentLength) {
throw new IOException("Only read " + offset
+ " bytes; Expected " + contentLength + " bytes");
}
urlString = u.getFile();
urlString = urlString.substring(urlString.lastIndexOf('/') + 1);
try (FileOutputStream fout = new FileOutputStream(dir)) {
fout.write(data);
fout.flush();
}
return data;
}
}
The error is simple and coming because you can't write data in directory.
Try to print dir.isDirectory() to confirm if it's directory. Since its part of argument, check the caller method.
I'm trying to perform an AsyncTask class in my Android application that analyzes the network connection speed in for downloading and uploading. I'm working on the download portion now, but I'm not getting results I expect. I'm testing on a Wifi network that gets 15Mbps down/up speeds consistently, however, the results I'm getting from my application are more around barely 1Mbps. When I run the speed test apk on the device I'm testing on that gets around 3.5Mbps. The function works, just seems to be half the speed it should be. Should the following code produce accurate results?
try {
String DownloadUrl = "http://ipv4.download.thinkbroadband.com:8080/5MB.zip";
String fileName = "testfile.bin";
File dir = new File (context.getFilesDir() + "/temp/");
if(dir.exists()==false) {
dir.mkdirs();
}
URL url = new URL(DownloadUrl); //you can write here any link
File file = new File(context.getFilesDir() + "/temp/" + fileName);
long startTime = System.currentTimeMillis();
Log.d("DownloadManager", "download begining: " + startTime);
Log.d("DownloadManager", "download url:" + url);
Log.d("DownloadManager", "downloaded file name:" + fileName);
/* Open a connection to that URL. */
URLConnection ucon = url.openConnection();
//Define InputStreams to read from the URLConnection.
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
//Read bytes to the Buffer until there is nothing more to read(-1).
ByteArrayBuffer baf = new ByteArrayBuffer(1024);
int current = 0;
while ((current = bis.read()) != -1) {
baf.append((byte) current);
}
long endTime = System.currentTimeMillis(); //maybe
/* Convert the Bytes read to a String. */
FileOutputStream fos = new FileOutputStream(file);
fos.write(baf.toByteArray());
fos.flush();
fos.close();
File done = new File(context.getFilesDir() + "/temp/" + fileName);
Log.d("DownloadManager", "Location being searched: "+ context.getFilesDir() + "/temp/" + fileName);
double size = done.length();
if(done.exists()) {
done.delete();
}
Log.d("DownloadManager", "download ended: " + ((endTime - startTime) / 1000) + " secs");
double rate = (((size / 1024) / ((endTime - startTime) / 1000)) * 8);
rate = Math.round( rate * 100.0 ) / 100.0;
String ratevalue;
if(rate > 1000)
ratevalue = String.valueOf(rate / 1024).concat(" Mbps");
else
ratevalue = String.valueOf(rate).concat(" Kbps");
Log.d("DownloadManager", "download speed: "+ratevalue);
} catch (IOException e) {
Log.d("DownloadManager", "Error: " + e);
}
Example output
10-08 15:09:52.658: D/DownloadManager(13714): download ended: 70 secs
10-08 15:09:52.662: D/DownloadManager(13714): download speed: 585.14 Kbps
Thanks in advance for the help. If there is a better method, please let me know.
Following on my comments, here is an example of how to read several bytes from the stream
//Define InputStreams to read from the URLConnection.
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
//I usually use a ByteArrayOutputStream, as it is more common.
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int red = 0;
// This size can be changed
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
baos.write(buf, 0, red);
}
What this does is it reads into a byte[] buffer, and return the amount of read bytes. This is in turn written to the OutputStream, specifying the amount of bytes to write.
ByteArrayOutputStream also have a toByteArray that behaves similarly.
Alternatively, you can also write directly to the file, if you consider that the write to file operation is significantly faster than the read function :
// Simply start by defining the fileoutputstream
FileOutputStream fos = new FileOutputStream(file);
int red = 0;
// This size can be changed
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
// And directly write to it.
fos.write(buf, 0, red);
}
long endTime = System.currentTimeMillis(); //maybe
// Flush after, as this may trigger a commit to disk.
fos.flush();
fos.close();
Moreover, if you really only care about the download speed, it is not mandatory to write to the file, or to anywhere, this would be sufficient :
long size = 0;
byte[] buf = new byte[1024];
while ((red = bis.read(buf)) != -1) {
size += red;
}
I am trying to download a large (11MB) JSON file from a web service using this code:
public static void downloadBigFile(final String serverUrl,
final String fileName) throws MalformedURLException, IOException {
System.out.println("Downloading " + serverUrl + " (" + fileName + ")");
URL url = new URL(serverUrl);
URLConnection con = url.openConnection();
con.setConnectTimeout(10000);
con.setReadTimeout(2 * 60 * 1000);
int totalFileSize = con.getContentLength();
System.out.println("Total file size: " + totalFileSize);
InputStream inputStream = con.getInputStream();
FileOutputStream outputStream = new FileOutputStream(fileName);
// Used only for knowing the amount of bytes downloaded.
int downloaded = 0;
final byte[] buffer = new byte[1024 * 8];
int bytesRead;
bytesRead = inputStream.read(buffer);
while (bytesRead != -1) {
downloaded += bytesRead;
outputStream.write(buffer, 0, bytesRead);
bytesRead = inputStream.read(buffer);
System.out.println(String.format("%d/%d (%.2f%%)", downloaded,
totalFileSize,
(downloaded * 1.0 / totalFileSize * 1.0) * 100));
}
System.out
.println(fileName + " downloaded! (" + downloaded + " bytes)");
inputStream.close();
outputStream.close();
}
However the call to con.getContentLength() blocks the thread for several minutes, while it downloads the whole file I presume.
The problem is I need a quick way to discover file size before the download starts so I can notify the user accordingly.
Note: already tried to call con.connect() and con.getHeaderField("Content-Length").
If the server does not specify the Content-Length header, the only way to get the content length is to download the whole file and see how big it is.