I am saving a file to disk after downloading it from server, but I believe it gets corrupted while saving on the disc. If the same file is downloaded using chrome on mac or using any other method, the file downloads and reads normally. The corruption seems to be in the saving process of the file. I am adding the code to help find out the problem. The file is a css file.
Corruption:
Some whitespace sort of characters appear when reading the file. A surprising thing that I tried and noticed is that if I reduce the BUFFER_SIZE to 32 from 4096, the file does not get corrupt, I couldn't figure out why. Also, reducing BUFFER_SIZE reduces whitespaces / corrupted characters.
Appreciate any pointers in the right direction. Thanks
private static final int BUFFER_SIZE = 4096;
// saves file to disk and returns the contents of the file.
public static String downloadFile(Context context, String filePath, String destParent) {
String content = null;
StringBuilder sb = new StringBuilder();
HttpURLConnection connection = null;
InputStream is = null;
FileOutputStream os = null;
String sUrl = Urls.makeWebAssetUrl(filePath); /// consider this my file URL
String destFile = getContextBaseDir(context) + (destParent != null ? File.separator + destParent : "") + File.separator + filePath;
try {
URL url = new URL(sUrl);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
int responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
File outFile = new File(destFile);
if (!outFile.getParentFile().exists()) {
if (!outFile.getParentFile().mkdirs()) {
throw new RuntimeException("Unable to create parent directories for " + filePath);
}
}
is = connection.getInputStream();
os = new FileOutputStream(outFile);
int bytesRead = 0;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = is.read(buffer)) != -1) {
sb.append(new String(buffer, 0, bytesRead, DEFAULT_ENCODING));
os.write(buffer);
}
content = sb.toString();
}
else {
LogUtils.LOGW(TAG, responseCode + " while connecting to " + sUrl + ": " + connection.getResponseMessage());
}
} catch(Exception e) {
LogUtils.LOGE(TAG, "Error while downloading " + sUrl, e);
} finally {
if (is != null) {
try {
is.close();
} catch (IOException e) {
LogUtils.LOGE(TAG, "Error closing inputStream while downloading " + sUrl, e);
}
}
if (os != null) {
try {
os.flush();
} catch (IOException e) {
LogUtils.LOGE(TAG, "Error flushing outputStream while downloading " + sUrl, e);
}
try {
os.close();
} catch (IOException e) {
LogUtils.LOGE(TAG, "Error closing outputStream while downloading " + sUrl, e);
}
}
}
return content;
}
os.write(buffer);
The problem is here. It should be:
os.write(buffer, 0, bytesRead);
I don't know why you are also accumulating the content in a StringBuffer and returning it as a String. That won't scale, and in any cast it's redundant. Remove.
Related
I need to download files with the possibility to download parts after disconnection. I have an error when trying to download a file that is already completely downloaded. I get 416 error code from the server when trying to get inputstream. I fixed it by adding a condition downloadedFileSize < urlConnection.getContentLength(). Here my code:
try {
File file = Path.of(downloadOutputFolder, fileName).toFile();
boolean append = false;
connection = (HttpURLConnection) url.openConnection();
if (file.exists()) {
connection.setRequestProperty("Range", String.format("bytes=%s-", file.length()));
append = true;
}
if (!file.exists() || file.exists() && file.length() < getFileSizeFromUrl(url)) {
try (InputStream downloadStream = connection.getInputStream();
FileOutputStream fileOutputStream = new FileOutputStream(file, append)) {
int byteRead;
byte[] buffer = new byte[BUFFER_SIZE];
while ((byteRead = downloadStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, byteRead);
}
}
}...
private int getFileSizeFromUrl(URL url) {
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) url.openConnection();
return connection.getContentLength();
} catch (IOException e) {
log.error("Error during open URL connection to {}", url, e);
} finally {
if (connection != null) {
connection.disconnect();
}
}
return -1;
}
Is it possible to handle these case with url connection api? Not by adding condition file.length()<getFileSizeFromUrl(url).
I am getting error when i am trying to decompressing the files coming from server,but i am getting error of invalid bit length in while loop.Is there any problem with code or encoding.
byte[] buffer = new byte[1024];
FileInputStream fileInput = null;
FileOutputStream fileOutputStream = null;
GZIPInputStream gzipInputStream = null;
System.out.println(source_compressed_filepath);
System.out.println(destinaton_decompressed_filepath);
try {
fileInput = new FileInputStream(source_compressed_filepath);
gzipInputStream = new GZIPInputStream(fileInput);
fileOutputStream = new FileOutputStream(destinaton_decompressed_filepath);
int len;
while ((len = gzipInputStream.read(buffer)) >=0) {
fileOutputStream.write(buffer, 0, len);
}
System.out.println("The file" + source_compressed_filepath + " was DeCompressed successfully!"
+ destinaton_decompressed_filepath);
}catch (IOException ex) {
System.out.println(" error in file decompression " + source_compressed_filepath);
} finally {
// close resources
try {
fileOutputStream.close();
gzipInputStream.close();
} catch (IOException e) {
}
}
I have implemented the download of files. It is downloading and saving it to memory.
Now my problem is, if there is folder and sub folder, I need to ZIP the folder and download the zip file and save it in memory. I tried lot but I did'nt find a solution. Can any one can guide me to get the solution?
Here my download single file code...
#Override
protected String doInBackground(String... sUrl) {
InputStream input = null;
OutputStream output = null;
ZipOutputStream zos = null;
HttpURLConnection connection = null;
try {
URL url = new URL(sUrl[0]);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
return "Server returned HTTP "
+ connection.getResponseCode() + " "
+ connection.getResponseMessage();
}
String fileName = tvtitle.getText().toString();
String fileExtension = tvtype.getText().toString();
File imageDirectory = new File(Path);
imageDirectory.mkdirs();
int fileLength = connection.getContentLength();
String _path = Path;
input = connection.getInputStream();
File outputFile = new File(_path, fileName + fileExtension);
output = new FileOutputStream(outputFile);
// zos = new ZipOutputStream(output);
byte data[] = new byte[4096];
long total = 0;
int count;
while ((count = input.read(data)) != -1) {
if (isCancelled()) {
input.close();
return null;
}
total += count;
if (fileLength > 0) // only if total length is known
publishProgress((int) (total * 100 / fileLength));
output.write(data, 0, count);
}
} catch (Exception e) {
return e.toString();
} finally {
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
}
if (connection != null)
connection.disconnect();
}
return null;
}
first add permission in manifest
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
call this method by passing "inputFolderPath" and "outputFolderPath"
private static void zipFolder(String inputFolderPath, String outZipPath) {
try {
FileOutputStream fos = new FileOutputStream(outZipPath);
ZipOutputStream zos = new ZipOutputStream(fos);
File srcFile = new File(inputFolderPath);
File[] files = srcFile.listFiles();
Log.d("", "Zip directory: " + srcFile.getName());
for (int i = 0; i < files.length; i++) {
Log.d("", "Adding file: " + files[i].getName());
byte[] buffer = new byte[1024];
FileInputStream fis = new FileInputStream(files[i]);
zos.putNextEntry(new ZipEntry(files[i].getName()));
int length;
while ((length = fis.read(buffer)) > 0) {
zos.write(buffer, 0, length);
}
zos.closeEntry();
fis.close();
}
zos.close();
} catch (IOException ioe) {
Log.e("", ioe.getMessage());
}
}
I'm attempting to download a JSON from the following URL - http://api.crunchbase.com/v/1/companies.js - to a local file. I'm using Java 1.7 and the following JSON Libraries - http://www.json.org/java/ - to attempt to make it work.
Here's my code:
public static void download(String address, String localFileName) {
OutputStream out = null;
URLConnection conn = null;
InputStream in = null;
try {
URL url = new URL(address);
out = new BufferedOutputStream(
new FileOutputStream(localFileName));
conn = url.openConnection();
in = conn.getInputStream();
byte[] buffer = new byte[1024];
int numRead;
long numWritten = 0;
while ((numRead = in.read(buffer)) != -1)
{
out.write(buffer, 0, numRead);
numWritten += numRead;
System.out.println(buffer.length);
System.out.println(" " + buffer.hashCode());
}
System.out.println(localFileName + "\t" + numWritten);
} catch (Exception exception) {
exception.printStackTrace();
} finally {
try {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
} catch (IOException ioe) {
}
}
}
When I run the code everything seems to work until midway through the loop the program seems to stop and not continue reading the JSON Object.
Does anyone know why this would stop reading? How could I fix the issue?
Try This:
public void saveUrl(String filename, String urlString) throws MalformedURLException, IOException
{
BufferedInputStream in = null;
FileOutputStream fout = null;
try
{
in = new BufferedInputStream(new URL(urlString).openStream());
fout = new FileOutputStream(filename);
byte data[] = new byte[1024];
int count;
while ((count = in.read(data, 0, 1024)) != -1)
{
fout.write(data, 0, count);
}
}
finally
{
if (in != null)
in.close();
if (fout != null)
fout.close();
}
}
Does anyone know why this would stop reading? How could I fix the issue?
I can't see anything obviously wrong with the client-side code. In the absence of any other evidence on the client side, I'd look at the server-side logs to see if there are any clues there.
IMO, the most likely explanation is one of the following:
There's a bug in the server-side code that is generating the JSON and it is crashing halfway through.
The server (or a proxy / reverse proxy) has a timeout on the time allowed for some part of the interaction, and this particular request is taking too long.
I am building an Android Application and I need to download a file from a url, which is 33 MB large.
Here the download task:
try {
int MAX_BUFFER_SIZE = 4096;
URL mUrl = new URL(params[0]);
HttpURLConnection connection = (HttpURLConnection) mUrl.openConnection();
connection.setRequestMethod("GET");
long length = connection.getContentLength(), downloaded = 0;
int read;
byte [] buffer = new byte[(((int)length) > MAX_BUFFER_SIZE) ? MAX_BUFFER_SIZE : (int)length];
String filename = getFilename(mUrl);
File file = new File (SDCARD_ROOT);
if (!file.exists() || !file.isDirectory()){
file.mkdir();
}
this.filename = filename;
file = new File (SDCARD_ROOT + this.filename);
FileOutputStream fos = new FileOutputStream (file);
//Start downloading
InputStream stream = connection.getInputStream();
while ((read=stream.read(buffer)) > -1){
fos.write(buffer, 0, read);
downloaded += read;
publishProgress((int) ((float) downloaded/length * 100));
}
fos.close();
return 1;
} catch (Exception e){
Log.e("REV-PARTS", "Revolver parts error in DownloadTask: " + e.getMessage());
return 2;
}
It works right with small files (1-15 MB), but it will return a "unexpected end of stream" exception with large files.
Setting a chunk size seemed to work for me.
connection.setChunkedStreamingMode(1048576);
For large files you need to set the connection time out manually by using the following code.
I have set the time out to 3 minutes
connection.setConnectTimeout(180000);
connection.setReadTimeout(180000);
While you catch the exception, I try the method downContinue(). I can show my code:
private void downloadApk() {
thread1 = new Thread() {
public void run() {
File oFile = null;
try {
URL url = new URL(PQGLApplication.resrootURL + "apk/PQGLMap.apk");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
ReadableByteChannel channel =
Channels.newChannel(urlConnection.getInputStream());
oFile =
new File(Environment.getExternalStorageDirectory().getAbsolutePath()
+ "/" + "hy_ht_new/" + "test2" + ".apk");
oFile.setWritable(true);
oFile.setReadable(true);
if (oFile.exists()) {
oFile.delete();
}
FileOutputStream fos = new FileOutputStream(oFile);
fileSize = urlConnection.getContentLength();
ByteBuffer buffer = ByteBuffer.allocate(1024);
int noOfBytes = 0;
byte[] data = null;
sendApkMessage(0, 0);
while ((noOfBytes = channel.read(buffer)) > 0) {
data = new byte[noOfBytes];
System.arraycopy(buffer.array(), 0, data, 0, noOfBytes);
buffer.clear();
fos.write(data, 0, noOfBytes);
downLoadFileSize += noOfBytes;
sendApkMessage(1, downLoadFileSize);
}
fos.flush();
fos.close();
channel.close();
sendApkMessage(2, oFile.getAbsolutePath());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
downContinue();
}
};
};
thread1.start();
}
private void downContinue() {
continueTime++;
try {
if (continueTime == 3) {
continueTime = 0;
sendApkMessage(4, 0);
Log.e("what is the continuetime", "continueTime" + continueTime);
} else {
URL url = new URL(PQGLApplication.resrootURL + "apk/PQGLMap.apk");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
File oFile =
new File(Environment.getExternalStorageDirectory().getAbsolutePath() + "/"
+ "hy_ht_new/" + "test2" + ".apk");
RandomAccessFile oSavedFile = new RandomAccessFile(oFile, "rw");
FileOutputStream fos = new FileOutputStream(oFile);
ReadableByteChannel channel = Channels.newChannel(urlConnection.getInputStream());
// oSavedFile.seek(nPos);
ByteBuffer buffer = ByteBuffer.allocate(1024);
byte[] data = null;
int temp = 0;
sendApkMessage(3, oFile.getAbsolutePath());
while ((temp = channel.read(buffer)) > 0) {
data = new byte[temp];
System.arraycopy(buffer.array(), 0, data, 0, temp);
buffer.clear();
fos.write(data, 0, temp);
}
fos.flush();
fos.close();
oSavedFile.close();
sendApkMessage(2, oFile.getAbsolutePath());
continueTime = 0;
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
Log.e("what is the exception", e.toString() + continueTime);
downContinue();
}
}
This downContinue method is used to solve this problem. At least, the file is downloaded successfully!