I have a url of a file on the Internet. I need to calculate the SHA1 hash, and read this file by each line. I know how to do this, but I read this file twice which probably isn't a very good solution.
How can I do this more effectively?
Here is my code:
URL url = new URL(url);
URLConnection urlConnection = url.openConnection();
urlConnection.setConnectTimeout(1000);
urlConnection.setReadTimeout(1000);
logger.error(urlConnection.getContent() + " ");
InputStream is = urlConnection.getInputStream();
// first reading of file is:
int i;
File file = new File("nameOfFile");
BufferedInputStream bis = new BufferedInputStream(is);
BufferedOutputStream bos =
new BufferedOutputStream(new FileOutputStream(file.getName()));
while ((i = bis.read()) != -1) {
bos.write(i);
}
bos.flush();
bis.close();
sha1(file);
// second reading of file is:
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = reader.readLine()) != null) {
// do something
}
protected byte[] sha1(final File file) throws Exception {
if (file == null || !file.exists()) {
return null;
}
final MessageDigest messageDigest = MessageDigest.getInstance(SHA1);
InputStream is = new BufferedInputStream(new FileInputStream(file));
try {
final byte[] buffer = new byte[1024];
for (int read = 0; (read = is.read(buffer)) != -1;) {
messageDigest.update(buffer, 0, read);
}
} finally {
IOUtils.closeQuietly(is);
}
return messageDigest.digest();
}
If you pass it through a DigestInputStream, it'll do the MessageDigest and still be usable as an InputStream.
DigestInputStream dis = new DigestInputStream(is,
MessageDigest.getInstance(SHA1));
BufferedInputStream bis = new BufferedInputStream(dis);
BufferedOutputStream bos = new BufferedOutputStream(
new FileOutputStream(file.getName()));
while ((i = bis.read()) != -1) {
bos.write(i);
}
bos.close();
return dis.getMessageDigest().digest();
Related
That's my android code which takes 30 min to copy a 3MB .log file into a .zip and gives lots of GC_FOR_ALLOC. I also tried to change buffersize from 1k to 8k
File tempFolder=new File(Environment.getExternalStorageDirectory()+LOG_FILE_DIRECTORY_TEMP);
String filePath= Environment.getExternalStorageDirectory()+LOG_FILE_DIRECTORY_TEMP+ "/";
String fileName ="";
String zipFileName="";
String date=DATE_FORMAT_TEMP.format(new Date());
fileName = Settings.LOG_FILE_PREFIX + date+"_" + IMEI +.log;
zipFileName = Settings.LOG_FILE_PREFIX + date+"_" + IMEI +.zip;
br = new BufferedReader(new InputStreamReader(new FileInputStream(file)));
String result="";
String line = "";
while((line = br.readLine())!=null)
result += line;
result = getHeader() + result;
fos = new FileOutputStream(file);
fos.write(result.getBytes());
File zipFile=new File(filePath+zipFileName);
iStream = new FileInputStream(file);
oStream = new FileOutputStream(zipFile);
zos = new ZipOutputStream(oStream);
ze = new ZipEntry(fileName);
zos.putNextEntry(ze);
byte[] buffer = new byte[1024];
while((length = iStream.read(buffer)) != -1)
zos.write(buffer, 0, length);
zos.flush();
oStream.flush();
In your code it looks like you are opening and reading the file twice (your BufferedReader and your iStream object). Also, you are loading the entire file into memory twice before writing anything to memory. That's still only 6MB but you probably are hitting your memory stack limit - unless you use android:largeHeap="true" in your manifest.
Before you do that though, try just reading/writing each part:
public void zip(String[] _files, String zipFileName) {
try {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(zipFileName);
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(
dest));
byte data[] = new byte[BUFFER];
for (int i = 0; i < _files.length; i++) {
Log.v("Compress", "Adding: " + _files[i]);
FileInputStream fi = new FileInputStream(_files[i]);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(_files[i].substring(_files[i].lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
}
out.close();
} catch (Exception e) {
e.printStackTrace();
}
}
Here is the reference: http://javatechig.com/android/how-to-programmatically-zip-and-unzip-file-in-android
My file unzips one folder and one file but does not unzip the rest for some reason. How can I make it so it unzips all of the zip files contents?
public void unzip(String filepath, String filename, String unzip_path) throws IOException {
InputStream is = new FileInputStream(filepath + filename);
Log.d("1st", filepath + filename);
ZipInputStream zis = new ZipInputStream(new BufferedInputStream(is));
try {
ZipEntry ze;
while ((ze = zis.getNextEntry()) != null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int count;
String filename_temp = ze.getName();
File fmd = new File(unzip_path + filename_temp);
Log.d("2nd", unzip_path + filename_temp);
if (!fmd.getParentFile().exists()) {
fmd.getParentFile().mkdirs();
}
FileOutputStream fout = new FileOutputStream(unzip_path + filename_temp);
while ((count = zis.read(buffer)) != -1) {
baos.write(buffer, 0, count);
byte[] bytes = baos.toByteArray();
fout.write(bytes);
baos.reset();
}
fout.close();
//}
}
} finally {
zis.close();
}
}
i'm trying to unGzip and unTar an inputStream in java , i have those methods :
public InputStream unTar(InputStream in) throws IOException {
TarInputStream myTarStream = new TarInputStream(in);
TarEntry entry = myTarStream.getNextEntry();
InputStream input = null;
while (entry != null) {
ByteArrayOutputStream output = new ByteArrayOutputStream();
byte[] buff = new byte[1024];
int read;
do {
read = myTarStream.read(buff);
if (read != -1) {
output.write(buff, 0, read);
}
} while (read != -1);
output.flush();
input = new ByteArrayInputStream(output.toByteArray());
entry = myTarStream.getNextEntry();
}
myTarStream.close();
return input;
}
public InputStream unGzipIt(InputStream in) {
byte[] buffer = new byte[1024];
InputStream outGZIPStream = null;
try {
ByteArrayOutputStream bytesOutput = new ByteArrayOutputStream();
GZIPInputStream gis = new GZIPInputStream(in);
int len;
while ((len = gis.read(buffer)) > 0) {
bytesOutput.write(buffer, 0, len);
}
in.close();
bytesOutput.close();
outGZIPStream = new ByteArrayInputStream(bytesOutput.toByteArray());
} catch (IOException e) {
e.printStackTrace();
}
return outGZIPStream;
}
the problem is when i passed an inputStream from a file in my local , it working,
but when i passed the inputStream from my server response , it doesn't work .
should i use reset and mark ? any help ? thank you
this i s how i'm getting the inputStream :
public InputStream getFolder(#PathParam("id") String envId, #PathParam("appName") String appName, #PathParam("imageType") String imageType,
#QueryParam("folderPath") String folderPath) throws EnvAutomationException, IOException {
Environment env = Envs.getEnvironmentManager().findEnvironment(envId);
ApplicationInstance appInst = env.getApplicationInstance(appName);
Container container = appInst.getContainer(imageType);
InputStream folderData = Envs.getContainerizationManager().getFolder(container, folderPath);
ByteArrayOutputStream output = new ByteArrayOutputStream();
byte[] buff = new byte[1024];
int size = 0;
int read;
do {
read = folderData.read(buff);
if (read != -1) {
size = size + read;
output.write(buff, 0, read);
}
} while (read != -1);
output.flush();
byte[] bo = output.toByteArray();
InputStream input = new ByteArrayInputStream(bo);
InputStream inputGZIP = gzipIt(input);
return inputGZIP;
}
since it's a .tar file , i gizip it and this is the method to gzip;
public InputStream gzipIt(InputStream source) {
byte[] buffer = new byte[1024];
InputStream outGZIPStream = null;
try {
ByteArrayOutputStream bytesOutput = new ByteArrayOutputStream();
GZIPOutputStream gzos = new GZIPOutputStream(bytesOutput);
int len;
int size = 0;
while ((len = source.read(buffer)) > 0) {
gzos.write(buffer, 0, len);
size = size + len;
}
source.close();
gzos.close();
byte[] bo = bytesOutput.toByteArray();
outGZIPStream = new ByteArrayInputStream(bytesOutput.toByteArray());
logger.info("folder tar size :" + size + " ; folder gzip size " + bo.length);
} catch (IOException e) {
e.printStackTrace();
}
return outGZIPStream;
}
An example would be a simple image.
I have tried so many things and it just refuses to work despite making a whole lot of sense.
What I've done so far is I'm able to grab 25 pictures and add them to
/sdcard/app name/sub/dir/filename.jpg
They all appear there according to the DDMS but they always have a filesize of 0.
I'm guessing it's probably because of my input stream?
Here's my function that handles the downloading and saving.
public void DownloadPages()
{
for (int fileC = 0; fileC < pageAmount; fileC++)
{
URL url;
String path = "/sdcard/Appname/sub/dir/";
File file = new File(path, fileC + ".jpg");
int size=0;
byte[] buffer=null;
try{
url = new URL("http://images.bluegartr.com/bucket/gallery/56ca6f9f2ef43ab7349c0e6511edb6d6.png");
InputStream in = url.openStream();
size = in.available();
buffer = new byte[size];
in.read(buffer);
in.close();
}catch(Exception e){
}
if (!new File(path).exists())
new File(path).mkdirs();
FileOutputStream out;
try{
out = new FileOutputStream(file);
out.write(buffer);
out.flush();
out.close();
}catch(Exception e){
}
}
}
It just keeps giving me 25 files in that directory but all of their file sizes are zero. I have no idea why. This is practically the same code I've used in a java program.
PS...
If you're gonna give me a solution... I've already tried code like this. It doesn't work.
try{
url = new URL(urlString);
in = new BufferedInputStream(url.openStream());
fout = new FileOutputStream(filename);
byte data[] = new byte[1024];
int count;
System.out.println("Now downloading File: " + filename.substring(0, filename.lastIndexOf(".")));
while ((count = in.read(data, 0, 1024)) != -1){
fout.write(data, 0, count);
}
}finally{
System.out.println("Download complete.");
if (in != null)
in.close();
if (fout != null)
fout.close();
}
}
Here's an image of what my directories look like
http://oi48.tinypic.com/2cpcprm.jpg
A bit change to your second option, try it as following way,
byte data[] = new byte[1024];
long total = 0;
int count;
while ( ( count = input.read(data)) != -1 )
{
total += count;
output.write( data,0,count );
}
This one is different in while statement while ((count = in.read(data, 0, 1024)) != -1)
Using Guava something like this should work:
String fileUrl = "xxx";
File file = null;
InputStream in;
FileOutputStream out;
try {
Uri url = new URI(fileUrl);
in = url.openStream();
out = new FileOutputStream(file)
ByteStreams.copy(in, out);
}
catch (IOException e) {
System.out.println(e.toString());
}
finally {
in.close();
out.flush();
out.close();
}
I have checked many code snippets, tried with and without buffer and I can't get to download whole file to SD card. The code I use currently is:
try {
url = new URL("http://mywebsite.com/directory/");
} catch (MalformedURLException e1) { }
String filename = "someKindOfFile.jpg"; // this won't be .jpg in future
File folder = new File(PATH); // TODO: add checking if folder exist
if (folder.mkdir()) Log.i("MKDIR", "Folder created");
else Log.i("MKDIR", "Folder not created");
File file = new File(folder, filename);
try {
conn = url.openConnection();
is = conn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int current = 0;
while ((current = bis.read()) != -1) {
baf.append((byte) current);
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(baf.toByteArray());
fos.close();
is.close();
} catch (IOException e) { }
This code creates directory on SD card but downloads only 77 bytes of files. What might be the problem?
The error here is that he was writing the count variable converted to byte datatype instead of the bytes read from the input stream (those should be stored in a temporary byte[] buffer via bis.read(buffer))
The proper code block should be:
BufferedInputStream bis = new BufferedInputStream(is);
FileOutputStream fos = new FileOutputStream(file);
int current = 0;
byte[] buffer = new byte[1024];
while ((current = bis.read(buffer)) != -1) {
fos.write(buffer, 0, current);
}
fos.close();
is.close();