We Keep Coding

How to download a remote file using Java

I'm trying to download a single file from a web server (http or https) using as few third party libraries as possible.
The method I've come up with is as follows:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
byte[] buffer;
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) { // TODO fix endless loop if server timeout
buffer = new byte[BUFFER_SIZE];
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}
However, it does by no means work as intended. I tried to download https://upload.wikimedia.org/wikipedia/commons/6/6d/Rubber_Duck_Florentijn_Hofman_Hong_Kong_2013d.jpg for example, the result was horrifying:
For some reason I was able to view the picture in IrfanView but not in any other viewer, so this is a re saved version.
I tried messing with the buffer size or downloading other images but the results are more or less the same.
If I look at the file, there are entire parts of the content simply replaced with dots:
I'm really lost on this one so thanks for any help :)

The problem occurs when there aren't 8 bytes of data to read. This leaves part of the array filled with zeros, which is why you're seeing so many in your hex editor. The solution is simple: replace out.write(buffer); with out.write(buffer, 0, n);. This tells the FileOutputStream to only read the bytes between indexes 0 and n.
Fixed code:
private static final int BUFFER_SIZE = 8;
public static boolean download(URL url, File f) throws IOException {
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
FileOutputStream out = new FileOutputStream(f);
BufferedInputStream in = new BufferedInputStream(conn.getInputStream());
// We can move the buffer declaration outside the loop
byte[] buffer = new byte[BUFFER_SIZE];
long dld = 0, expected = conn.getContentLengthLong(); // TODO expected will be -1 if the content length is unknown
while (true) {
int n = in.read(buffer);
if (n == -1) break;
else dld += n;
out.write(buffer, 0, n);
}
out.close();
System.out.println(dld + "B transmitted to " + f.getAbsolutePath());
return true;
}

Try something like this to download pictures
public static byte[] download(String param) throws IOException {
InputStream in = null;
ByteArrayOutputStream out = null;
try {
URL url = new URL(param);
HttpURLConnection con = (HttpURLConnection)url.openConnection();
con.setConnectTimeout(120000);
con.setReadTimeout(120000);
con.setRequestMethod("GET");
con.connect();
in = new BufferedInputStream(con.getInputStream());
out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
return out.toByteArray();
} finally {
try {
out.close();
} catch (Exception e1) {
}
try {
in.close();
} catch (Exception e2) {
}
}
}

how to write encoded stream in JSON and retrieve back in ajax?

I am geeting image in input stream and encoding it into base64 and sending it to client using JSON
Following is code snippet.
Server side :
JsonObject myObj = new JsonObject();
StringBuilder responseStrBuilder = new StringBuilder();
int ch =0; ;
sun.misc.BASE64Encoder encoder= new sun.misc.BASE64Encoder();
byte[] contents = new byte[5000000];
int bytesRead = 0;
String strFileContents;
while ((bytesRead = bin.read(contents)) != -1) {
responseStrBuilder.append(encoder.encode(contents).getBytes());
}
myObj.addProperty("1",responseStrBuilder.toString());
out.println(myObj.toString());
Client side ajax code :
success: function(result)
{
if(result)
{
$('#dynamicCamping01').html('<img src='+result[Object.keys(result)[0]]+'/>');
$('#dynamicCampingDesc01').html("<h3>"+allData[0]+"</h3>");
}
else
{
alert("Something went wrong while retriving events");
}
getting data at client side but image is not displaying.

This one worked :
BufferedInputStream bin = new BufferedInputStream(fin);
BufferedOutputStream bout = new BufferedOutputStream(out);
int ch =0; ;
sun.misc.BASE64Encoder encoder= new sun.misc.BASE64Encoder();
byte[] contents = new byte[5000000];
int bytesRead = 0;
String strFileContents;
while ((bytesRead = bin.read(contents)) != -1) {
bout.write(encoder.encode(contents).getBytes());
}
bout.close();
fin.close();
bin.close();
out.close();
and main important part is on client converting image into UTF-8 and Base64 decoding.
$(imageIDSData[i]).html('<img src="data:image/jpeg;charset=utf-8;base64,'+imageData[i]+'"/> ');
$(imageNameIDSData[i]).html("<h3>"+allData[i]+"</h3>");

JAVA not in gzip format error

I have an inputstream and I tried to process it but it gave me this error "not in gzip format" but the file is in gzip format "Content-Encoding: gzip"
protected String readResponse(InputStream is) throws IOException {
StringBuffer string;
int b;
byte[] buffer;
String eol, s = null;
GZIPInputStream gis;
int read;
int index;
eol = new String(new byte[] {(byte)0, (byte)0, (byte)-1, (byte)-1});
buffer = new byte[1];
string = new StringBuffer();
while ( (b = is.read()) > 0 ) {
buffer[0] = (byte)b;
s = new String(buffer);
string.append(s);
index = string.indexOf(eol);
if ( index > 0 && index == string.length() - 4 ) {
break;
}
}
System.out.println(string);
gis = new GZIPInputStream(is); << here I got the error
buffer = new byte[1024];
while ( (read = gis.read(buffer)) > 0 ) {
string.append(new String(buffer, 0, read));
}
return string.toString();
}
any thoughts?
thanks

Seeing this line:
eol = new String(new byte[] {(byte)0, (byte)0, (byte)-1, (byte)-1});
is enough to arrive to a conclusion: you are doomed from the start.
DO NOT USE STRING FOR BINARY DATA.
bytes and chars have no relationship to one another; what you are doing here is roughly equivalent to the following:
final CharsetDecoder decoder = Charset.defaultCharset()
.newDecoder().onMalformedInput(CodingErrorAction.REPLACE);
final ByteBuffer buf = ByteBuffer.wrap(new byte[]{...});
final CharBuffer cbuf = decoder.decode(buf);
final String eol = new String(cbuf.array());
Note the REPLACE action. Any unmappable byte sequence will trigger the decoder to output the Unicode replacement character, U+FFFD (looks familiar, right?).
Now try and put REPORT instead.
What is more, you use the default charset... Which differs from platform to platform.
Your code should really just read the input stream and return a byte array. use a ByteArrayOutputStream.
And if you want to write to a file directly, it's easy: use Files.copy().
Anyway, fixed that for you:
// Note: return code is byte[]
protected byte[] readResponse(final InputStream in)
throws IOException
{
try (
final InputStream gzin = new GzipInputSream(in);
final ByteArrayOutputStream out = new ByteArrayOutputStream();
) {
final byte[] buf = new byte[4096];
int bytesRead;
while ((bytesRead = gzin.read(buf)) != -1)
out.write(buf, 0, bytesRead);
return out.toByteArray();
}
}

The problem could be you're advancing the file pointer in the input stream before you pass it to GZIPInputStream. GZIPInputStream expects the first few bytes to be a standard header.
Try moving new GZIPInputStream(is); before your while loop

There is so many things wrong in your code..... But lets try anyway.
So you have ascii header and after that there shoulbe gzipped part? Gzip file always starts with id bytes. These have the fixed values 'ID1 = 31 (0x1f, \037), ID2 = 139 (0x8b, \213)'. Can you find those from your inputstream. There you should start the gzipstream.

I have tested this with a file composed from a few header lines, followed by an empty line, and an appended gzipped text file. The latter is written, unexpanded, to x.gz and unzipped and read from there, assuming that it is a text file. (If it is a binary file, a BufferedReader is pointless.)
try/with resources and catch should be added, but that's just a technicality.
InputStream is = ...;
StringBuilder lsb = new StringBuilder();
int c = -1;
while( (c = is.read()) != -1 ){
if( c == '\n' ){
String line = lsb.toString();
if( line.matches( "\\s*" ) ){
break;
}
System.out.println( line );
lsb.delete( 0, lsb.length() );
} else {
lsb.append( (char)c );
}
}
byte[] buffer = new byte[1024];
int nRead = 0;
OutputStream os = new FileOutputStream( "x.gz" );
while ( (nRead = is.read(buffer, 0, buffer.length )) > 0 ) {
os.write( buffer, 0, nRead );
}
os.close();
is.close();
InputStream gis = new GZIPInputStream( new FileInputStream( "x.gz" ) );
InputStreamReader isr = new InputStreamReader( gis );
BufferedReader br = new BufferedReader(isr);
String line;
while( (line = br.readLine()) != null ){
System.out.println("line: " + line );
}
br.close();

How to write files to an SD Card from the internet?

An example would be a simple image.
I have tried so many things and it just refuses to work despite making a whole lot of sense.
What I've done so far is I'm able to grab 25 pictures and add them to
/sdcard/app name/sub/dir/filename.jpg
They all appear there according to the DDMS but they always have a filesize of 0.
I'm guessing it's probably because of my input stream?
Here's my function that handles the downloading and saving.
public void DownloadPages()
{
for (int fileC = 0; fileC < pageAmount; fileC++)
{
URL url;
String path = "/sdcard/Appname/sub/dir/";
File file = new File(path, fileC + ".jpg");
int size=0;
byte[] buffer=null;
try{
url = new URL("http://images.bluegartr.com/bucket/gallery/56ca6f9f2ef43ab7349c0e6511edb6d6.png");
InputStream in = url.openStream();
size = in.available();
buffer = new byte[size];
in.read(buffer);
in.close();
}catch(Exception e){
}
if (!new File(path).exists())
new File(path).mkdirs();
FileOutputStream out;
try{
out = new FileOutputStream(file);
out.write(buffer);
out.flush();
out.close();
}catch(Exception e){
}
}
}
It just keeps giving me 25 files in that directory but all of their file sizes are zero. I have no idea why. This is practically the same code I've used in a java program.
PS...
If you're gonna give me a solution... I've already tried code like this. It doesn't work.
try{
url = new URL(urlString);
in = new BufferedInputStream(url.openStream());
fout = new FileOutputStream(filename);
byte data[] = new byte[1024];
int count;
System.out.println("Now downloading File: " + filename.substring(0, filename.lastIndexOf(".")));
while ((count = in.read(data, 0, 1024)) != -1){
fout.write(data, 0, count);
}
}finally{
System.out.println("Download complete.");
if (in != null)
in.close();
if (fout != null)
fout.close();
}
}
Here's an image of what my directories look like
http://oi48.tinypic.com/2cpcprm.jpg

A bit change to your second option, try it as following way,
byte data[] = new byte[1024];
long total = 0;
int count;
while ( ( count = input.read(data)) != -1 )
{
total += count;
output.write( data,0,count );
}
This one is different in while statement while ((count = in.read(data, 0, 1024)) != -1)

Using Guava something like this should work:
String fileUrl = "xxx";
File file = null;
InputStream in;
FileOutputStream out;
try {
Uri url = new URI(fileUrl);
in = url.openStream();
out = new FileOutputStream(file)
ByteStreams.copy(in, out);
}
catch (IOException e) {
System.out.println(e.toString());
}
finally {
in.close();
out.flush();
out.close();
}

No number of bytes received in java

I would like to determine the number of bytes downloaded from the following working URL connection:
I have following code to implement:
.......
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.connect();
InputStream is = connection.getInputStream(); // throws an IOException
DataInputStream dis = new DataInputStream(new BufferedInputStream(is));
FileOutputStream fos = new FileOutputStream("C:\\Picture.jpeg");
int read =0;
byte[] bytes = new byte[1024];
while((read = dis.read(bytes)) != -1)
{
fos.write(bytes, 0, read);
}
System.out.println(read + " byte(s) copied");
The output from the last line is as follows:
Opening connection to http://www.xyz.com//Picture.jpeg...
Copying image resource (type: application/jpeg, modified on: 02/02/2010 4:19:21 AM)...
-1 byte(s) copied
What is the error of my code. please help me

int read =0;
int reddit = 0;
byte[] bytes = new byte[1024];
while((read = dis.read(bytes)) != -1)
{
fos.write(bytes, 0, read);
reddit += read;
}
//your read variable must have the value -1 at this point
System.out.println(reddit + " byte(s) copied");

int totalBytes = 0;
...
while((read = dis.read(bytes)) != -1)
{
totalBytes += read;
fos.write(bytes, 0, read);
}