We Keep Coding

Migration from HTTPClient 3.1 to 4.3.3, Method.getResponseBody(int)

I'm updating an old software using HTTPClient 3.1 to use HTTPClient 4.3.3.
I noticed that in the old code there is a specific requirement: when getting a remote page/resource the client is able to verify the dimension, generating an exception if the content is too big WITHOUT downloading the full resource.
This was accomplished in the following manner:
int status = client.executeMethod(method);
...
byte[] responseBody= method.getResponseBody(maxAllowedSize+1);
Notice the "+1" after maxAllowedSize: it's requested to have a proof that the original page/resource was in fact too big.
If the last byte was used, an exception was thrown; otherwise the page was processed.
I'm trying to implement the same thing in HTTPClient 4.3.3, but I can't find a way to download only a defined number of bytes from the server... this is critical in my application.
Can you help me? Thank you in advance.
Javadoc of the old getResponseBody(int) method: https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpMethodBase.html#getResponseBody(int)

One generally should be consuming content directly from the content stream instead of buffering it in an intermediate buffer, but this is roughly the same thing with 4.3 APIs:
CloseableHttpClient client = HttpClients.custom()
.build();
try (CloseableHttpResponse response = client.execute(new HttpGet("/"))) {
HttpEntity entity = response.getEntity();
long expectedLen = entity.getContentLength();
if (expectedLen != -1 && expectedLen > MAX_LIMIT) {
throw new IOException("Size matters!!!!");
}
ByteArrayOutputStream buffer = new ByteArrayOutputStream();
InputStream inputStream = entity.getContent();
byte[] tmp = new byte[1024];
int chunk, total = 0;
while ((chunk = inputStream.read(tmp)) != -1) {
buffer.write(tmp, 0, chunk);
total += chunk;
if (total > MAX_LIMIT) {
throw new IOException("Size matters!!!!");
}
}
byte[] stuff = buffer.toByteArray();
}

Can a byte stream be written straight to SDCard from HTTP bypassing the HEAP?

I'm downloading video files that are larger than the memory space that Android apps are given. When they're *on the device, the MediaPlayer handles them quite nicely, so their overall size isn't the issue.
The problem is that if they exceed the relatively small number of megabytes that a byte[] can be then I get the dreaded OutOfMemory exception as I download them.
My intended solution is to just write the incoming byte stream straight to the SD card, however, I'm using the Apache Commons library and the way I'm doing it tries to get the entire video read in before it hands it back to me.
My code looks like this:
HttpClient client = new HttpClient();
PostMethod filePost = new PostMethod(URL_PATH);
client.setConnectionTimeout(timeout);
byte [] ret ;
try{
if(nvpArray != null)
filePost.setRequestBody(nvpArray);
}catch(Exception e){
Log.d(TAG, "download failed: " + e.toString());
}
try{
responseCode = client.executeMethod(filePost);
Log.d(TAG,"statusCode>>>" + responseCode);
ret = filePost.getResponseBody();
....
I'm curious what another approach would be to get the byte stream one byte at a time and just write it out to disk as it comes.

You should be able to use the GetResponseBodyAsStream method of your PostMethod object and stream it to a file. Here's an untested example....
InputStream inputStream = filePost.getResponseBodyAsStream();
FileInputStream outputStream = new FileInputStream(destination);
// Per your question the buffer is set to 1 byte, but you should be able to use
// a larger buffer.
byte[] buffer = new byte[1];
int bytesRead;
while ((bytesRead = input.read(buffer)) != -1)
{
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();

Forwarding http package in java

I'm trying to write a HTTP proxy-server in java. My application takes a GET request from a browser and forwards it to its destination. I would like to read the headers of response package and then forward it back to the browser. This works great for me with text/html-content aslong as its not encoded in gzip. I've tried multiple ways to do this and I'm currently using a DataInputStream and a DataOutputStream but the browser only shows weird symbols.
Here is a simplified version of the code:
ArrayList<String> headerlist = new ArrayList<String>();
InputStream input = clientsocket.getInputStream();
dis = new DataInputStream(input);
serverinputstream = new InputStreamReader(input);
bufferreader = new BufferedReader(serverinputstream);
while(!(line = bufferedreader.readLine()).equals("")) {
headerlist.add(line);
}
PrintWriter pw = new PrintWriter(serveroutputstream, false);
DataOutputStream out = new DataOutputStream(serveroutputstream);
for (int i = 0; i < headerlist.size(); i++) {
pw.println(headerlist.get(i));
}
pw.println();
int bit;
while((bit = dis.read()) != -1) {
out.writeByte(bit);
}
out.flush();
dis.close();
out.close();
This code only handles data that isnt plain text but it doesnt seem to be working. Should I use another method or I am just doing something wrong?

I think you may be overcomplicating things a bit. Your proxy is just forwarding a request on to another destination. There's no reason for it to care about whether it is forwarding text or binary data. It should make no difference.
There's also no reason to read and write the headers individually. All you should need to do is copy the entire request body to the new output-stream.
What about something like:
//...
InputStream input = clientsocket.getInputStream();
streamCopy(input, serveroutputstream);
//...
public void streamCopy(InputStream in, OutputStream out) throws IOException {
int read = 0;
byte[] buffer = new byte[4096];
while ((read = in.read(buffer)) != -1) {
out.write(buffer, 0, read);
}
}

Socket: premature end of JPEG file

I'm trying to send an image file from a server to a client via a socket. The socket was previously used to send some strings from the server to the client (with buffered input/output streams).
The trouble is the image file can't be received properly, with "Premature end of JPEG file" error.
The server first sends the file size to the client, the client then creates a byte[] of that size, and starts to receive the file.
Here are the codes:
Server:
DataOutputStream dos = new DataOutputStream(socket.getOutputStream());
//Send file size
dos.writeInt((int) file.length());
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
byte[] fileBytes = new byte[bis.available()];
bis.read(fileBytes);
bis.close();
BufferedOutputStream bos = new BufferedOutputStream(socket.getOutputStream());
bos.write(fileBytes);
bos.flush();
Client:
DataInputStream dis = new DataInputStream(socket.getInputStream());
//Receive file size
int size = dis.readInt();
BufferedInputStream bis = new BufferedInputStream(socket.getInputStream());
byte[] fileBytes = new byte[size];
bis.read(fileBytes, 0, fileBytes.length);
More interestingly, if I let server sleep for about 2 seconds between sending the file size and writing the byte[], then the image is received properly. I wonder if there's some kind of race condition between the server and the client

The error is most likely here:
byte[] fileBytes = new byte[bis.available()];
The method available does not return the size of the file. It might return only the size of the input buffer, which is smaller than the size of the file. See the API documentation of the method in BufferedInputStream.
Also, read in the line below is not guaranteed to read the whole file in one go. It returns the number of bytes that were actually read, which can be less than what you asked for. And in the client code, you are using read in the same way, without actually checking if it read all the data.

Please check commons-io with FileUtils and IOUtils. This should make work a lot easier.
http://commons.apache.org/io/

The correct way to copy a stream in Java is as follows:
int count;
byte[] buffer = new byte[8192]; // more if you like, but over a network it won't make much difference
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Your code fails to logically match this at several points.
Also available() is not a valid way to determine either a file size or the size of an incoming network transmission - see the Javadoc. It has few if any correct uses and these aren't two of them.

Write a binary downloaded file to disk in Java

I have a software that allow to write add-on in javascript files (.js) that allow to use Java function (I don't know if this is common, I never saw java call in javascript file before)
I need to download a binary file from a webserver and write it to the hard drive. I tried the following code:
baseencoder = new org.apache.commons.codec.binary.Base64();
url = new java.net.URL("https://server/file.tgz");
urlConnect = url.openConnection();
urlConnect.setDoInput(true);
urlConnect.setDoOutput(true);
urlConnect.setRequestProperty("authorization","Basic "+ java.lang.String(baseencoder.encodeBase64(java.lang.String( username + ":" + password ).getBytes())));
urlConnect.setRequestProperty("content-type","application/x-www-form-urlencoded");
is = new java.io.DataInputStream(urlConnect.getInputStream());
fstream = new FileWriter("C:\\tmp\\test.tgz");
out = new BufferedWriter(fstream);
while((data = is.read()) != -1){
out.write(data);
}
out.close();
is.close();
The resulting file is no longer a valid gzip archive. I'm sorry if I did a huge error but I'm not a programmer and don't know Java too much.

Don't use a FileWriter - that's trying to convert the data into text.
Just use FileOutputStream.
byte[] buffer = new byte[8 * 1024];
InputStream input = urlConnect.getInputStream();
try {
OutputStream output = new FileOutputStream(filename);
try {
int bytesRead;
while ((bytesRead = input.read(buffer)) != -1) {
output.write(buffer, 0, bytesRead);
}
} finally {
output.close();
}
} finally {
input.close();
}

I know this question is already answered, but a simpler approach is to use Apache Commons IO's IOUtils.copy() method, which can fully copy an InputStream to an OutputStream.

DataInputStream is meant for reading Java primitives, not for generic data.
It's also redundant, as urlConnect.getInputStream(); already returns an InputStream, and all InputStreams support read().
is = urlConnect.getInputStream();
P.S. This is assuming is and bis are the same variable. Otherwise, you're reading the wrong stream in the loop.

Just read about LimitInputStream sounds like it does exactly what you are doing, buffering the input stream for greater efficiency.

You can even use NIO FileChannel#transferFrom method.
URL website = new URL(urlToDownload);
try (ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(filePath);) {
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
}
Reference link1, link2