We Keep Coding

Support for Content-Encoding and Content-Decoding with Vert.x WebClient

public VertxHttpClient(Vertx vertx) {
this(vertx, new WebClientOptions().setTryUseCompression(true));
}
when I add accept-encoding header to the request.
kernelHttpRequest.setHeader("Accept-Encoding", "gzip");
The Vert.x-WebClient/3.9.5 ignores this header and The response which i receive from server does not have "content-encoding" header option.
Rather it have header as "Transfer-Encoding": "chunked".
"headers": {
"Transfer-Encoding": [
"chunked"
],
How can i pass accept-encoding = gzip and decompress the response which i am getting from server with Vert.x-WebClient/3.9.5

you should create the client with the appropriate option independantly of chunking:
client = vertx.createHttpClient(createBaseClientOptions().setTryUseCompression(true));
The setTryUseCompression actually tell the client to set an accept-encoding header to gzip and will decompress the response when the server has set the content-encoding header to gzip.
This is explained in the documentation https://vertx.io/docs/vertx-core/java/#_enabling_compression_on_the_client

Use setTryUseCompression= true while creating your Webclient which tells client to set an accept-encoding header to gzip and will decompress under the hood the response when the server has set the content-encoding header to gzip.
For my use I could not afford to set setTryUseCompression on my whole WebClient as it was being used in a complicated application, which in turn was making my other calls to fail.
So I had to write a custom decoder to decompress the response . Here is what I did. May be it will be helpful for someone in the future.
private String decompressErrorBody(HttpResponse<Buffer> response) throws IOException {
if ("gzip".equals(response.getHeader("Content-Encoding"))) {
InputStream is = new ByteArrayInputStream(response.body().getBytes());
GZIPInputStream zipped = new GZIPInputStream(is);
BufferedReader r = new BufferedReader(new InputStreamReader(zipped));
StringWriter responseBody = new StringWriter();
PrintWriter responseWriter = new PrintWriter(responseBody);
String line = null;
while ((line = r.readLine()) != null) {
responseWriter.println(line);
}
return responseBody.toString();
} else {
return response.bodyAsString();
}
}

POST file (as binary stream) using java.net.HttpURLConnection as a param file=<file binary stream>

I am trying to upload (POST) a file to an endpoint using java.net.HttpURLConnection but I keep getting http code 400 (bad request).
I refered to Send File And Parameters To Server With HttpURLConnection in android API 23
but problem is that I need to send this file as request body param (file=).
Note: The files will be of small size only (4-5mb) so I am reading it entirely in memory.
Corresponding curl request is:
curl -X POST "API" -H "Content-Type: multipart/form-data" -F "file="
Excerpts of Code that I am using:
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("
<proxy host>", <proxy_port>));
HttpURLConnection http_conn = (HttpURLConnection)
url.openConnection(webproxy);
String authorization = getAuthorization(access_token);
http_conn.setRequestMethod("POST");
http_conn.setRequestProperty("Accept-Charset", "UTF-8");
http_conn.setRequestProperty("Authorization", authorization);
http_conn.setRequestProperty("Connection", "Keep-Alive");
http_conn.setRequestProperty("Content-Type", "multipart/form-data);
http_conn.setDoOutput(true);
http_conn.setDoInput(true);
DataOutputStream outputStream;
outputStream = new DataOutputStream(http_conn.getOutputStream());
File file_obj = new File(this.file);
byte[] allBytes = new byte[(int) file_obj.length()];
FileInputStream fileInputStream = new FileInputStream(file_obj);
outputStream.write("file=".getBytes("UTF-8")); <---Trying to add file param here
fileInputStream.read(allBytes);
outputStream.write(allBytes);
Post that I just read response using below piece of code (works fine for different GET requests):
InputStream inputStream = http_conn.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
String line = "";
while ((line = bufferedReader.readLine()) != null) {
data = data + line;
}
Note: I use java rarely an am not very familiar with it so please be descriptive in your response.

When looking at your curl command line, it shows that the file needs to be send as a multipart/form-data request. This is actually a complex way of formatting your data when it is requires.
An example of the format you need to send is:
Headers:
Content-Type: multipart/form-data; boundary=AaB03x
Body:
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
At the moment, your code is sending the file as a POST/GET formatted request, and this doesn't work as the backend isn't expecting that.
To solve this problem, we need to format the source files into the format required by the backend, and once you know that the "boundary" header option is just a randomly generated value, it becomes more easy to send the request.
String boundary = "MY_AWESOME_BOUNDARY"
http_conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try(DataOutputStream outputStream = new DataOutputStream(http_conn.getOutputStream())) {
File file_obj = new File(this.file);
// Write form-data header
outputStream.write(("--" + boundary + "\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Disposition: form-data; name=\"file\"; filename=\"file1.txt\"\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Type: text/plain\r\n").getBytes("UTF-8"));
outputStream.write(("\r\n").getBytes("UTF-8"));
// Write form-data body
Files.copy(file_obj.toPath(), outputStream)
// Write form-data "end"
outputStream.write(("--" + boundary + "--\r\n").getBytes("UTF-8"));
}
// Read backend response here
try(InputStream inputStream = http_conn.getInputStream()) {
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
StringBuilder lines = new StringBuilder(); // StringBuilder is faster for concatination than appending strings
while ((line = bufferedReader.readLine()) != null) {
lines.append(line);
}
System.out.println(lines);
}
Note that I used "try-with-resource" blocks, these blocks make sure that any external resources are closed and disposed when you are done using them, generally the open resource limit of the OS is very low, compared to the amount of memory your program has, so what happens is that your program could give weird errors that only happens after some time of running or when the user executes certain actions inside your application

The above didnt worked for me so I switched to different package (okhttp3), here is what worked for me:
File file_obj = new File(this.file);
String authorization = "my authorization string";
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("proxy", <port>));
OkHttpClient client = new OkHttpClient.Builder().proxy(webproxy).build();
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addFormDataPart("file", "filename",
RequestBody.create(MediaType.parse("application/octet-stream"), file_obj)).build();
Request request = new Request.Builder().header("Authorization", authorization).url(this.url).post(requestBody).build();
try (Response response = client.newCall(request).execute()){
if(!response.isSuccessful()) return "NA";
return (response.body().string());
}

HttpGet in Android/Java with GZip encoding

I am running MVC 4 on my server and to save a bit of data for my users I figured I would enable GZip encoding, to do this I simply used:
(C#)
Response.AddHeader("Content-Encoding", "gzip");
Response.Filter = new GZipStream(Response.Filter, CompressionMode.Compress);
In my android application I use:
(Java)
String response = "";
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse execute = client.execute(httpGet);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(
new InputStreamReader(content));
String s = "";
while ((s = buffer.readLine()) != null) {
response += s;
}
} catch (Exception e) {
e.printStackTrace();
}
return response;
When I use GZip the Java code nuts out and causes GC to run, I was never patient enough to wait for it to return.
When I took off GZip from the server it ran perfectly fine. The function to get the response returns straight away with no problem.
I tried adding this to the java code:
httpGet.addHeader("Accept-Encoding", "gzip");
With no success.
Question is, is there something I'm not getting? Can I not put the response in a stream if it is using GZip? Am I meant to use the stream and uncompress it after?
What am I doing wrong?

Instead of using
DefaultHttpClient client = new DefaultHttpClient();
you can use
ContentEncodingHttpClient client = new ContentEncodingHttpClient();
which is a subclass of DefaultHttpClient and supports GZIP content.
You need Apache HttpClient 4.1 for this.
If you have Apache HttpClient 4.2, you should use
DecompressingHttpClient client = new DecompressingHttpClient();
if you have Apache HttpClient 4.3, you should use the HttpClientBuilder

Http Response Entity on upload to S3 is not empty but is acting like it

I'm trying to upload a file to my Amazon S3 bucket with authorization. I know that my signature works: using the same url, headers, and signature with hurl.it leads to a successful upload.
I also know that regular file upload with my program works: if I turn off S3 security and remove the headers, the upload works fine.
HttpClient client = new DefaultHttpClient();
url = "http://mybucket.s3.amazonaws.com/test/" + filename;
HttpPut put = new HttpPut(url);
put.setEntity(entity);
put.addHeader("Date", formattedDate);
put.addHeader("Authorization",
"AWS " + c.getString(R.string.AWSAccessKeyId) +
":" + signature);
put.addHeader("Content-Type",contentType);
HttpResponse httpResponse = client.execute(put);
Log.e(TAG, allHeaders(httpResponse));
HttpEntity responseEntity = httpResponse.getEntity();
InputStream content = responseEntity.getContent();
InputStreamReader is = new InputStreamReader(content);
BufferedReader br = new BufferedReader(is,2000);
String read = br.readLine();
Log.e("AUTH", ">>>"+read+"<<<");
String response = "";
while (read != null){
Log.e("AUTH", read);
read = br.readLine();
response += read + "\n";
}
Log.e("Code:", ""+httpResponse.getStatusLine().getStatusCode());
It's also worth noting that I'm doing this asynchronously on an Android emulator.
The resulting headers (displayed using Log.e(TAG, allHeaders(httpResponse))) are:
Date: Thu, 31 May 2012 05:58:08 GMT
Connection: close
Server: AmazonS3
Transfer-Encoding: chunked
The response status code is 400: Bad Request.
So what's weird is that I get these headers, but the content stream from responseEntity.getContent() is completely empty - I do not enter the while loop.
I also tried switching to an HttpGet targeted towards www.google.com and this worked completely fine.
Am I doing something wrong with the chunked encoding? Any obvious mistakes?

I have no idea why this worked, but somehow my signature encoding was adding some hidden character that completely screwed with the whole http model. I just used the substring of the signature without that last character, and I was able to read the chunked stream.

Getting a compressed version of web page

I am using HttpClient 4.1 to download a web page. I would like to get a compressed version:
HttpGet request = new HttpGet(url);
request.addHeader("Accept-Encoding", "gzip,deflate");
HttpResponse response = httpClient.execute(request,localContext);
HttpEntity entity = response.getEntity();
response.getFirstHeader("Content-Encoding") shows "Content-Encoding: gzip"
however, entity.getContentEncoding() is null.
If I put:
entity = new GzipDecompressingEntity(entity);
I get:
java.io.IOException: Not in GZIP format
It looks like the resulting page is plain text and not compressed even though "Content-Encoding" header shows it's gzipped.
I have tried this on several URLs (from different websites) but get the same results.
How can I get a compressed version of a web page?

Don't use HttpClient if you don't want your API to handle mundane things like unzipping.
You can use the basic URLConnection class to fetch the compressed stream, as demonstrated by the following code :
public static void main(String[] args) {
try {
URL url = new URL("http://code.jquery.com/jquery-latest.js");
URLConnection con = url.openConnection();
// comment next line if you want to have something readable in your console
con.addRequestProperty("Accept-Encoding", "gzip,deflate");
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String l;
while ((l=in.readLine())!=null) {
System.out.println(l);
}
} catch (Exception e) {
e.printStackTrace();
}
}