EntityUtils toString takes too long on big responses - java

The following code :
//
// Define HTTP Post and content
//
HttpPost httppost = new HttpPost(url);
ByteArrayEntity be = new ByteArrayEntity(strPostData.getBytes("UTF-8"));
httppost.setEntity(be);
//
// Define HTTP Client
//
HttpClient httpclient = new DefaultHttpClient();
HttpParams httpParameters = httpclient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParameters, 10 * 1000);
//
// Sets the default socket timeout
// in milliseconds which is the timeout for waiting for data.
//
int timeoutSocket = 10000;
HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);
HttpResponse response = httpclient.execute(httppost);
//
// This line takes too long on big responses
//
String content = EntityUtils.toString(response.getEntity());
The last line (EntityUtils.toString) takes too long when my response contains a large amount of Bytes.
I'm using HTTP Post to retrieve PDF files (up to 500 Kb) and it may take 1 or 2 seconds on each request, which is too much.
EDIT For Information : The PDF file is base 64 encoded and wrapped into XML Tag (the string is parsed after reception).
Is there any way to get my String response a faster way ?
EDIT 2 : in order to know how much time took my EntityUtils.toString, I made a method :
public static void logger(String header, String content) {
Date dateActualLog = new Date();
long milliseconds = (dateActualLog.getTime() - dateLastLog.getTime());
Log.d(header, String.valueOf(milliseconds) + " => " + content);
dateLastLog = dateActualLog;
}
(fyi : dateLastLog is a static variable)
I modified the above code like this :
//
// This line takes too long on big responses
//
logger(TAG, "toString ...");
String content = EntityUtils.toString(response.getEntity());
logger(TAG, "toString OK");
Thanks in advance.

Well, the first simple thing to try would be to ensure that your web-server is supplying a correct ContentLength header in the HTTP response. Looking at some version of the source-code for HttpCore's EntityUtils class, we see that if this information is not available, it defaults to using a CharArrayBuffer of just 4k, buffering 1k of data when writing. On the 4th, 5th, and subsequent writes to the CharArrayBuffer (all the way up to 500, you say), it gradually increments the buffer by 1k ... using System.arrayCopy(). Yuck. There's your performance misery right there.
If speed is really important to you however, you'll avoid using EntityUtils entirely. It's just not responsible to turn a stream into a temporary 500k String ... especially on a phone! You'll need to find or write a Base64DecodingInputStream or Base64DecodingReader to wrap your InputStream from response.getEntity().getContent(), and feed that ... instead of a String ... to your parser.

Related

OkHttp3 How I can retrieve the Http Content Length from http header?

I try to download a file and check whether has been downloaded as a whole:
Request request = new Request.Builder().url("http://example.com/myfile.tar.gz").build();
Response response = client.newCall(request).execute();
// Status code checks goes here
if (downloadedTar == null) {
throw new SettingsFailedException();
}
ResponseBody downloadedTar = response.body();
double contentLength = Double.parseDouble(response.header("content-length"));
File file = File.createTempFile(System.currentTimeMillis()+"_file", ".tar.gz", getContext().getCacheDir());
FileOutputStream download = new FileOutputStream(file);
download.write(downloadedTar.body().bytes());
download.flush();
download.close();
if(file.exists() && (double)file.length() == contentLength){
// file has been downloaded
}
But the line:
double contentLength = Double.parseDouble(response.header("content-length"));
But response.header("content-length") is Null and has no integer value, I also tried the following variation response.header("Content-Length") and response.header("Content-Length") without any success.
So why I cannot retrieve Content-Length header and how I can ensure that file has been sucessfully downloaded?
Content-Length is removed in a number of cases such as Gzip responses
https://github.com/square/okhttp/blob/3ad1912f783e108b3d0ad2c4a5b1b89b827e4db9/okhttp/src/jvmMain/kotlin/okhttp3/internal/http/BridgeInterceptor.kt#L98
But generally isn't guaranteed to be present for streamed responses (chunked of h2).
You should try to avoid requiring content-length as it is guaranteed to be present and may change. Also you can probably optimise your IO with
Okio.buffer(Okio.sink(file)).writeAll(response.body().source())
or kotlin
file.sink().buffer().writeAll(response.body!!.source())

Google Maps Geolocation API only ever returning location of 1st tower in my array

I'm trying to make use of the Google Maps Geolocation API. I'm POST-ing a JSON file only containing cell towers to https://www.googleapis.com/geolocation/v1/geolocate?key=MYKEY.
Example JSON:
{
"cellTowers":[
{
"cellId":65535,
"locationAreaCode":5160,
"mobileCountryCode":234,
"mobileNetworkCode":10,
"signalStrength":-53
},
{
"cellId":34177,
"locationAreaCode":5160,
"mobileCountryCode":234,
"mobileNetworkCode":10,
"signalStrength":-55
}
]}
This has been validated correct JSON. I'm however having issues getting decent data back from the API.
I've implemented the POST in Java using the Apache HTTPClient & HTTPPost libraries. I get a valid response back, but it's always the LAT,LNG of the first cell tower in the cellTowers array. As if the API is reading the first cell tower, getting the location then ignoring the rest. Of course I've validated that this is the behavior by reversing and shuffling the list and the response changes each time to the first towers location.
Here's my code:
HttpParams httpParams = new BasicHttpParams();
ConnRouteParams.setDefaultProxy(httpParams, new HttpHost(PROXY_HOST, PROXY_PORT));
HttpConnectionParams.setConnectionTimeout(httpParams, TIMEOUT_MILLISEC);
HttpConnectionParams.setSoTimeout(httpParams, TIMEOUT_MILLISEC);
HttpClient client = new DefaultHttpClient(httpParams);
HttpPost request = new HttpPost(GOOGLE_GEOLOCATION_API);
request.setHeader("Content-type", "application/json");
try {
StringEntity se = new StringEntity(json);
request.setEntity(se);
HttpResponse response = client.execute(request);
if(response!=null){
String jsonResult = EntityUtils.toString(response.getEntity());
GeolocationResponse geolocationResponse = gson.fromJson(jsonResult, GeolocationResponse.class);
logger.info(jsonResult);
logger.info("Est. Location: " + geolocationResponse.toString());
}
} catch (Exception e) {
e.printStackTrace();
}
Does the Geolocation API only support one cell tower at a time? I thought that it would be more clever than this, using some sort of signal propagation and distance estimation across all the towers.
In a recent communication with Google about our mapping API subscription, I asked for some general info on the Geo-location API. Turns out it doesn't do any complex triangulation of cell tower signals.
They just use the first one considered valid. Reference: http://code.google.com/p/gmaps-api-issues/issues/detail?id=6929

HttpPost not sending params once over a certain size

Here is the code:
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(SERVER_URL);
NameValuePair[] data = {
new NameValuePair("html", html)
};
method.setRequestBody(data);
Once the value for the html var goes over a certain size all params become null for the receiving URL. Any ideas why?
This doesn't sound like a fault on the client side. I suspect that you may be hitting a server-side limit; e.g. a request-size limit specified in the web container configuration ... or a front-end.

Java Proxy Servlet for submitting files

I'm attempting to use Panda with my GWT application. I can upload videos directly to my panda server using
POST MY_PANDA_SERVER/videos/MY_VIDEO_ID/upload
However I would like hide my panda server behind my J2EE (glassfish) server. I would like to achieve this:
Start upload to some servlet on my J2EE server
Authenticate user
POST the file to my panda server while still uploading to servlet
Ideally I would like to never store the file on the J2EE server, but just use it as a proxy to get to the panda server.
Commons FileUpload is nice, but not sufficient in your case. It will parse the entire body in memory before providing the file items (and streams). You're not interested in the individual items. You basically just want to stream the request body from the one to other side transparently without altering it or storing it in memory in any way. FileUpload would only parse the request body into some "useable" Java objects and HttpClient would only create the very same request body again based on those Java objects. Those Java objects consumes memory as well.
You don't need a library for this (or it must be Commons IO to replace the for loop with an oneliner using IOUtils#copy()). Just the basic Java NET and IO API's suffices. Here's a kickoff example:
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
URLConnection connection = new URL("http://your.url.to.panda").openConnection();
connection.setDoOutput(true); // POST.
connection.setRequestProperty("Content-Type", request.getHeader("Content-Type")); // This one is important! You may want to check other request headers and copy it as well.
// Set streaming mode, else HttpURLConnection will buffer everything.
int contentLength = request.getContentLength();
if (contentLength > -1) {
// Content length is known beforehand, so no buffering will be taken place.
((HttpURLConnection) connection).setFixedLengthStreamingMode(contentLength);
} else {
// Content length is unknown, so send in 1KB chunks (which will also be the internal buffer size).
((HttpURLConnection) connection).setChunkedStreamingMode(1024);
}
InputStream input = request.getInputStream();
OutputStream output = connection.getOutputStream();
byte[] buffer = new byte[1024]; // Uses only 1KB of memory!
for (int length = 0; (length = input.read(buffer)) > 0;) {
output.write(buffer, 0, length);
output.flush();
}
output.close();
connection.getInputStream(); // Important! It's lazily executed.
}
You can use apache commons file upload to receive the file. Then you can use http client to upload the file to your panda server with POST. With apache commons file upload you can process the file in memory so you don't have to store it.
Building upon Enrique's answer, I also recommend to use FileUpload and HttpClient. FileUpload can give you a stream of the uploaded file:
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload();
// Parse the request
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (item.isFormField()) {
System.out.println("Form field " + name + " with value "
+ Streams.asString(stream) + " detected.");
} else {
System.out.println("File field " + name + " with file name "
+ item.getName() + " detected.");
// Process the input stream
...
}
}
You could then use HttpClient or HttpComponents to do the POST. You can find an example here.
The best solution is to use apache-camel servlet component:
http://camel.apache.org/servlet.html

need code to Convert rupee to dollar in java

I am writing an application java ,
is any way to write a java code to convert rupee to u.s. dollar and it shoud fetch the current u.s. dollar.
I need the program in java
Thanks
Google is our friend. I found this webservice which offers currency conhttp://www.webservicex.net/WS/WSDetails.aspx?WSID=10
You'll either hardcode the exchange rate in your application or database, or you'll be fetching that in real time (or asynchronously & cache every x minutes) from a third party site, for example http://www.google.com/search?q=1+usd+in+inr
In order to do this you would need to fetch the current exchange rate from a web service. The easiest way, if you just need to fetch a page, is something like this:
InputStream is = new URL("http://someexchangesite.com...").openStream();
Then read and parse the InputStream to find the exchange rate, and then use it with some simple multiplication!
Please find the below code which returns json response for getting Conversion Rate.
HttpClient client = new HttpClient();
NameValuePair arg1 = new NameValuePair("method","runJob");
//Change your currency types here in which you would want to convert
NameValuePair arg2 = new NameValuePair("from","USD");
NameValuePair arg3 = new NameValuePair("to", "PKR");
//getting the method
GetMethod method = new GetMethod("http://rate-exchange.appspot.com/currency");
method.setQueryString(new NameValuePair[]{arg1, arg2, arg3});
//executes the link
client.executeMethod(method);
//getting response in string
JSONObject obj = new JSONObject(method.getResponseBodyAsString());
//getting rate from the json response
double rate = obj.getDouble("rate");
//closing conncetion
method.releaseConnection();
//returning value
return rate;

Categories