Trying to consume json response from url runs out of memory, or have to declare android:largeHeap="true".
This is the url
https://www.mtgjson.com/json/AllCards.json
When I enter this url in web browser on my PC, the page is populated within a second. why does it take android such long time to finish this api request? EDIT: I realize the web browser does not load all the data immediately, i see it showing the data as it comes before scrolling down in the window.
Are there ways to make the android device respond as fast as the web browser on pc when accessing this url?
When using emulators, outOfMemoryExeption unless if I increase RAM size. on samsung S7 device, outOfMemoryException unless I have largeHeap="true" in the manifest.
Same url in web browser doesn't take as long.
#Override
protected String doInBackground(String... params) {
HttpURLConnection urlConnection = null;
BufferedReader reader = null;
String apiJsonStr = null;
try {
String baseUrl = "https://www.mtgjson.com/json/AllCards.json";
String uri = Uri.parse(baseUrl).buildUpon().toString();
URL url = new URL(baseUrl);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.connect();
InputStream inputStream = urlConnection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
buffer.append(line + "\n");
}
if (buffer.length() == 0) {
return null;
}
apiJsonStr = buffer.toString();
} catch (IOException e) {
return null;
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
if (reader != null) {
}
}
return apiJsonStr;
}
#Override
protected void onPostExecute(String stringReturn) {
super.onPostExecute(stringReturn);
mListener.allCardsResponse(stringReturn);
}
Before you venture that route of trying to fit a square peg in a round hole by using a big hammer to smash the hole. I suggest you think of a better approach.
First of all, I looked at the json document. My browser took 1 min to load it and browser these days have gotten pretty efficient so they start to stream chunks they receive immediately. Second the file in 65.5 MB uncompressed.
This is a disaster ready to happen when you publish the app. User experience is going to suck. Everyone internet speed is going to be different and not to mention their connection proximity to your server (network latency). If it's going to take forever to load this thing if it's happening synchronously which it looks like it is, user are going to abandon the app.
Here is what I propose you look into -
If you have control over URL which returns the JSON, see if you can
use pagination and only return a fraction that you are going to
display in your mobile app. I'm pretty sure you can't cram all these
information a 6' - 9' mobile screen.
Perhaps you can transform this endpoint to only return few interesting metadata and when users show interest, you pull down all the details of that specific metadata (which will be one json object in your list)? That's assuming you control the API.
If this is an API that you don't control, you can create an
intermediary service that will do #1 for you so you can better
optimize it.
Another approach if you don't control the URL and the content is
static enough you can package with your app or configure out a
background sync that chunks these into 1 MB each and works
asynchronously.
Related
I have observed that one of my api is taking much more time if called through Java (URLConnection or Apache Http Client or OKHttp) for the first time. For the subsequent calls, the time is much lesser.
Although Postman or curl.exe takes very less time(comparable to the second iterations of java)
For my machine, the first time overhead is around 2 secs. But on some machines this is rising to around 5-6 secs for the first time. Thereafter it is around 300 ms roundtrip.
Here is my sample code:
public static String DoPostUsingURLConnection(String s_uri) throws Exception {
try {
URL uri = new URL(s_uri);
HttpURLConnection connection = (HttpURLConnection) uri.openConnection();
// Logger.log("Opened Connection");
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
connection.setRequestProperty("Authorization", authorizationHeader);
// Create the Request Body
try (OutputStream os = connection.getOutputStream()) {
byte[] input = jsonRequestBody.getBytes("utf-8");
os.write(input, 0, input.length);
}
// Logger.log("Written Output Stream");
int responseCode = connection.getResponseCode();
InputStream is = null;
if (responseCode == HttpURLConnection.HTTP_OK)
is = connection.getInputStream();
else
is = connection.getErrorStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine).append("\n");
;
}
in.close();
return response.toString();
} catch (Exception ex) {
return ex.getMessage();
} finally {
// Logger.log("Got full response");
}
You can investigate where time is taken by logging OkHttp connections events.
https://square.github.io/okhttp/events/
It will be particularly relevant if you are getting an IPv4 address and IPv6 and one is timing out and the other one succeeding.
This is just a guess. But the way Http connection works, that when you invoke it for the first time the connection gets established and that takes time. After that Http protocol doesn't really close connection for some time in expectation that some more requests would come and the connection could be re-used. And in your case indeed you send subsequent requests that re-use the previously created connection rather then re-establishing it which is expansive. I have written my own Open Source library that has a simplistic Http client in it. I noticed the same effect that first request takes much longer time than subsequent requests. But that doesn't explain why in Postman and curl we don't see the same effect. Anyway, if you want to solve this problem and you know your URL in advance, send a request upon your app initialization (you can even do it in separate thread). That will solve your problem.
If you are interested to look at my library here is Javadoc link. You can find it as maven artifact here and on github here. Article about the library covering partial list of features here
I'm using HttpURLConnection to send JSON data from an Android Application to my Tomcat Server.
The POST works fine with small sized JSONs. On bigger data sets it fails with a FileNotFoundException.
What can it be?
Here's the code:
try {
URL url = new URL(urlIn);
strOut = "";
huc = (HttpURLConnection) url.openConnection();
huc.setRequestProperty("Connection", "Close");
huc.setRequestMethod("POST");
huc.setRequestProperty("User", userId);
huc.setRequestProperty("Action", action);
huc.setRequestProperty("JSON", jsonData);
huc.setConnectTimeout(10000);
in = new BufferedReader(new InputStreamReader(huc.getInputStream()));
while ((inputLine = in.readLine()) != null){
if (strOut.equalsIgnoreCase("")){
strOut = inputLine;
} else {
strOut = strOut + inputLine;
}
}
} catch (Exception e) {
strOut = "";
e.printStackTrace();
}
When jsonData get to a certain size (arround 10000 chars), the POST fails with the error mentioned. The content of the JSON does not have any special character.
Thanks in advance.
Best regards, Federico.
HTTPUrlConnection throws a FileNotFoundException if the server responds with a 404 response code, so the reason why this happens seems to be located on the server side rather than the client side. Most likely the server is configured to accept request headers up to a particular length and will return an error if that size is exceeded. A short Google-search brought up a couple of results, sizes of 16 KB are mentioned but shorter values are also reasonable.
As I mentioned in my comment to your question, you should change your process to receive the JSON-data (and the other values for User and Action as well BTW) as part of the request body, e.g. as url-encoded query string or as multipart formdata. Both ways are supported by HTTP client libraries you can use or are easily built manually.
After lots of reading and trying I gave up with configuring Tomcat to accept larger headers.
So I convinced the team in charge of the Tomcat app to make a servlet that is able to receive this data in the body, just as Lothar suggested.
Thanks!
I'm trying to use the open data sets that data.LACity.org publishes using Socrata software.
They have a Java API for it, but first I tried to just build and send a URL, as
a variant on the 'Sunshine' app several people have learned from on Udacity.
Now I'm actually building a URL, and then sending it out, but then I get a FileNotFoundException, as follows:
java.io.FileNotFoundException: http://data.lacity.org/resource/yv23-pmwf.json?$select=zip_code, issue_date, address_start, address_end, street_name, street_suffix, work_description, valuation&$where=issue_date >= '2015-02-27T00:00:00' AND zip_code = 90291
Here's the pisser: That whole URL is, as a final attempt, hardwired as a complete string, not built from pieces. The URL works if I plug it into Chrome, but not from my app.
But from my app, the old URL string that the Sunshine sample app builds, plugged in from logcat from a Sunshine run, to replace the URL on the lacity URL, well, that call works, and returns the JSON data.
So I'm doing something wrong when I call the LACity URL for Socrata data from my Android app. I've tried this both as https and http, and both failed. But the same code works when I call the weathermap data from the sample app.
Here are the two URLs:
http://api.openweathermap.org/data/2.5/forecast/daily?q=94043&mode=json&units=metric&cnt=7 <<< this works, both in Chrome and from Android
https://data.lacity.org/resource/yv23-pmwf.json?$select=zip_code, issue_date, address_start, address_end, street_name, street_suffix, work_description, valuation&$where=issue_date >= '2015-02-27T00:00:00' AND zip_code = 90291
This works in Chrome but not from Android.
Any suggestions would be appreciated. I'm going to try again to make heads or tails of the Socrata Soda2 Java API (and why, in this case, it might be necessary.)
Thanks
-k-
The immediate code fragment (pardon my newness to Android/Java):
final String PERMIT_BASE_URL = "one of the url strings above";
Uri builtUri = Uri.parse(PERMIT_BASE_URL).buildUpon()
.build();
URL url = new URL(builtUri.toString());
Log.v(LOG_TAG, "Build URL: " + url.toString());
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.connect();
InputStream inputStream = urlConnection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
//simplify debugging
buffer.append(line + "\n");
}
if (buffer.length() == 0) {
return null;
}
permitJsonStr = buffer.toString();
Log.v(LOG_TAG, "Permit JSON string: " + permitJsonStr);
} catch (IOException e) {
Log.e(LOG_TAG, "Error on Xoom", e);
// Nothing to parse.
return null;
} finally{
if (urlConnection != null) {
urlConnection.disconnect();
}
if (reader != null) {
try {
reader.close();
} catch (final IOException e) {
Log.e(LOG_TAG, "Error closing stream on Xoom", e);
}
}
Figured this out from the way this page highlighted the URLs in my question.
Spaces.
The call out of Android seems to cough because of the spaces in the URL string.
I closed them all up, but then the 'AND' caused issues.
Replaced it with &, now it works, hardwired.
I'll work on constructing it from input parameters, as intended, but I think this is OK.
As Emily Litella would say...
I'm trying to get an image hosting on our server available to be displayed on a client. As per the specs of the project:
"When a Client receives such a URL, it must download the
contents (i.e., bytes) of the file referenced by the URL.
Before the Client can display the image to the user, it must first retrieve (i.e., download) the bytes of the
image file from the Server. Similarly, if the Client receives the URL of a known data file or a field help file
from the Server, it must download the content of those files before it can use them."
I'm pretty sure we have the server side stuff down, because if I put the url into a browser it retrieves and displays just fine. So it must be something with the ClientCommunicator class; can you take a look at my code and tell me what the problem is? I've spent hours on this.
Here is the code:
Where I actually call the function to get and display the file: (This part is working properly insofar as it is passing the right information to the server)
JFrame f = new JFrame();
JButton b = (JButton)e.getSource();
ImageIcon image = new ImageIcon(ClientCommunicator.DownloadFile(HOST, PORT, b.getLabel()));
JLabel l = new JLabel(image);
f.add(l);
f.pack();
f.setVisible(true);
From the ClientCommunicator class:
public static byte[] DownloadFile(String hostname, String port, String url){
String image = HttpClientHelper.doGetRequest("http://"+hostname+":"+port+"/"+url, null);
return image.getBytes();
}
The pertinent httpHelper:
public static String doGetRequest(String urlString,Map<String,String> headers){
URL url;
HttpURLConnection connection = null;
try {
//Create connection
url = new URL(urlString);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
if(connection.getResponseCode() == 500){
return "failed";
}
//Get Response
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer response = new StringBuffer();
while((line = rd.readLine()) != null) {
response.append(line);
}
rd.close();
return response.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if(connection != null) {
connection.disconnect();
}
}
}
After that, it jumps into the server stuff, which as I stated I believe is working correctly because clients such as Chrome can retrieve the file and display it properly. The problem has to be somewhere in here.
I believe that it has to do with the way the bytes are converted into a string and then back, but I do not know how to solve this problem. I've looked at similar problems on StackOverflow and have been unable to apply them to my situation. Any pointers in the right direction would be greatly appreciated.
If your server is sending binary data, you do not want to use an InputStreamReader, or in fact a Reader of any sort. As the Java API indicates, Readers are for reading streams of characters (not bytes) http://docs.oracle.com/javase/7/docs/api/java/io/Reader.html, which means you will run into all sorts of encoding issues.
See this other stack overflow answer for how to read bytes from a stream:
Convert InputStream to byte array in Java
Do your homework.
Isolate the issue. Modify the server side to send only 256 all possible bytes. Do a binary search and reduce it to small set of bytes.
Use http proxy tools to monitor the bytes as they are transmitted. Fiddler in windows world. Find other ones for the *nix environments.
Then see where the problem is happening and google/bing the suspicions or share the result.
I am using Apache HTTPClient 4 to connect to twitter's streaming api with default level access. It works perfectly well in the beginning but after a few minutes of retrieving data it bails out with this error:
2012-03-28 16:17:00,040 DEBUG org.apache.http.impl.conn.SingleClientConnManager: Get connection for route HttpRoute[{tls}->http://myproxy:80->https://stream.twitter.com:443]
2012-03-28 16:17:00,040 WARN com.cloudera.flume.core.connector.DirectDriver: Exception in source: TestTwitterSource
java.lang.IllegalStateException: Invalid use of SingleClientConnManager: connection still allocated.
at org.apache.http.impl.conn.SingleClientConnManager.getConnection(SingleClientConnManager.java:216)
Make sure to release the connection before allocating another one.
at org.apache.http.impl.conn.SingleClientConnManager$1.getConnection(SingleClientConnManager.java:190)
I understand why I am facing this issue. I am trying to use this HttpClient in a flume cluster as a flume source. The code looks like this:
public Event next() throws IOException, InterruptedException {
try {
HttpHost target = new HttpHost("stream.twitter.com", 443, "https");
new BasicHttpContext();
HttpPost httpPost = new HttpPost("/1/statuses/filter.json");
StringEntity postEntity = new StringEntity("track=birthday",
"UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
HttpResponse response = httpClient.execute(target, httpPost,
new BasicHttpContext());
BufferedReader reader = new BufferedReader(new InputStreamReader(
response.getEntity().getContent()));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>30000) break;
}
return new EventImpl(buffer.toString().getBytes());
} catch (IOException ie) {
throw ie;
}
}
I am trying to buffer 30,000 characters in the response stream to a StringBuffer and then return this as the data received. I am obviously not closing the connection - but I do not want to close it just yet I guess. Twitter's dev guide talks about this here It reads:
Some HTTP client libraries only return the response body after the
connection has been closed by the server. These clients will not work
for accessing the Streaming API. You must use an HTTP client that will
return response data incrementally. Most robust HTTP client libraries
will provide this functionality. The Apache HttpClient will handle
this use case, for example.
It clearly tells you that HttpClient will return response data incrementally. I've gone through the examples and tutorials, but I haven't found anything that comes close to doing this. If you guys have used a httpclient (if not apache) and read the streaming api of twitter incrementally, please let me know how you achieved this feat. Those who haven't, please feel free to contribute to answers. TIA.
UPDATE
I tried doing this: 1) I moved obtaining stream handle to the open method of the flume source. 2) Using a simple inpustream and reading data into a bytebuffer. So here is what the method body looks like now:
byte[] buffer = new byte[30000];
while (true) {
int count = instream.read(buffer);
if (count == -1)
continue;
else
break;
}
return new EventImpl(buffer);
This works to an extent - I get tweets, they are nicely being written to a destination. The problem is with the instream.read(buffer) return value. Even when there is no data on the stream, and the buffer has default \u0000 bytes and 30,000 of them, so this value is getting written to the destination. So the destination file looks like this.. " tweets..tweets..tweeets.. \u0000\u0000\u0000\u0000\u0000\u0000\u0000...tweets..tweets... ". I understand the count won't return a -1 coz this is a never ending stream, so how do I figure out if the buffer has new content from the read command?
The problem is that your code is leaking connections. Please make sure that no matter what you either close the content stream or abort the request.
InputStream instream = response.getEntity().getContent();
try {
BufferedReader reader = new BufferedReader(
new InputStreamReader(instream));
String line = null;
StringBuffer buffer = new StringBuffer();
while ((line = reader.readLine()) != null) {
buffer.append(line);
if (buffer.length()>30000) {
httpPost.abort();
// connection will not be re-used
break;
}
}
return new EventImpl(buffer.toString().getBytes());
} finally {
// if request is not aborted the connection can be re-used
try {
instream.close();
} catch (IOException ex) {
// log or ignore
}
}
It turns out that it was a flume issue. Flume is optimized to transfer events of size 32kb. Anything beyond 32kb, Flume bails out. (The workaround is to tune event size to be greater than 32KB). So, I've changed my code to buffer 20,000 characters at least. It kind of works, but it is not fool proof. This can still fail if the buffer length exceeds 32kb, however, it hasn't failed so far in an hour of testing - I believe it has to do with the fact that Twitter doesn't send a lot of data on its public stream.
while ((line = reader.readLine()) != null) {
buffer.append(line);
if(buffer.length()>20000) break;
}