Deal with huge JSON responses - java

I think I need to rewrite some modules of my app because when the number of entities that are rendered increases, it fails and errors too. At this moment, I'm using Jackson and HttpClient. As much as I trust in Jackson, something tells me that the problem is the second lib. Can HttpClient deal with large responses? (e.g. this one that is about 400 lines)
Besides that, in my app, the way I parse the request goes something like this:
public Object handle(HttpResponse response, String rootName) {
try {
String json = EntityUtils.toString(response.getEntity());
// better "new BasicResponseHandler().handleResponse(response)" ????
int statusCode = response.getStatusLine().getStatusCode();
if ( statusCode >= 200 && statusCode < 300 ) {
return createObject(json, rootName);
}
else{
return null;
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public Object createObject (String json, String rootName) {
try {
this.root = this.mapper.readTree(json);
String className = Finder.findClassName(rootName);
Class clazz = this.getObjectClass(className);
return mapper.treeToValue(root.get(rootName), clazz);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
How I can improve this piece of code to be more efficient with large responses?
Thanks in advance!

There's no need to create the String json, as ObjectMapper#readTree can accept an InputStream as well. For example, this will be slightly more efficient:
public Object handle(HttpResponse response, String rootName) {
try {
int statusCode = response.getStatusLine().getStatusCode();
if ( statusCode >= 200 && statusCode < 300 ) {
return createObject(response.getEntity().getContent(), rootName);
}
else{
return null;
}
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public Object createObject (InputStream json, String rootName) {
try {
this.root = this.mapper.readTree(json);
String className = Finder.findClassName(rootName);
Class clazz = this.getObjectClass(className);
return mapper.treeToValue(root.get(rootName), clazz);
} catch (Exception e) {
throw new RuntimeException(e);
}
}

Ive had 1000+ line json responses handled without issue, so that shouldnt be a problem. As for better ways of doing it, Google GSON is amazing, itll map your json to you java object with no special parsing code whatsoever.

I guess you could read data to a good old StringBuffer. Something like
HttpEntity httpEntity = httpResponse.getEntity();
if (httpEntity != null) {
InputStream is = AndroidHttpClient.getUngzippedContent(httpEntity);
br = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder(8192);
String s;
while ((s = br.readLine()) != null) sb.append(s);
}

Related

Restlet Filter Post request

I would like to filter a Post request in Filter (prior it getting to the Resource).
To filter the request I want to retrieve a token from the bode request and do some testing on it.
Current Resource:
#Post
public JsonRepresentation init(JsonRepresentation jRep) {
String token = jRep.getJsonObject().getString("token");
.
.
.
}
Current Filter:
#Override
protected int beforeHandle(Request request, Response response) {
int result = STOP;
String token = (String) Request.getCurrent().getAttributes().get("token");
.
.
.
}
These code does not retrieve the token.
My question is how can I retrieve a body request?
You can directly get the payload text of the request from its associated entity object, as described below:
Representation repr = request.getEntity();
String content = repr.getText();
Hope it helps you,
Thierry
you can try something like this to retreive the body :
public static String getBody(HttpServletRequest request) throws IOException {
String body = null;
StringBuilder stringBuilder = new StringBuilder();
BufferedReader bufferedReader = null;
try {
InputStream inputStream = request.getInputStream();
if (inputStream != null) {
bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
char[] charBuffer = new char[128];
int bytesRead = -1;
while ((bytesRead = bufferedReader.read(charBuffer)) > 0) {
stringBuilder.append(charBuffer, 0, bytesRead);
}
} else {
stringBuilder.append("");
}
} catch (IOException ex) {
throw ex;
} finally {
if (bufferedReader != null) {
try {
bufferedReader.close();
} catch (IOException ex) {
throw ex;
}
}
}
body = stringBuilder.toString();
return body;
}
As it is dangerous to store request entities directly into memory (imagine if a client send a tera-bytes representation), the framework does not persist representations into memory by default, they can only be read once (from the socket).
I guess the answers to your issue may be read from here: Restlet reuse InputStream

How do I read HttpEntity without consuming it?

I've got org.apache.http.HttpResponse object, which I'm using at different places in the code. One of those places is for logging.
The problem is that when I run following log code:
HttpEntity entity = response.getEntity();
try {
String content = Base64.encodeToString(
EntityUtils.toByteArray(entity), Base64.DEFAULT);
sb.append(content + "\r\n");
} catch (Exception e) {
sb.append("\r\n\r\n====EXCEPTION=====\r\n" + e.toString()
+ "\r\n");
}
and than I try to read entry content in the actual processing code, that causes the code to throw following exception:
java.lang.IllegalStateException: Content has been consumed
My question is: how do I read the entity without consuming it in the log code?
UPDATE
here's the full code of the function I use to transform httpresponse to string:
static String toString(org.apache.http.HttpResponse response) {
try {
if (response == null) {
return "null";
}
StringBuilder sb = new StringBuilder();
sb.append("==============BEGIN HttpResponse================\r\n");
StatusLine sl = response.getStatusLine();
if (sl == null) {
sb.append("status line is null\r\n");
} else {
sb.append(String.format("%s %s\r\n", sl.getStatusCode(),
sl.getReasonPhrase()));
}
for (Header h : response.getAllHeaders()) {
if (h == null) {
sb.append("header is null\r\n");
continue;
}
sb.append(String.format("%s: %s\r\n", h.getName(), h.getValue()));
}
sb.append("\r\r\r\n");
HttpEntity entity = response.getEntity();
if (entity == null) {
sb.append("content is null");
} else {
try {
String content = Base64.encodeToString(
EntityUtils.toByteArray(entity), Base64.DEFAULT);
sb.append(content + "\r\n");
} catch (Exception e) {
sb.append("\r\n\r\n====EXCEPTION=====\r\n" + e.toString()
+ "\r\n");
}
}
sb.append("\r\n==============END HttpResponse================\r\n");
return sb.toString();
} catch (Exception e) {
return e.toString();
}
}
Ok. So what I ended up doing is implementing my own HttpEntity class, and than just using response.setEntity(...) to replace the previous entity. That class stores the result as binary array and returns it as many times as necessary.
It might give you some performance issues, but will work:
Example of my HttpClient with logging.
private CloseableHttpResponse invoke(HttpRequestBase http) {
try {
CloseableHttpResponse response = client.execute(http);
if (http instanceof HttpPost) {
InputStream inputStream = ((HttpPost) http).getEntity().getContent();
String body = IOUtils.toString(inputStream, Charset.defaultCharset());
HttpEntity respBody = response.getEntity();
String responseBody = StreamUtils.copyToString(respBody.getContent(), Charset.defaultCharset());
response.setEntity(new StringEntity(responseBody));
LOG.info(String.format("Sending request: [%s] %s => [%s] \nPayload:\n%s \nResponse:\n%s", http.getMethod(), http.getURI(), response.getStatusLine(), body, responseBody));
} else {
LOG.info(String.format("Sending request: [%s] %s => [%s]", http.getMethod(), http.getURI(), response.getStatusLine()));
}
return response;
} catch (IOException e) {
throw new RuntimeException("HTTP request failed: " + http.toString(), e);
}
}
Main idea is following:
1. make http call
2. copy to string your response body:
HttpEntity respBody = response.getEntity();
String responseBody = StreamUtils.copyToString(respBody.getContent(), Charset.defaultCharset());
log it
set new response entity like response.setEntity(new StringEntity(responseBody));
This example work good for small test framework, not sure it's good code for production application

PHP $_POST is empty or null

I'm desperately trying to solve an issue with my android app. I submit a List to my server with an enum set as a 'tag'. The PHP pages should look at this tag and then proceed to perform the associated functions and return as a json array or object. This works fine with one version of the app but a cloned version fails to fetch data. The PHP just jumps straight over tag checking at the isset tag and tag is not empty conditions so it must be flat out seeing an empty POST or the object I submit doesn't meet some requirement I'm unaware of.
I've looked through so many posts and searched and searched but haven't found a solution. Why would it work for one version of the app but not for the upgraded version, that hasn't made any changes to the methods used for sending data??
So here's what I'm dealing with. To begin with, an AsyncTask takes the objects and passes to a class that handles communication:
private class UpdateJobList extends AsyncTask<User, Void, Boolean> {
private List<Message> messages;
public UpdateJobList() {
super();
messages = new ArrayList<Message>();
}
#Override
protected Boolean doInBackground(User... params){
try {
CloudConnect cConn = new CloudConnect(sAddress);
this.messages = cConn.getAll(params[0]);
return true;
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return false;
}
#Override
protected void onPostExecute(Boolean result) {
if (true)
{
handleMessageList(messages);
}
}
}
Using the CloudConnect class to get single Json objects or an array of Objects:
public class CloudConnect {
private String site;
private InputStream is;
private Gson gson;
public CloudConnect(String site) throws MalformedURLException {
this.site = site;
this.gson = new GsonBuilder().setDateFormat("yyyy-MM-dd").create();
is = null;
}
public synchronized Message get(Message m) throws IOException {
Message msg = null;
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(this.site);
post.setEntity(new UrlEncodedFormEntity(validateMessage(m)));
HttpResponse response = client.execute(post);
StatusLine status = response.getStatusLine();
if ( status.getStatusCode() == 200) {
HttpEntity entity = response.getEntity();
is = entity.getContent();
try {
Reader read = new InputStreamReader(is);
String str = (String) gson.fromJson(read, Object.class);
JsonParser parser = new JsonParser();
JsonElement jElem = parser.parse(str);
JsonObject jObject = (JsonObject) jElem;
msg = gson.fromJson(jObject, Message.class);
is.close();
} catch (Exception e) {
e.printStackTrace();
}
}
return msg;
}
public synchronized List<Message> getAll(Message m) throws IOException {
List<Message> mList = new ArrayList<Message>();
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(this.site);
post.setEntity(new UrlEncodedFormEntity(validateMessage(m)));
HttpResponse response = client.execute(post);
StatusLine status = response.getStatusLine();
if ( status.getStatusCode() == 200) {
HttpEntity entity = response.getEntity();
is = entity.getContent();
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
JsonArray jArray = null;
JsonReader jReader = new JsonReader(reader);
jReader.setLenient(true);
JsonParser parser = new JsonParser();
if (parser.parse(jReader).isJsonArray()){
jArray = parser.parse(jReader).getAsJsonArray();
if ( m instanceof User ){
for (JsonElement je : jArray) {
mList.add(gson.fromJson(je, Job.class));
Log.d("json", je.toString());
}
} else if ( m instanceof Job ) {
for (JsonElement je : jArray) {
mList.add(gson.fromJson(je, Update.class));
Log.d("json", je.toString());
}
}
} else {
JsonElement jElem = parser.parse(jReader);
JsonObject jObject = (JsonObject) jElem;
Error msg = null;
msg = gson.fromJson(jObject, Error.class);
mList.add(msg);
}
is.close();
} catch (Exception e) {
e.printStackTrace();
}
}
return mList;
}
And the PHP code that checks for the tag:
if ( isset($_POST['messageType']) && $_POST['messageType'] != "") {
$tag = $_POST['messageType'];
//
//various functions depending on messageType tag here. Such as getUser($email).
//functions appear to work fine if the PHP doesn't find the initial conditions
//false and skips them all.
} else {
$response["success"] = 0;
$response["error"]["errorMsg"] = "Tags are null";
$response["error"]["messageType"] = $tag;
$response["error"]["varDump"] = var_dump($_POST);
echo json_encode($response);
}
Ok, now I get it. You do not have a problem on your Android code but in PHP (I came here because of the TAG Android). I am not an expert in PHP, but remember that isset ($ _POST ['messageType']) returns true only if the payload of your POST request contains something like: messageType=some_value. So you need to check if the value that you passsed to post.setEntity(value) is something in this format.
You can use tools like Fiddler to see the payload of your request and debug properly.

android use asynctask to parse json

I am working on android application and I need to parse my json object with data. How you can see I create JSONParser class and try to use asynctask but there is something wrong and I can't understand where is the problem. Every time I use it resultJSON is null. Hope that you can give me an advice!
public class JSONParser {
private String resultJSON;
public JSONArray getJSON(String url) throws JSONException {
Parser parser = new Parser();
parser.execute(url);
return json;
}
private class Parser extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
for (String url : urls) {
StringBuilder builder = new StringBuilder();
HttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse response = client.execute(httpGet);
StatusLine statusLine = response.getStatusLine();
int statusCode = statusLine.getStatusCode();
if (statusCode == 200) {
HttpEntity entity = response.getEntity();
InputStream content = entity.getContent();
BufferedReader reader = new BufferedReader(
new InputStreamReader(content));
String line;
while ((line = reader.readLine()) != null) {
builder.append(line);
}
resultJSON = builder.toString();
} else {
Log.e(JSONParser.class.toString(), "Failed to download file");
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
return resultJSON;
}
#Override
protected void onPostExecute(String result) {
super.onPostExecute(result);
try {
json = new JSONArray(result);
} catch (JSONException e) {
e.printStackTrace();
}
}
}
}
Why don't you JSONArray json = new JSONArray(resultJSON); do this on post execute method of async task .
And i will not suggest varevarao way , as it will create extra burden of one thread .
You should use the get() method of the AsyncTask class to retrieve the result of the task. It waits for the task to complete and gets the result (which means it'd be best if you enclose it within a separate thread with a progress dialog or just a background thread).
public JSONArray getJSON(String url) throws JSONException {
Parser parser = new Parser();
parser.execute(url);
resultJSON = parser.get(); // Probably put this in a Thread to avoid spending too much time waiting for a result on the main thread
JSONArray json = new JSONArray(resultJSON);
return json;
}
The problem is fixed. It's an awful workaround but it works. Add this line
while(json==null) {}
after calling the execute method.

Reading line from website

I am making a little program thath will read data from website. String in the html file is already managed every info is divided with ; . Now i should read complete line here is example of this line:
14:47;24.02.12;18.7°C;18.7°C;285;0.5m/s; 6:48;17:37; Warm ;36;1.8;0.0;
So first how should i read them with HTTP Get or is there anything other? And then i would like to save each info, they are seperated with ; into a variable. And how should i cut each info from this line.
You definitely need to do some homework, but this methods will help you:
public static String getContentFromUrl(String url) throws ClientProtocolException, IOException {
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
HttpResponse response;
response = httpClient.execute(httpGet);
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream inStream = entity.getContent();
String result = HttpService.convertStreamToString(inStream);
inStream.close();
return result;
}
return null;
}
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
This allows you to get data from a URL. Then lookup String.split to chop your string into usable entities.
Hope this helps!
Use GET request to fetch the data from the website
Separate the string-data from the HTML-markup
Parse the string into multiple strings or a list of strings, using ';' as the delimiter.
Android Java SDK String reference

Categories