We Keep Coding

Avoiding OutOfMemoryError

I have code in which I am given a large JSON string (could be anywhere from 50MB to 250MB) that is an array of JSON objects to be parsed and sanitized then serialized to a file. Everything was going fine with 50MB JSON strings but when the string gets over a hundred or so MB my app crashes with OutOfMemoryError. I know I can increase the size of the heap but would like to avoid doing so if possible. I have included some thoughts I have been having recently. I tried moving try blocks around a little bit to no avail.
1) I suspect there is some way to do this with streams but I have no idea how to stream the result String (which is a json array string of json objects) one JSON object at a time.
2) Since result is a Java string, it is immutable. How can we consume this string and get it out of memory ASAP?
3) Would cleanedResult be better to instantiate a new object each time rather than just assign the same object something different each time?
4) At the end of the for loop shouldn't there only be roughly 2x memory used as before the loop as now json stringbuilder variable contains the same memory as the result string which should be the two largest variables in memory?
I have included the code below.
String result = getLargeJSONString(...); // function that gives me a large JSON string which is an array of JSON objects
StringBuilder json = new StringBuilder(); // to hold final JSON values to write to file
// try to parse said large JSON String
JSONArray results = new JSONArray();
try {
results = new JSONArray(result);
} catch (JSONException j) {
j.printStackTrace();
}
// do json sanitation on each object and then append to stringbuilder
// note the final result should be a string with a JSON object on each newline
JSONObject cleanedResult = new JSONObject();
for (int i = 0; i < results.length(); i++) {
try {
cleanedResult = JSONSanitizer.sanitize((JSONObject) results.get(i));
} catch (JSONException j) {
cleanedResult = new JSONObject();
}
json.append(cleanedResult.toString());
json.append('\n');
}
// write built string to file
try {
Files.write(Paths.get("../file.json"), json.toString().getBytes());
} catch (IOException i) {
System.out.println(i);
}

Of corse you should prefere streaming over contiguous memory allocation (String, StringBuilder, arrays and so) to process large amounts of data. So your best chance is to use a streaming JSON parser/serializer.
However, you should first try to optimize your code through several easy-gain fixes:
One: If you really need to store the result before wrinting it to a file, pre-size the StringBuilder to the estimated maximum final size it will have, so it won't need to be resized on every execution of append. For example, like this:
StringBuilder json = new StringBuilder(result.length());
You'd better even take in account an extra size for the newline characters. For example, oversizing 5%:
StringBuilder json = new StringBuilder((int)(1.05d*result.length()));
Two: If you just need to write the result out to a file, do not even store it into a StringBuilder:
String result = getLargeJSONString(...);
JSONArray results = new JSONArray(result);
try(Writer output=new OutputStreamWriter(new FileOutputStream(outputFile), "UTF8")) {
for (int i = 0; i < results.length(); i++) {
JSONObject cleanedResult = JSONSanitizer.sanitize((JSONObject) results.get(i));
output.write(cleanedResult.toString());
output.write('\n');
}
}

how to read JSON from HTTP GET request?

I have created a java server which gets HTTP GET request url as
/Star/getDetails?sentMsg=data.requestorName:ABC,data.companyName:EFG,portfolios:
[{name:,placeholder:Portfolio 1,positions:[{ticker:T1234,weight:29.85},
{ticker:T2345,weight:70.15}],active:false}],analyticsDate:20140630}
I have to parse sentMsg parameter such as I am able to read each variable individually. For eg, i should be able to read data.requestorName, companyName. I am not able to find a way to do it.
request.getParameter("sentMsg") always return String.
Tried parsing it through json-simple
JSONParser jp = new JSONParser();
try {
Object obj = jp.parse(sentMsg);
JSONArray ja = (JSONArray)obj;
} catch (ParseException e) {
e.printStackTrace();
}
But this gives parse exception. I have limitation to use json-simple jar only. Any suggestion on how to do it?

Get the paramter sentMsg from HttpRequest object store it into a string. Split from comma i.e. "," and the last second token would be the json string. You can now parse it using Json simple lib and extract values from it.

Provided you have valid JSON like:
private static String jsonString = "[{name : \"stackOverFlow\"}]";
Convert it to JSONArray like:
JSONArray jsonArray = new JSONArray(jsonString );
Then you can get value out of JSONArray by looping through it:
for (int i = 0; i < jsonArray.length(); i++) { //Iterating over mediaArray
JSONObject media = jsonArray.getJSONObject(i);
String nameFromJSON = media.getString("name");
System.out.println("Name = " + nameFromJSON);
}
Output will be:
//Name = stackOverFlow

Unable to split json response?

This is my method
public String buildJsonData(String username , String message)
{
JsonObject jsonObject = Json.createObjectBuilder().add("Username",username+":"+message).build();
StringWriter stringWriter = new StringWriter();
try(JsonWriter jsonWriter = Json.createWriter(stringWriter))
{
jsonWriter.write(jsonObject);
}
catch(Exception e)
{
System.out.print("buildJsonData ="+e);
}
return stringWriter.toString();
}
If i input username as john and message as hello.I get output as
{"Username":"john:hello"}
But I want output without braces and doublequotes I want my output as
John:hello
I tried to split it using array[0] but didn't get the output.Is it possible in json to get my desired output(without braces and quotes).

On the sending end, you would put the Username and Message entities into a JSONObject and send the resulting string over the network.
On the receiving end, you would unmarshal the JSON to extract the entities. You can then format them however you like.
Please read about JSON encoding here.
This is a simple example:
private String getResponse(){
JSONObject json = new JSONObject();
try {
json.put("Username", "John");
json.put("Message", "Hellow");
} catch (JSONException e) {
e.printStackTrace();
}
return json.toString();
}
private void receiver(){
try {
JSONObject response = new JSONObject(getResponse());
String username = response.getString("Username");
String message = response.getString("Message");
System.out.println(String.format("%s : %s", username,message));
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

Your structure is not really JSON.
A json structure would be like
{
Username : "John",
Message : "Hello"
}
Anf if your want to really use JSON, there is not way to remove braces and quotes. This IS Json.
If you want to output only the part you quoted, store the json value in a variable
String myoutput = stringWriter.toString();
And then remove the parts you don't want with replace() or a regexp

Braces are part of the JSON notation - they indicate an object. If you remove them, then it's not JSON any more. Same goes for double quotes.You are creating your JSON object as:
Json.createObjectBuilder().add("Username",username+":"+message)
This creates an object with property named Username and value john:hello. Again, this is the JSON notation. It's not intended to be read directly, but to facilitate data transfer between applications (on the same or different devices).
If all you want to create is john:message, then instead of creating a JSON object, you should simply do:
String result = username + ":" + message;
return result;

how do i add String array to JSON

I am currently writing some code in a servlet that gets data from the database and returns it to the client. What I am having problems with is inserting the array of dates I have collected and adding them to my JSON object that I will return to the client.
Here is the code I'm trying but it keeps giving errors
dates = ClassdatesDAO.getdate(dates);
ArrayList<String> ClassDates = new ArrayList<String>();
ClassDates = dates.getClassdates();
response.setContentType("application/json");
JSONObject Dates = new JSONObject();
Dates.put("dates", new JSONArray(ClassDates));
In my IDE I get this error over the ClassDates in the JSONArray
The constructor JSONArray(ArrayList) is undefined

You are passing ArrayList instance instead of an Array. So, convert the list into an array and then pass it as an argument like this
Dates.put("dates", new JSONArray(ClassDates.toArray(new String[ClassDates.size()])));
Note : json API has a method signature accepting java.util.Collection. So, you are using some other library or older version

JSONObject Dates = new JSONObject();
JSONArray datesJSONArray = new JSONArray();
for (String date : ClassDates)
datesJSONArray.put(date);
try {
Dates.put("dates", datesJSONArray);
} catch (JSONException e) {
e.printStackTrace();
}

Why am I getting a JsonSyntaxException here?

I am experiencing a JSON parsing error. My code is as follows:
try {
//Read the server response and attempt to parse it as JSON
Reader reader = new InputStreamReader(content);
GsonBuilder gsonBuilder = new GsonBuilder();
gsonBuilder.setDateFormat("M/d/yy hh:mm a");
Gson gson = gsonBuilder.create();
List<JsonObject> posts = (List) gson.fromJson(reader, JsonObject.class);
Log.e(TAG, "Results Size: " + posts.size());
// for(int i=0; i<posts.size(); i++){
// Log.e(TAG, "Checking: " + posts.get(i).title());
// }
content.close();
} catch (Exception ex) {
Log.e(TAG, "Failed to parse JSON due to: " + ex);
}
I get the following error from my posts.size() check:
Failed to parse JSON due to: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was BEGIN_ARRAY at line 1 column 2
For the JSON I am trying to read, if successful my posts.size() should be returning 5.
What am I doing wrong here?

I think your problem stems from the fact that you're trying to deserialize to a generic type (a List). The documentation states that in this case you need to pass in a Type rather than a Class to the fromJson() method. Try this:
Type type = new TypeToken<List<JsonObject>>(){}.getType();
List<JsonObject> posts = gson.fromJson(reader, type);

As it mentions in the exception line, your JSON starts with [ so it indicates a JSON Array rather than a JSON Object, but JSON have to start with a JSON Object. So wrap your JSON file with { } couple (add { to at beginning and } to end). It should resolve the issue.