I have an approximately 1MB JSON file stored in my assets folder that I need to load in my app every time it runs. I find that the built-in JSON parser (org.json) parses the file very slowly, however once it's parsed, I can access and manipulate the data very quickly. I've counted out as many as 7 or 8 seconds from the moment I click on the app to the moment the Activity1 is brought up, but just a few milliseconds to go from Activity1 to Activity2 which depends on data processed from the data loaded in Activity1.
I'm reading the file into memory and parsing it using:
String jsonString = readFileToMemory(myFilename)
JSONArray array = new JSONArray(jsonString);
where readFileToMemory(String) looks like this:
private String readFileToMemory(String filename) {
StringBuilder data = new StringBuilder();
BufferedReader reader = null;
try {
InputStream stream = myContext.getAssets().open(filename);
reader = new BufferedReader(new InputStreamReader(stream, "UTF-8"));
int result = 0;
do
{
char[] chunk = new char[512];
result = reader.read(chunk);
data.append(chunk);
}
while(result != -1);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return data.toString();
}
Does anyone have any suggestions to how I can speed up the initial loading and parsing of the data? Should I perhaps mask the whole process behind a loading screen?
JSONObject -- the one from json.org, is the simplest API to use to parse JSON. However it comes with a cost -- performace. I have done extensive experiments with JSONObject, Gson and Jackson. It seems no matter what you do, JSONObject (and hence JSONArray) will be the slowest. Please switch to Jackson or Gson.
Here is the relative performance of the two
(fastest) Jackson > Gson >> JSONObject (slowest)
Refer:
- Jackson
- Gson
You should make an SQLite table to store the data and move it from JSON to SQL the first time the app runs. As an added benefit, this makes the data easier to search through and makes it possible for you to modify the data from within the app.
Related
I'm trying to write some unit tests for Kafka Streams and have a number of quite complex schemas that I need to incorporate into my tests.
Instead of just creating objects from scratch each time, I would ideally like to instantiate using some real data and perform tests on that. We use Confluent with records in Avro format, and can extract both schema and a text JSON-like representation from the Control Center application. The JSON is valid JSON, but it's not really in the form that you'd write it in if you were just writing JSON representations of the data, so I assume it's some representation of the underlying AVRO in text form.
I've already used the schema to create a Java SpecificRecord class (price_assessment) and would like to use the JSON string copied from the Control Center message to populate a new instance of that class to feed into to my unit test InputTopic.
The code I've tried so far is
var testAvroString = "{JSON copied from Control Center topic}";
Schema schema = price_assessment.getClassSchema();
DecoderFactory decoderFactory = new DecoderFactory();
Decoder decoder = null;
try {
DatumReader<price_assessment> reader = new SpecificDatumReader<price_assessment>();
decoder = decoderFactory.get().jsonDecoder(schema, testAvroString);
return reader.read(null, decoder);
} catch (Exception e)
{
return null;
}
which is adapted from another SO answer that was using GenericRecords. When I try running this though I get the exception Cannot invoke "org.apache.avro.Schema.equals(Object)" because "writer" is null on the reader.read(...) step.
I'm not massively familiar with streams testing or Java and I'm not sure what exactly I've done wrong. Written in Java 17, streams 3.1.0, though flexible with version
The solution that I've managed to come up with is the following, which seems to work:
private static <T> T avroStringToInstance(Schema classSchema, String testAvroString) {
DecoderFactory decoderFactory = new DecoderFactory();
GenericRecord genericRecord = null;
try {
Decoder decoder = decoderFactory.jsonDecoder(classSchema, testAvroString);
DatumReader<GenericData.Record> reader =
new GenericDatumReader<>(classSchema);
genericRecord = reader.read(null, decoder);
} catch (Exception e)
{
return null;
}
var specific = (T) SpecificData.get().deepCopy(genericRecord.getSchema(), genericRecord);
return specific;
}
Im trying to understand the procedure to do what the title says.
Im doing this in java with Gson dependency.
I am getting information from another service I use, in JSON format. So I want to get that info, put some additional info in there (like date/time) and use it afterwards for searching purposes.
The procedure is :
Get the JSON info (lets say "id") and add it to the JSON file you have
Add more info to that JSON file (lets say "Date and time of upload")
Finally, save that updated JSON file
So I read the file:
JsonReader reader = new JsonReader(new FileReader(filename));
Do I have now to convert it to string, and then update the string, so I can finally write it back to json?
If it doesn't exist, I create an empty file and then, can I update it with Json/Gson data? or do I have to create a Json File?
try {
File jsonFile = new File("C:\\uploads\\datasets");
if (jsonFile.createNewFile()){
System.out.println("File is created!");
}else{
System.out.println("File already exists.");
}
} catch (IOException e) {
e.printStackTrace();
}
Excuse any newbie/stupid mistakes I've probably made, I'm trying to understand JSON. Actually, the philosophy behind it.
JSON stands for JavaScript Object Notation and it's nothing more than a way to format data.
Taken from here:
JSON is built on two structures:
A collection of name/value pairs. In various languages, this is
realized as an object, record, struct, dictionary, hash table, keyed
list, or associative array.
An ordered list of values. In most
languages, this is realized as an array, vector, list, or sequence.
To address your questions:
Get the JSON info (lets say "id") and add it to the JSON file you have
JsonReader reader = new JsonReader(new FileReader(inputFilename));
reader.beginArray();
reader.beginObject();
long id = -1;
while (reader.hasNext()) {
String value = reader.nextName();
if (value.equals("id")) {
id = reader.nextLong();
} else {
reader.skipValue();
}
reader.endObject();
reader.endArray();
Add more info to that JSON file (lets say "Date and time of upload")
This will get the format in YYYY.MM.DD-HH.MM.SS
String timeStamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(Calendar.getInstance().getTime());
Finally, save that updated JSON file
Create a JsonWriter.
JsonWriter writer = new JsonWriter(new FileWriter(outputFilename));
writer.beginArray();
writer.beginObject();
writer.name("id").value(id);
writer.name("timestamp").value(timestamp);
writer.endObject();
writer.endArray();
You can read more about JsonReader and JsonWriter here and here.
I want to create an android application that uses an API to show some information on a listview. The problem I am facing is that the json file that I want to download is too big for String the object. My instructor told me that Android Studio somehow limited the size of the String. I get the error "constant string too long".
Is it possible for me to download that information as a json file (it is .geojson actually) and store it somewhere in the disk (actually I can download the file), and then parse it without using any String object by reading it from the file itself. OR Can you suggest another way of doing it since I am new in programming?
The solution I found is that using simple JSON simple library in my project.
You can get it from here.
After I downloaded the JSON file to the storage, I parsed it by using the code below:
JSONParser parser = new JSONParser();
Object obj = null;
try {
obj = parser.parse(new FileReader(directory of the folder + "/fileName.extension"));
} catch (IOException e) {
e.printStackTrace();
} catch (org.json.simple.parser.ParseException e) {
e.printStackTrace();
}
JSONObject jsonObject = (JSONObject) obj;
Now you can use jsonObject as how you want.
I'm working on a java project to optimize existing code. Currently i'm using BufferedReader/FileInputStream to read content of an XML file as String in Java.
But my question is , is there any faster way to read XML content.Are SAX/DOM faster than BufferedReader/FileInputStream?
Need help regarding the above issue.
Thanks in advance.
I think that your code shown in other question is faster than DOM-like parsers which would definitely require more memory and likely some computation in order to reconstruct the document in full. You may want to profile the code though.
I also think that your code can be prettified a bit for streaming processing if you would use javax XMLStreamReader, which I found quite helpful for many tasks. That class is "... is designed to be the lowest level and most efficient way to read XML data", according to Oracle.
Here is the excerpt from my code where I parse StackOverflow users XML file distributed as a public data dump:
// the input file location
private static final String fileLocation = "/media/My Book/Stack/users.xml";
// the target elements
private static final String USERS_ELEMENT = "users";
private static final String ROW_ELEMENT = "row";
// get the XML file handler
//
FileInputStream fileInputStream = new FileInputStream(fileLocation);
XMLStreamReader xmlStreamReader = XMLInputFactory.newInstance().createXMLStreamReader(
fileInputStream);
// reading the data
//
while (xmlStreamReader.hasNext()) {
int eventCode = xmlStreamReader.next();
// this triggers _users records_ logic
//
if ((XMLStreamConstants.START_ELEMENT == eventCode)
&& xmlStreamReader.getLocalName().equalsIgnoreCase(USERS_ELEMENT)) {
// read and parse the user data rows
//
while (xmlStreamReader.hasNext()) {
eventCode = xmlStreamReader.next();
// this breaks _users record_ reading logic
//
if ((XMLStreamConstants.END_ELEMENT == eventCode)
&& xmlStreamReader.getLocalName().equalsIgnoreCase(USERS_ELEMENT)) {
break;
}
else {
if ((XMLStreamConstants.START_ELEMENT == eventCode)
&& xmlStreamReader.getLocalName().equalsIgnoreCase(ROW_ELEMENT)) {
// extract the user data
//
User user = new User();
int attributesCount = xmlStreamReader.getAttributeCount();
for (int i = 0; i < attributesCount; i++) {
user.setAttribute(xmlStreamReader.getAttributeLocalName(i),
xmlStreamReader.getAttributeValue(i));
}
// all other user record-related logic
//
}
}
}
}
}
That users file format is quite simple and similar to your Bank.xml file:
<users>
<row Id="1567200" Reputation="1" CreationDate="2012-07-31T23:57:57.770" DisplayName="XXX" EmailHash="XXX" LastAccessDate="2012-08-01T00:55:12.953" Views="0" UpVotes="0" DownVotes="0" />
...
</users>
There are different parser options available.
Consider using a streaming parser, because the DOM may become quite big. I.e. either a push or a pull parser.
It's not as if XML parsers are necessarily slow. Consider your web browser. It does XML parsing all the time, and tries really hard to be robust to syntax errors. Usually, memory is the bigger issue.
The JSON example file consists of:
{
"1st_key": "value1",
"2nd_key": "value2",
"object_keys": {
"obj_1st": "value1",
"obj_2nd": "value2",
"obj_3rd": "value3",
}
}
I read the JSON file into a String with this StringBuilder method, in order to add the newlines into the string itself. So the String looks exactly like the JSON file above.
public String getJsonContent(String fileName) {
StringBuilder result = new StringBuilder("");
File file = new File(fileName);
try (Scanner scanner = new Scanner(file)) {
while (scanner.hasNextLine()) {
String line = scanner.nextLine();
result.append(line).append("\n");
}
scanner.close();
} catch (IOException e) {
e.printStackTrace();
}
return result.toString();
}
Then I translate the JSON file into an Object using MongoDB API (with DBObject, BasicDBObject and util.JSON) and I call out the Object section I need to change, which is 'object_keys':
File jsonFile = new File(C:\\example.json);
String jsonString = getJsonContent(jsonFile.getAbsolutePath());
DBObject jsonObject = (DBObject)JSON.parse(jsonString);
BasicDBObject objectKeys = (BasicDBObject) jsonObject.get("object_keys");
Now I can write new values into the Object using the PUT method like this:
objectKeys.put("obj_1st","NEW_VALUE1");
objectKeys.put("obj_2nd","NEW_VALUE2");
objectKeys.put("obj_3rd","NEW_VALUE3");
! This following part not needed, check out my answer below.
After I have changed the object, I need to write it back into the json file, so I need to translate the Object into a String. There are two methods to do this, either one works.
String newJSON = jsonObject.toString();
or
String newJSON = JSON.serialize(jsonObject);
Then I write the content back into the file using PrintWriter
PrintWriter writer = new PrintWriter(C:\\example.json)
writer.print(newJSON);
writer.close();
The problem I am facing now is that the String that is written is in a single line with no formatting whatosever. Somewhere along the way it lost all the newlines. So it basically looks like this:
{"1st_key": "value1","2nd_key": "value2","object_keys": { "obj_1st": "NEW_VALUE1","obj_2nd": "NEW_VALUE2","obj_3rd": "NEW_VALUE3", }}
I need to write the JSON file back in the same format as shown in the beginning, keeping all the tabulation, spaces etc.
Is this possible somehow ?
When you want something formatted the way you said it is addressed as writing to a file in a pretty/beautiful way. For example: Output beautiful json. A quick search on google found what i believe to solve your problem.
Solution
You're going to have to use a json parser of some sort. I personally prefer org.json and would recommend it if you are manipulating the json data, but you may also like json-io which is really good for json serialization with no external dependencies.
With json-io, it's as simple as
String formattedJson = JsonWriter.formatJson(jsonObject.toString())
With org.json, you simply pass an int to the toString method.
Thanks Saraiva, I found a surprisingly simple solution by Googling around with the words 'pretty printing JSON' and used the Google GSON library. I downloaded the .jar and added it to my project in Eclipse.
These are the new imports I needed:
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
Since I already had the JSON Object (jsonObject) readily available from my previous code, I only needed to add two new lines:
Gson gson = new GsonBuilder().setPrettyPrinting().create();
String newJSON = gson.toJson(jsonObject);
Now when I use writer.print(newJSON); it will write the JSON in the right format, beautifully formatted and indented.