JSON parsing vs Regex based String parsing - java

I need to process a big JSON payload(~1MB) coming from an API, a portion of the JSON is something like this:
{
"id": "013dd2a7-fec4-4cc5-b819-f3cf16a1f820",
//more attributes
"entry_mode": "LDE",
"periods": [
{
"type": "quarter",
"id": "fe96dc03-660c-423c-84cc-e6ae535edd2d",
"number": 1,
"sequence": 1,
"scoring": {
//more attribtues
},
"events": [
{
"id": "e4426708-fadc-4cae-9adc-b7f170f5d607",
"clock": "12:00",
"updated": "2013-12-22T03:41:40+00:00",
"description": "J.J. Hickson vs. DeAndre Jordan (Blake Griffin gains possession)",
"event_type": "opentip",
"attribution": {
"name": "Clippers",
"market": "Los Angeles",
"id": "583ecdfb-fb46-11e1-82cb-f4ce4684ea4c",
"team_basket": "left"
},
"location": {
"coord_x": 572,
"coord_y": 296
},
"possession": {
"name": "Clippers",
"market": "Los Angeles",
"id": "583ecdfb-fb46-11e1-82cb-f4ce4684ea4c"
}
},
//more events
]
}
]
}
This is a nearly-realtime API that I need to process only the events, identify a set of event UUIDs, look for duplicates in the database and save new events.
I could use a JSONObject/JSONArray or use regex with string parsing to and fetch the events portion. Processing time is critical since this should be nearly-realtime and memory efficiency is important since there can be multiple payloads coming in at once.
Which one is more efficient for this use case?

Use a proper streaming JSON parser. You know what you want to pull out of the stream, you know when you can quit parsing it, so read the stream in small, manageable chunks, and quit as soon as you know you are done.
Circa 2017, I'm not aware of any browser/native JSON streaming APIs, so you'll need to find a Javascript-based streaming library. Fortunately, streaming is not a new concept, so there are a number of options already in existence:
http://oboejs.com/
https://github.com/dominictarr/JSONStream
https://github.com/creationix/jsonparse
https://github.com/dscape/clarinet
http://danieltao.com/lazy.js/demos/json/

Related

JSON Schema - Enum of Objects

I'm new to JSON schema, so bear with me. My goal is to have a JSON property that is an object. It's keys relate to each other, meaning multiple keys always have the same values together. This will probably help make it clear, it's my attempt to do this with an enum:
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"title": "Part",
"type": "object",
"properties": {
"relationship": {
"type": "object",
"enum": [
{
"code": "1",
"value": "MEMBER"
},
{
"code": "2",
"value": "SPOUSE"
},
{
"code": "3",
"value": "CHILD"
},
{
"code": "4",
"value": "STUDENT"
},
{
"code": "5",
"value": "DISABILITY_DEPENDENT"
},
{
"code": "6",
"value": "ADULT_DEPENDENT"
},
{
"code": "8",
"value": "DOMESTIC_PARTNER"
}
]
}
}
}
So using an enum like this works, even though I can't find it anywhere in the JSON Schema spec. However, the error message sucks. Normally I get the most extremely detailed error messages from schema validation, however in this case I do not.
$.part.relationship: does not have a value in the enumeration [, , , , , , ]
I'm not sure what I'm doing wrong. I'm using a Java parser for JSON Schema:
<dependency>
<groupId>com.networknt</groupId>
<artifactId>json-schema-validator</artifactId>
<version>1.0.53</version>
</dependency>
Not sure if the error message is the fault of the parser or something I'm doing bad with the schema. Help would be appreciated.
It was news to me, but according to the spec it does seem that objects are valid enum values. That said, your usage is quite unusual. I've not seen it used before.
the six primitive types ("null", "boolean", "object", "array", "number", or "string")
...
6.1.2. enum
...
Elements in the array might be of any type, including null.
Your problem is fundamentally that the library that you're using doesn't know how to convert those objects to printable strings. Even if it did give it a reasonable go, you might end up with
does not have a value in the enumeration [{"code": "1", "value":"MEMBER"}, {"code": "2" ...
which might be okay, but it's hardly amazing. If the code and value were both valid but didn't match, you might have to look quite closely at the list before you ever saw the problem.
JSON Schema in general is not very good at enforcing constraints between what it considers to be 2 unrelated fields. That's beyond the scope of it what it aims to do. It's trying to validate the structure. Dependencies between fields are business constraints, not structural ones.
I think the best thing you could do to achieve readable error messages would be to have 2 sub-properties, each with an enumeration containing 8 values; one for the codes, one for the values.
Then you'll get
$.part.relationship.code does not have a value in the enumeration [1,2,3,4 ...
or
$.part.relationship.value does not have a value in the enumeration ["MEMBER", "SPOUSE", ...
You can do some additional business validation on top of the schema validation if enforcing that constraint is important to you. Then generate your own error such as
code "1" does not match value "SPOUSE"
If code and value always have the same values relative to each other, why encode both in the JSON? Just encode a single value in the JSON and infer the other in the application.
This will be much easier to validate.

fhir.executeBundle replacing resource id...How to prevent this?

I am using this Java code to upload a resource to a FHIRstore.
The resource is as follows
{
"resourceType": "Bundle",
"id": "bundle-transaction",
"meta": {
"lastUpdated": "2018-03-11T11:22:16Z"
},
"type": "transaction",
"entry": [
{
"resource": {
"resourceType": "Patient", "id" : 123456,
"name": [
{
"family": "Smith",
"given": [
"Darcy"
]
}
],
"gender": "female",
"address": [
{
"line": [
"123 Main St."
],
"city": "Anycity",
"state": "CA",
"postalCode": "12345"
}
]
},
"request": {
"method": "POST",
"url": "Patient"
}
}
]
}
But the id i am using(123456) is getting replaced by a hexadecimal number.
This does not happen while using fhirstores.import method
Is there any way to stop executeBundle method from replacing my id...as i want to use custom id in my resource?
Any help would be appreciated.
Thank you
When you're performing a transaction, the effect is going to be the same as if you were POSTing the resources individually. On a POST, the server determines the resource id. On a regular POST, the id is just ignored or raises an error. Within a transaction, the id is used to manage resolution of references across the transaction, but the server still chooses what the id will be of the persisted resources (and updates all references accordingly). If you want to control the resource id values within a transaction, use PUT rather than POST. (Note that not all servers will allow an 'upsert' - i.e. a PUT that performs a create at a specific resource location.) For details, see http://hl7.org/fhir/http.html#upsert.

how to get json response without references with "ids"

so i have my project there is this part where there are 2 one to many relation ship to the same entity
what happens is that the response on the get request on postman come like this :
the one to many relationship is writen the same for both elements
{
"elemnt1withonetomany": {
"id": 2,
"name": "something",
"last_name": "something",
"email": "something"
},
"elemnt2withonetomany": {
"#id": 4,
"id": 4,
"code": "details",
"email": "details",
"name": "details",
"lastname": "details"
},
{
"elemnt1withonetomany": {
"id": 2,
"name": "something",
"last_name": "something",
"email": "something"
},
"element2withonetomany": 4,
}
so is there any way to make the get request gives the same form of information with elemnt2withonetomany
i kinda found where it came from but then it's gonna need a lot of JsonBackReference and similar annotations
and yep it was from the #JsonIdentityInfo on top of the entities
kinda took me a while to find the source so i'm just gonna post what i found if someone needed it
so my new question is there a way to by pass it without deleting this one

Format json file proberly and parse it to java

For an Android App I'm working on I need to parse json files with various informations to my App, for now the files look like this:
{
"R6":{
"Typ": "KnotenRaum",
"ID": 1,
"X-Koor": 3,
"Y-Koor": 11,
"Ebene": 0,
"Kantenliste": [ "m7" ],
"GruppenID": 1,
"Raum": {
"Nummer": "A.00.01",
"Typ": null,
"Person": null
}
},
"H107":{
"Typ": "KnotenTreppe",
"ID": 115,
"X-Koor": 7,
"Y-Koor": 3,
"Ebene": 1,
"Kantenliste": [ "h108","b1002" ],
"GruppenID": 1,
"Raum": {}
}
}
As this is my (or to be more specific our) first time using json I'm not sure if this is a "good" way of formating the file. The problem is, that I don't only have 5 or 10 objects but i guess hundreds, so i think using specifiers like "H107" is the wrong way and I should just put it all into one large array?
And second: if I put it in such an array, what is the best way to parse it to java?
I don't need class objects (at least not necessarily) as all those data will be stored in a sqlite database when the app runs for the first time.
I found some examples but mostly those only consist of one object and are a lot simpler than my resulting json file I guess.
H107 and R6 are object names?
I think this can be a array with multiple objects where "R6" can be a JSON key on object.
for example:
[
{
"name": "R6",
"Typ": "KnotenRaum",
"ID": 1,
"X-Koor": 3,
"Y-Koor": 11,
"Ebene": 0,
"Kantenliste": [
"m7"
],
"GruppenID": 1,
"Raum": {
"Nummer": "A.00.01",
"Typ": null,
"Person": null
}
},
{
"name": "H107",
"Typ": "KnotenTreppe",
"ID": 115,
"X-Koor": 7,
"Y-Koor": 3,
"Ebene": 1,
"Kantenliste": [
"h108",
"b1002"
],
"GruppenID": 1,
"Raum": {}
}
]
About objects I recommend you use an object to represent this JSON, is more readable and more easy to save on SQLite, you can use a framework like Jackson or GSON to parse this JSON automatically into objects.
I think it would be wise to store the data into Objects.
It's easier to insert objects into SQLdatabase as well as making a JSON objects array.
Andhere is a nice tutorial on the subject:
Android JSON tutorial

Update the json file in android file

I am saving a json response inside my app using sharedPreference(jsonObject.toString()). It contains JSONArray, when the user updates some value of any one element, I wish to save the updated changes on the sharedPreference. Please help me for this task.
Example:-
{
"locations": {
"record": [
{
"id": 8817,
"loc": "NEW YORK CITY"//update this as California and save the response
},
{
"id": 2873,
"loc": "UNITED STATES"
},
{
"id": 1501
"loc": "NEW YORK STATE"
}
]
}
}
It seems like you're trying to override the purpose of SharedPreferenecs,
It's purpose is to save primitive values such as strings integers or booleans, for simple use of single values, I wouldn't treat a Json Array as a single primitive value.
If I were you I would go with QuokMoon's offer with the Local Sqlite Database, this will allow you simple access for CRUD operations, the setup time is a bit longer, but the benefits you'll find are far beyond SharedPreferences.

Categories