{
"id": "12345678",
"data": {
"address": {
"street": "Address 1",
"locality": "test loc",
"region": "USA"
},
"country_of_residence": "USA",
"date_of_birth": {
"month": 2,
"year": 1988
},
"links": {
"self": "https://testurl"
},
"name": "John Doe",
"nationality": "XY",
"other": [
{
"key1": "value1",
"key2": "value2
},
{
"key1": "value1",
"key2": "value2"
}
],
"notified_on": "2016-04-06"
}
}
I am trying to read data from a GraphQL API that returns paginated JSON response. I need to write this into a CSV. I have been exploring Spring Batch for implementation where I would read JSON data in the ItemReader and flatten each JSON entry (in ItemProcessor) and then write this flattened data into a CSV (in ItemWriter). While I could use something like Jackson for flattening the JSON, I am concerned about possible performance implications if the JSON data is heavily nested.
expected output:
id, data.address.street, data.address.locality, data.address.region, data.country_of_residence, data.date_of_birth.month, data.date_of_birth.year, data.links.self, data.name, data.nationality, data.other (using jsonPath), data.notified_on
I need to do process more than a million records. While I believe flattening the CSV would be a linear operation O(n), I was still wondering if there could be other caveats if the JSON structure gets severely nested.
Related
Im trying to get JSONPath expression to filter my JSON and get whole sport object using value of child array.
I have following JSON:
[{
"name": "Soccer",
"regions": [{
"name": "Australia",
"leagues": [{
"name": "Australia league",
"inplay": 5,
}
]
}
]
}, {
"name": "Tennis",
"regions": [{
"name": "Germany",
"leagues": [{
"name": "Germany league",
"inplay": 0,
}
]
}
]
}
]
I need to get whole sport object where "inplay == 0" using JsonPath expression.
Result should look like that:
{
"name": "Tennis",
"regions": [{
"name": "Germany",
"leagues": [{
"name": "Germany league",
"inplay": 0,
}
]
}
]
}
Regions and Leagues count can be > 1
Therefore $[?(#.regions[0].leagues[0].inplay == 0)] is not suitable
Tried $[?(#.regions[*].leagues[*].inplay == 0)] but it doesnt work
This works for me
$[?(#.regions[0].leagues[0].inplay == 0)]
Since this is not directly supported (as of now) in JayWay JSONPath we leverage contains as a workaround:
$[?(#.regions..inplay contains '0')]
Note: It may look like contains would work similar to a 'like' operator or instr function but this is not the case here. If the inplay value contains a 0, e.g. 10 it would not pull the record (according to my tests;)
I want to get the value at the field first inside name.
How i can access in this field using HashMap in java
{ "payload":{
"name": {
"first": "jean",
"last": "bob,
},
"address": {
"code": "75",
"city": "paris",
"country": "France"
},
}}
Use one of the available Java libraries for handling JSON. E.g. Gson from Guava API. They are pretty straing fw.
i am trying to read sample Json file through Apache Spark, during this process i observed one thing is that you need to keep entire json object into single line. If i keep entire json object into single line,code is working well otherwise getting exception.
This is my json data:
[
{
"id": 2,
"name": "An ice sculpture",
"price": 12.50,
"tags": ["cold", "ice"],
"dimensions": {
"length": 7.0,
"width": 12.0,
"height": 9.5
},
"warehouseLocation": {
"latitude": -78.75,
"longitude": 20.4
}
},
{
"id": 3,
"name": "A blue mouse",
"price": 25.50,
"dimensions": {
"length": 3.1,
"width": 1.0,
"height": 1.0
},
"warehouseLocation": {
"latitude": 54.4,
"longitude": -32.7
}
}
]
This is my code:
SparkSession session = new SparkSession.Builder().appName("JsonRead").master("local").getOrCreate();
Dataset<Row> json = session.read().json("/Users/mac/Desktop/a.json");
json.select("tags").show();
In case of small datasets its okay, is any other way to process large json datasets?
see the document:
http://spark.apache.org/docs/2.0.1/sql-programming-guide.html#json-datasets
JSON Datasets
Note that the file that is offered as a json file is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a consequence, a regular multi-line JSON file will most often fail.
``I am having an Json Array as response like:
[
{
"status": "Active",
"entityName": "fghgfhfghfgh",
"entityCode": 14,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "64eab9299eed9455b3683da074cf175c",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7"
}
,
{
"status": "Active",
"entityName": "ghghhguyutgh6re58rrt",
"entityCode": 13,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "912eff0613fa140c100af435c033e195",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7"
}
]
I want to split this json into two like
{
"status": "Active",
"entityName": "fghgfhfghfgh",
"entityCode": 14,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "64eab9299eed9455b3683da074cf175c",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7" }
and the other one.I am using GSON and simplejson,when I try to remove the delimiters([ and ])the json comes as malformed one.Is there any better otpion to split the json array to two or more json strings as per the json response coming.
Is there any reason you can't parse the entire array, and then iterate over/process each element individually?
Just removing the brackets does make the JSON invalid, and splitting at the comma is going to be unreliable. The parser exists to figure out where to split the array and turn each element into an object for you.
Assuming you have some data structure defined to hold a single element once you've broken the array down, you should be able to parse the array into a list of those and step through them (or pick one out) as needed.
After that point, you can do whatever you want with the data (including formatting it back into JSON). I would definitely recommend using a proper parser to break the array down, though; it will be a lot simpler and more reliable, and should work unless you have serious performance concerns.
I've a csv file something similar to this
"name.firstName","name.givenName","name.DisplayName","phone.type","phone.value"
"john","maverick","John Maverick","mobile","123-123-123"
"jim","lasher","Jim Lasher","mobile","123-123-123"
I want to convert the 2nd and 3rd row into JSON objects.Using the first row as Header. So the result will be
[
{
"name": {
"firstName": "john",
"givenName": "maverick",
"DisplayName": "John Maverick"
},
"phone": {
"type": "mobile",
"value": "123-123-123"
}
},
{
"name": {
"firstName": "john",
"givenName": "maverick",
"DisplayName": "John Maverick"
},
"phone": {
"type": "mobile",
"value": "123-123-123"
}
]
Any idea how to achieve this?
Here is a Java library that may help you. http://www.jonathanhfisher.co.uk/playpen/csv2json/index.htm
Here is a JavaScript library that may or may not be useful to you. http://www.cparker15.com/code/utilities/csv-to-json/
And finally, here is a past answer that may be useful. I like the OpenCSV solution. However, instead of JAXB, you could use Jackson. Converting an CSV file to a JSON object in Java