We Keep Coding

Scala - Ids lists of objects with duplicated values from spark dataset

I need to create an IDs lists for all objects that have identical (same value and quantity) parameters. I am looking for a solution that will be more efficient than two nested loops and an if.
Object structure in the dataset:
case class MergedProduct(id: String,
products: List[Product])
case class Product(productUrl: String, productId: String)
Example of data in dataset:
[ {
"id": "ID1",
"products": [
{
"product": {
"productUrl": "SOMEURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID2",
"products": [
{
"product": {
"productUrl": "SOMEURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID3",
"products": [
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID4",
"products": [
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
}
],
},
{
"id": "ID5",
"products": [
{
"product": {
"productUrl": "NOTDUPLICATEDURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
}
],
}
]
In this example, we have 4 objects that are duplicated, so I would like to get their ID in the corresponding lists.
Example output is List[List[String]]:
List(List("ID1", "ID2"), List("ID3","ID4"))
I am looking for something efficient and readable - the dataset we are talking about has nearly 700 million objects.
As I can remove the listed duplicates from the dataset (it does not affect the database) because the goal is one - logging them exists, so I was thinking about the solution of taking MergedProduct one by one, searching for other MergedProduct with identical Products, getting their ID, logging in they exist and then remove the mentioned MergedProduct ID from the dataset and move on to the next one until I check the whole dataset but in this case I would have to collect it first as a list of MergedProducts and then do all operations - seems like going around

After trying some options and looking for neat solutions- I think this is kinda ok:
private def getDuplicates(mergedProducts: List[MergedProduct]): List[List[String]] = {
val duplicates = mergedProducts.groupBy(_.products.sortBy(_.product.productId)).filter(_._2.size > 1).values.toList
duplicates.map(duplicates => duplicates.map(_.id))
}

Flattening a heavily nested JSON in Java - Time Complexity

{
"id": "12345678",
"data": {
"address": {
"street": "Address 1",
"locality": "test loc",
"region": "USA"
},
"country_of_residence": "USA",
"date_of_birth": {
"month": 2,
"year": 1988
},
"links": {
"self": "https://testurl"
},
"name": "John Doe",
"nationality": "XY",
"other": [
{
"key1": "value1",
"key2": "value2
},
{
"key1": "value1",
"key2": "value2"
}
],
"notified_on": "2016-04-06"
}
}
I am trying to read data from a GraphQL API that returns paginated JSON response. I need to write this into a CSV. I have been exploring Spring Batch for implementation where I would read JSON data in the ItemReader and flatten each JSON entry (in ItemProcessor) and then write this flattened data into a CSV (in ItemWriter). While I could use something like Jackson for flattening the JSON, I am concerned about possible performance implications if the JSON data is heavily nested.
expected output:
id, data.address.street, data.address.locality, data.address.region, data.country_of_residence, data.date_of_birth.month, data.date_of_birth.year, data.links.self, data.name, data.nationality, data.other (using jsonPath), data.notified_on
I need to do process more than a million records. While I believe flattening the CSV would be a linear operation O(n), I was still wondering if there could be other caveats if the JSON structure gets severely nested.

JSONPath: Get root array object using filter of child value

Im trying to get JSONPath expression to filter my JSON and get whole sport object using value of child array.
I have following JSON:
[{
"name": "Soccer",
"regions": [{
"name": "Australia",
"leagues": [{
"name": "Australia league",
"inplay": 5,
}
]
}
]
}, {
"name": "Tennis",
"regions": [{
"name": "Germany",
"leagues": [{
"name": "Germany league",
"inplay": 0,
}
]
}
]
}
]
I need to get whole sport object where "inplay == 0" using JsonPath expression.
Result should look like that:
{
"name": "Tennis",
"regions": [{
"name": "Germany",
"leagues": [{
"name": "Germany league",
"inplay": 0,
}
]
}
]
}
Regions and Leagues count can be > 1
Therefore $[?(#.regions[0].leagues[0].inplay == 0)] is not suitable
Tried $[?(#.regions[*].leagues[*].inplay == 0)] but it doesnt work

This works for me
$[?(#.regions[0].leagues[0].inplay == 0)]

Since this is not directly supported (as of now) in JayWay JSONPath we leverage contains as a workaround:
$[?(#.regions..inplay contains '0')]
Note: It may look like contains would work similar to a 'like' operator or instr function but this is not the case here. If the inplay value contains a 0, e.g. 10 it would not pull the record (according to my tests;)

Parse json response of rest API and delete certain jsonObjects - JAVA

I have a json file as below which I am getting as a response from rest API:
{
"label": " MARA LEYZIN",
"ClassCode": "PROFESSIONAL",
"actvFlg": "A",
"name": "MARA LEYZIN",
"Typ": {
"label": "C_TYP_LU",
"TypCode": "PROFESSIONAL "
},
"Address": {
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item": [
{
"label": "Address",
"addrTypFk": {
"label": "C_ADDRESS_TYPE_LU",
"addrTypCd": "INDUSTRY",
"addrTypDesc": "Industry"
}
}
]
}
I am trying to parse this in Java and to remove some unwanted json objects. Like I want the following string to be replaced by blank:
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item":
To achieve this I am trying the following approach:
String jsonStr = new String(Files.readAllBytes(Paths.get(inputFile)));
System.out.println(jsonStr);
jsonStr.replaceAll("link", "");
But it is not replacing the required string with blanks. Please help me in this.

string object is immutable , so basically if do you want to replace something
System.out.println(jsonStr.replaceAll("link", "")); this will print the replaced string but it will not affect the original string, however if you do this
jsonStr=jsonStr.replaceAll("link", "");
System.out.println(jsonStr); this will print the replaced string

First of all:
Your JSON is not validate. You're missing a closing curly bracket at the end of it.
{
"label": " MARA LEYZIN",
"ClassCode": "PROFESSIONAL",
"actvFlg": "A",
"name": "MARA LEYZIN",
"Typ": {
"label": "C_TYP_LU",
"TypCode": "PROFESSIONAL "
},
"Address": {
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item": [{
"label": "Address",
"addrTypFk": {
"label": "C_ADDRESS_TYPE_LU",
"addrTypCd": "INDUSTRY",
"addrTypDesc": "Industry"
}
}]
}
}
Second of all you should just change order of your commands to this:
jsonStr.replaceAll("link", "");
System.out.println(jsonStr);
Important addition:
And I would suggest you to use org.json library or even better JACKSON to parse JSON files.
Here's tutorial how to use jackson and it's my warmest suggestion.
You will save a lot of time and you can do whatever you like.

How can I cast json recursively using gson.fromjson

Given I have the following json structure:
{
"Role": {
"id": "5",
"name": "bb1",
"description": "desc1",
"PermissionDeck": [
{
"id": "7",
"Permission": [
{
"id": "398"
},
{
"id": "399"
},
{
"id": "400"
},
{
"id": "401"
},
{
"id": "402"
}
],
"Limit": [
{
"id": "4"
},
{
"id": "5"
}
]
}
]
}
}
If I want to cast this into a LinkedTreeMap result so that its content could be a retrieved by:
result.get("Role") returns Map
and
result.get("Role").get("PermissionDeck").size() == 5
and
result.get("Role").get("PermissionDeck").get(0).get("id") == 398
basically makes gson.fromjson recursively go into the structure, and fold any nested structure into LinkedTreeMap until it gets to the most inner layer, which then gets into LinkedTreeMap
Is this possible without writing custom recursive methods?

You can't. The closest you'll get is with JsonObject and using the appropriate getter for each nested member. In other words, your code needs to be explicit about what it expects in the JSON.
For example
JsonObject result = new Gson().fromJson(theJson), JsonObject.class);
System.out.println(result.getAsJsonObject("Role").getAsJsonArray("PermissionDeck").size());
will print 1 since you only have one element in the JSON array named PermissionDeck in the JSON object named Role in the root JSON object.