We Keep Coding

JsonPath: How to get the whole tree except one node

everyone.
My question is quite simple, I think.
My use case:
Jenkins receives a huge JSON payload from a Gitlab WebHook (more than 2500 lines). I want to get rid of a specific node with more than 2000 lines. The JSON I get is so big that Jenkins is unable to parse it correctly so I want to remove a node I don't need.
Assume the sample tree in the documentation page:
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
},
{
"category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
},
{
"category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
How can I get the whole tree except one? For example, if I want to get everything but the book node ...
{
"store": {
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
I more or less understand the filters feature, and I assume I need to figure out a proper filter but it seems they are only useful to search nodes based on some criteria. I'm not sure if they are useful to remove elements based on filtering conditions.
Thanks so much for your help.

JSON Path is a tool for querying, not manipulation. You're not going to be able to alter the input value with it. You need another tool.
I'd suggest looking at something like https://jsonnet.org/ which is designed for template-based transformations and generation.

How to get data from an Old DB to a new One with different coulmns settings?

I'm using Spring Boot, MYSQL and Angular 7.
I have to redo the social network of my company taking data from their old DB.
I improved it changing some columns settings but I have to get every data for each user.
OLD DB JSON EXAMPLE:
{
***PROBLEM, have no idea why they do the first row like this***
"e92b088c-5546-47bc-a9cf-2dc6da22c594": {
"uuid": "e92b088c-5546-47bc-a9cf-2dc6da22c594",
"skills": [
{
"id": 64,
"name": "C#",
"certified": false
},
{
"id": 66,
"name": "ASP.NET",
"certified": false
},
{
"id": 73,
"name": "Sharepoint",
"certified": false
},
],
}
}
NEW DB JSON must look like this:
{
"uuid": "e92b088c-5546-47bc-a9cf-2dc6da22c594",
"skills": [
{
"id": 64,
"name": "C#",
"certified": false
},
{
"id": 66,
"name": "ASP.NET",
"certified": false
},
{
"id": 73,
"name": "Sharepoint",
"certified": false
},
]
}
I have 16k rows and I need to associate the first row of the old JSON with my uuid in new JSON.
I don't know why they did the first row like that and I have no idea how to solve it.
Is it possible to do it in Java?

How to parse a flattened Json?

I have a requirement to use flattened structure JSON, for example below hierarchical Json:
{
"employees": [
{
"employee1": {
"employeeId": 123,
"name": "ABC",
"type": "permanent",
"address": {
"street": "",
"city": "",
"zipcode": 123456
},
"phoneNumbers": [
123456,
987654
],
"designation": "Manager",
"properties": {
"age": "29 years",
"joiningDate": "17-may-2017",
"salary": "1000 USD"
}
}
},
{
"employee2": {
"employeeId": 123,
"name": "XYZ",
"type": "parttime",
"address": {
"street": "",
"city": "",
"zipcode": 345645
},
"phoneNumbers": [
345332,
675444
],
"designation": "Contractor",
"properties": {
"age": "35 years",
"joiningatDate": "17-june-2015",
"salary": "700 USD"
}
}
}
]
}
**Could be represented as flat structure Json as below(generated this using json-flattener):**
{
"employees[0].employee1.address.zipcode": 123456,
"employees[0].employee1.address.city": "",
"employees[0].employee1.address.street": "",
"employees[0].employee1.name": "ABC",
"employees[0].employee1.employeeId": 123,
"employees[0].employee1.designation": "Manager",
"employees[0].employee1.type": "permanent",
"employees[0].employee1.phoneNumbers[0]": 123456,
"employees[0].employee1.phoneNumbers[1]": 987654,
"employees[0].employee1.properties.joiningDate": "17-may-2017",
"employees[0].employee1.properties.salary": "1000 USD",
"employees[0].employee1.properties.age": "29 years",
"employees[1].employee2.address.zipcode": 345645,
"employees[1].employee2.address.city": "",
"employees[1].employee2.address.street": "",
"employees[1].employee2.name": "XYZ",
"employees[1].employee2.employeeId": 123,
"employees[1].employee2.designation": "Contractor",
"employees[1].employee2.type": "parttime",
"employees[1].employee2.phoneNumbers[0]": 345332,
"employees[1].employee2.phoneNumbers[1]": 675444,
"employees[1].employee2.properties.joiningDate": "17-june-2015",
"employees[1].employee2.properties.salary": "700 USD",
"employees[1].employee2.properties.age": "35 years"
}
My problem is if my service receives flattened Json like above, how to convert it to java domain objects automatically like:
class Employee{
Address address;
Properties property;
}
Does any Java Json parser supports this automatic conversion or i will have to implement own parsing logic splitting the keys based on dot in the keys?
Thanks in Advance.

You can take a look at json-flattener.
It does what you want, and even more.
The usage is simple as this:
String json = "{ \"a\" : { \"b\" : 1, \"c\": null, \"d\": [false, true] }, \"e\": \"f\", \"g\":2.3 }";
String jsonStr = JsonFlattener.flatten(json);
System.out.println(jsonStr);
// Output: {"a.b":1,"a.c":null,"a.d[0]":false,"a.d[1]":true,"e":"f","g":2.3}

Remove selected nodes from a JSON stream

I need to serve JSON from from my backend to the user. But before sending it over the wire I need to remove some data because it's confidential, every element who's key starts with conf_.
Assume I have the following JSON source:
{
"store": {
"book": [
{
"category": "reference",
"conf_author": "Nigel Rees",
"title": "Sayings of the Century",
"conf_price": 8.95
},
{
"category": "fiction",
"conf_author": "Evelyn Waugh",
"title": "Sword of Honour",
"conf_price": 12.99
},
{
"category": "fiction",
"conf_author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"conf_price": 8.99
},
{
"category": "fiction",
"conf_author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"conf_price": 22.99
}
],
"bicycle": {
"color": "red",
"conf_price": 19.95
}
},
"expensive": 10
}
Since the structure of the soruce JSON may vary (is not known), I need a way to identify the elements to remove by a pattern based on the key name (^conf_).
So the resulting JSON should be:
{
"store": {
"book": [
{
"category": "reference",
"title": "Sayings of the Century"
},
{
"category": "fiction",
"title": "Sword of Honour"
},
{
"category": "fiction",
"title": "Moby Dick",
"isbn": "0-553-21311-3"
},
{
"category": "fiction",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8"
}
],
"bicycle": {
"color": "red"
}
},
"expensive": 10
}
Since my source JSON will have 1m+ entries in the books array where every entry will have 100+ fields (child objects), I'm looking for some stream / event based approach like StAX rather then parsing the whole JSON into a JSONObject for manipulation for performance and resource reasons.
I looked at things like Jolt, JSONPath and JsonSurfer but these libraries did me get anywhere so far.
Can anyone provide some details on how my use case could be implemented best?
Regards!

You can use Jackson's Streaming API which can be used to parse huge JSON upto even giga bytes of size.It can be used to process huge files without loading them completely in memory.It allows get the data you want and ignore what you don't want also
Read more: http://wiki.fasterxml.com/JacksonStreamingApi

Way to query nearby location base on Latitude, Longitude and Radius

For example, I have a JSON database like this:
[
{
"address": "America",
"location": {
"lng": 106.669633,
"lat": 10.771883
},
"name": "724 (100) Pyrmon"
},
{
"address": "America",
"location": {
"lng": 106.668821,
"lat": 10.767997
},
"name": "900 (100) Glassdor"
},
{
"address": "328A, Đường Lê Hồng Phong, Ward 1, Quận 10, TP.HCM, Việt Nam",
"location": {
"lng": 106.676276,
"lat": 10.763239
},
"name": "Booking Office Of Phuong Trang Buslines"
},
{
"address": "America",
"location": {
"lng": 106.690329,
"lat": 10.768556
},
"name": "Opposite Pyrmont"
},
{
"address": "America",
"location": {
"lng": 106.691759,
"lat": 10.763937
},
"name": "150 Washington"
},
{
"address": "America",
"location": {
"lng": 106.681683,
"lat": 10.76965
},
"name": "596 - 598 Nguyễn Đình Chiểu"
},
{
"address": "America",
"location": {
"lng": 106.690158,
"lat": 10.77509
},
"name": "17 Bà Huyện Thanh Quan"
},
{
"address": "America",
"location": {
"lng": 106.699749,
"lat": 10.774052
},
"name": "101 Jase"
},
{
"address": "America",
"location": {
"lng": 106.682621,
"lat": 10.778547
},
"name": "274 Lost Region"
},
{
"address": "America",
"location": {
"lng": 106.67305,
"lat": 10.765617
},
"name": "66 Office Base"
},
{
"address": "America",
"location": {
"lng": 106.689733,
"lat": 10.763817
},
"name": "118 Yen The"
},
{
"address": "America",
"location": {
"lng": 106.69313,
"lat": 10.765497
},
"name": "112 Glangdie"
},
]
Now I have to make an algorithm (in any programming language, if in Java, it will be good) that will input the latitude, longitude, and the distance D . The output will be JSON data that contain the information of nearby places that have the distance to the input distance smaller than D .
I have researched about this algorithm, especially on Wikipedia, http://en.wikipedia.org/wiki/Geographical_distance#Singularities_and_discontinuity_of_latitude.2Flongitude.
But I cannot find a way to query from the database, if the database is BIG, I have to query each row(or each JSON Object in this case) to compare with the chosen location, that will be a very slow performance.
How can I do this?

You can try to reduce comparisons by filtering locations inside circumscribed square around circle with input radius. In SQL it looks like: WHERE lng>=(:lng - :radius) AND lng<=(:lng + :radius) AND lat>=(:lat - :radius) AND lat<=(:lat + :radius), where :lng, :lat - input coordinates, :radius - input radius in radians.