We Keep Coding

How to remove one set of "{}" from the output of JOLT transform array for each item and add a single set around all items?

Working with Nifi JoltTransformJSON processor, fairly new to using JOLT Specs and can't figure out how to move a set of {} from each item of an array to a single set of {} around all items of the array form the output JSON. Please see the OUTPUT and the NEEDED OUTPUT sections below for comparisons. I tried creating the spec from reading the JOLT definition and some examples which got me pretty close, but this last part has been kicking my rear for a while, any hints would be awesome!
INPUT JSON:
{
"AssetID": "1",
"AssetNumber": "2",
"AssetMaterial": "Cisco MDS 9706",
"RackUnits": "9.0",
"MaterialType": "Chassis",
"AssetName": "Cisco-MDS-9706_1",
"CustRID": "A001",
"SerialNumber": "OU812",
"Room": "ROOM5",
"Datacenter": "DC69",
"UMountingID": "86",
"CabinetAssetID": "181",
"CabinetName": "CAB666"
}
CURRENT SPEC:
[
{
"operation": "shift",
"spec": {
"AssetID": "data[].6.value",
"AssetNumber": "data[].7.value",
"AssetMaterial": "data[].8.value",
"AssetName": "data[].9.value",
"CustRID": "data[].10.value",
"SerialNumber": "data[].11.value",
"Room": "data[].12.value",
"Datacenter": "data[].13.value",
"UMountingID": "data[].14.value",
"CabinetAssetID": "data[].15.value",
"CabinetName": "data[].16.value"
}
},
{
"operation": "default",
"spec": {
"to": "table1"
}
},
{
"operation": "default",
"spec": {
"fieldsToReturn": [6, 7, 8, 9, 10, 11, 12]
}
}
]
OUTPUT JSON:
{
"data": [
{
"6": {
"value": "1"
}
},
{
"7": {
"value": "2"
}
},
{
"8": {
"value": "Cisco MDS 9706"
}
},
{
"9": {
"value": "Cisco-MDS-9706_1"
}
},
{
"10": {
"value": "A001"
}
},
{
"11": {
"value": "OU812"
}
},
{
"12": {
"value": "ROOM5"
}
},
{
"13": {
"value": "DC69"
}
},
{
"14": {
"value": "86"
}
},
{
"15": {
"value": "181"
}
},
{
"16": {
"value": "CAB666"
}
}
],
"to": "table1",
"fieldsToReturn": [ 6, 7, 8, 9, 10, 11, 12 ]
}
REQUIRED/EXPECTED OUTPUT:
{
"data" : [
{
"6" : {
"value" : "1"
},
"7" : {
"value" : "2"
},
"8" : {
"value" : "Cisco MDS 9706"
},
"9" : {
"value" : "Cisco-MDS-9706_1"
},
"10" : {
"value" : "A001"
},
"11" : {
"value" : "OU812"
},
"12" : {
"value" : "ROOM5"
},
"13" : {
"value" : "DC69"
},
"14" : {
"value" : "86"
},
"15" : {
"value" : "181"
},
"16" : {
"value" : "CAB666"
}
}
],
"to" : "table1",
"fieldsToReturn" : [ 6, 7, 8, 9, 10, 11, 12 ]
}

What you need seems that the top level object should reside at the common index of the array data, so use index 0 such as data[0]. to combine those object as single object such as
[
{
"operation": "shift",
"spec": {
"AssetID": "data[0].6.value",
"AssetNumber": "data[0].7.value",
"AssetMaterial": "data[0].8.value",
...
...
"#table1": "to"
}
},
{
"operation": "default",
"spec": {
"fieldsToReturn": [6, 7, 8, 9, 10, 11, 12]
}
},
{
"operation": "sort"
}
]
btw, the first default transformation spec is superfluous, rather use
"#table1": "to"
within the shift transformation spec

Scala - Ids lists of objects with duplicated values from spark dataset

I need to create an IDs lists for all objects that have identical (same value and quantity) parameters. I am looking for a solution that will be more efficient than two nested loops and an if.
Object structure in the dataset:
case class MergedProduct(id: String,
products: List[Product])
case class Product(productUrl: String, productId: String)
Example of data in dataset:
[ {
"id": "ID1",
"products": [
{
"product": {
"productUrl": "SOMEURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID2",
"products": [
{
"product": {
"productUrl": "SOMEURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID3",
"products": [
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
}
],
},
{
"id": "ID4",
"products": [
{
"product": {
"productUrl": "SOMEOTHERURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
}
],
},
{
"id": "ID5",
"products": [
{
"product": {
"productUrl": "NOTDUPLICATEDURL",
"productId": "1"
}
},
{
"product": {
"productUrl": "DIFFERENTURL",
"productId": "1"
}
}
],
}
]
In this example, we have 4 objects that are duplicated, so I would like to get their ID in the corresponding lists.
Example output is List[List[String]]:
List(List("ID1", "ID2"), List("ID3","ID4"))
I am looking for something efficient and readable - the dataset we are talking about has nearly 700 million objects.
As I can remove the listed duplicates from the dataset (it does not affect the database) because the goal is one - logging them exists, so I was thinking about the solution of taking MergedProduct one by one, searching for other MergedProduct with identical Products, getting their ID, logging in they exist and then remove the mentioned MergedProduct ID from the dataset and move on to the next one until I check the whole dataset but in this case I would have to collect it first as a list of MergedProducts and then do all operations - seems like going around

After trying some options and looking for neat solutions- I think this is kinda ok:
private def getDuplicates(mergedProducts: List[MergedProduct]): List[List[String]] = {
val duplicates = mergedProducts.groupBy(_.products.sortBy(_.product.productId)).filter(_._2.size > 1).values.toList
duplicates.map(duplicates => duplicates.map(_.id))
}

How to split JSON into Dataset rows?

I have the following JSON input data:
{
"lib": [
{
"id": "a1",
"type": "push",
"icons": [
{
"iId": "111"
}
],
"id": "a2",
"type": "pull",
"icons": [
{
"iId": "111"
},
{
"iId": "222"
}
]
}
]
I want to get the following Dataset:
id type iId
a1 push 111
a2 pull 111
a2 pull 222
How can I do it?
This is my current code. I use Spark 2.3 and Java 1.8:
ds = spark
.read()
.option("multiLine", true).option("mode", "PERMISSIVE")
.json(jsonFilePath);
ds = ds
.select(org.apache.spark.sql.functions.explode(ds.col("lib.icons")).as("icons"));
However the result is wrong:
+---------------+
| icons|
+---------------+
| [[111]]|
|[[111], [222...|
+---------------+
How can I get the correct Dataset?
UPDATE:
I tries this code, but it generates some extra combinations of id, type and iId that do not exist in the input file.
ds = ds
.withColumn("icons", org.apache.spark.sql.functions.explode(ds.col("lib.icons")))
.withColumn("id", org.apache.spark.sql.functions.explode(ds.col("lib.id")))
.withColumn("type", org.apache.spark.sql.functions.explode(ds.col("lib.type")));
ds = ds.withColumn("its", org.apache.spark.sql.functions.explode(ds.col("icons")));

As already pointed out, the JSON String seems to be malformed. with the updated one, you can use the following to get result you wanted:
import org.apache.spark.sql.functions._
spark.read
.format("json")
.load("in/test.json")
.select(explode($"lib").alias("result"))
.select($"result.id", $"result.type", explode($"result.icons").alias("iId"))
.select($"id", $"type", $"iId.iId")
.show

Your JSON appears to be malformed. Fixing the indenting makes this slightly more apparent:
{
"lib": [
{
"id": "a1",
"type": "push",
"icons": [
{
"iId": "111"
}
],
"id": "a2",
"type": "pull",
"icons": [
{
"iId": "111"
},
{
"iId": "222"
}
]
}
]
Does your code work correctly if you feed it this JSON instead?
{
"lib": [
{
"id": "a1",
"type": "push",
"icons": [
{
"iId": "111"
}
]
},
{
"id": "a2",
"type": "pull",
"icons": [
{
"iId": "111"
},
{
"iId": "222"
}
]
}
]
}
Note the inserted }, { just before "id": "a2" to break the object with duplicate keys into two, and the closing } at the very end which had previously been omitted.

How to index a Json object with object and its reference in elasticsearch?

I am working with Elasticsearch recently, and I meet a problem that don't know how to solve it.
I have a Json like:
{
"objects": [
"object1": {
"id" : "12345",
"name":"abc"
},
"12345"
]
}
Object2 is a reference of object1, when I trying to saving(or called indexing) into elastic search, it says:
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
After I google I found that because object1 is an object, but object 2 is considered as a string.
We cannot change our json in our project, so in this case how can I save it in the elasticsearch?
Thanks for any help and suggestion.

How do you do that?
I run this command and it works.
PUT test/t1/1
{
"objects": {
"object1": {
"id" : "12345",
"name":"abc"
},
"object2": "12345"
}
}
and the result is:
{
"_index": "test",
"_type": "t1",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"created": true
}
UPDATE 1
Depending on your requirements one of these may solve your problem:
PUT test/t1/2
{
"objects": [
{
"object1": {
"id": "12345",
"name": "abc"
}
},
{
"object2": "12345"
}
]
}
PUT test/t1/2
{
"objects": [
{
"object1": {
"id": "12345",
"name": "abc"
},
"object2": "12345"
},
{
...
}
]
}

Parsing Json data giving ClassCastException

I am currently using the following JSON parser but keep getting the error Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.json.JSONObject. How can I fix this, it has worked many times before.
ublic List<String> handleResponse(HttpResponse response)
throws ClientProtocolException, IOException {
List<String> result = new ArrayList<String>();
String JSONResponse = new BasicResponseHandler()
.handleResponse(response);
try {
// Get top-level JSON Object - a Map
JSONObject responseObject = (JSONObject) new JSONTokener(
JSONResponse).nextValue(); //Problem Line
JSONObject item1 = responseObject.getJSONObject("searchResult");
// Extract value of "Item" key -- a List
JSONArray items = item1
.getJSONArray("Item");
Below is what I'm looking to parse
{
"findItemsAdvancedResponse":[
{
"ack":[
"Success"
],
"version":[
"1.13.0"
],
"timestamp":[
"2015-02-24T18:43:51.778Z"
],
"searchResult":[
{
"#count":"100",
"item":[
{
"itemId":[
"381098176861"
],
"title":[
" Tokyo Laundry Women's Kenzie Sweatpants - Eclipse Blue - TP1"
],
"globalId":[
"EBAY-GB"
],
"subtitle":[
"FREE UK DELIVERY | DISPATCHED WITHIN 1 WORKING DAY"
],
"primaryCategory":[
{
"categoryId":[
"137085"
],
"categoryName":[
"Activewear"
]
}
],
"secondaryCategory":[
{
"categoryId":[
"155226"
],
"categoryName":[
"Hoodies & Sweats"
]
}
],
"galleryURL":[
"http:\/\/thumbs2.ebaystatic.com\/pict\/3810981768614040_1.jpg"
],
"viewItemURL":[
"http:\/\/www.ebay.co.uk\/itm\/Tokyo-Laundry-Womens-Kenzie-Sweatpants-Eclipse-Blue-TP1-\/381098176861?pt=LH_DefaultDomain_3&var="
],
"paymentMethod":[
"PayPal"
],
"autoPay":[
"true"
],
"location":[
"United Kingdom"
],
"country":[
"GB"
],
"shippingInfo":[
{
"shippingServiceCost":[
{
"#currencyId":"GBP",
"__value__":"0.0"
}
],
"shippingType":[
"Free"
],
"shipToLocations":[
"AU",
"Europe"
]
}
],
"sellingStatus":[
{
"currentPrice":[
{
"#currencyId":"GBP",
"__value__":"14.99"
}
],
"convertedCurrentPrice":[
{
"#currencyId":"GBP",
"__value__":"14.99"
}
],
"sellingState":[
"Active"
],
"timeLeft":[
"P25DT14H10M23S"
]
}
],
"listingInfo":[
{
"bestOfferEnabled":[
"false"
],
"buyItNowAvailable":[
"false"
],
"startTime":[
"2014-12-22T08:49:14.000Z"
],
"endTime":[
"2015-03-22T08:54:14.000Z"
],
"listingType":[
"FixedPrice"
],
"gift":[
"false"
]
}
],
"galleryPlusPictureURL":[
"http:\/\/galleryplus.ebayimg.com\/ws\/web\/381098176861_1_1_1.jpg"
],
"condition":[
{
"conditionId":[
"1000"
],
"conditionDisplayName":[
"New with tags"
]
}
],
"isMultiVariationListing":[
"true"
],
"topRatedListing":[
"false"
]
},
{
"itemId":[
"131321072184"
],
"title":[
"Brave Soul Women's Dallas Parka - Mulberry - DD6 - Size 8-16"
],
"globalId":[
"EBAY-GB"
],
"subtitle":[
"FREE UK DELIVERY | DISPATCHED WITHIN 1 WORKING DAY"
],
"primaryCategory":[
{
"categoryId":[
"137085"
],
"categoryName":[
"Activewear"
]
}
],
"secondaryCategory":[
{
"categoryId":[
"63862"
],
"categoryName":[
"Coats & Jackets"
]
}
],
"galleryURL":[
"http:\/\/thumbs1.ebaystatic.com\/pict\/1313210721844040_3.jpg"
],
"viewItemURL":[
"http:\/\/www.ebay.co.uk\/itm\/Brave-Soul-Womens-Dallas-Parka-Mulberry-DD6-Size-8-16-\/131321072184?pt=LH_DefaultDomain_3&var="
],
"paymentMethod":[
"PayPal"
],
"autoPay":[
"true"
],
"location":[
"United Kingdom"
],
"country":[
"GB"
],
"shippingInfo":[
{
"shippingServiceCost":[
{
"#currencyId":"GBP",
"__value__":"0.0"
}
],
"shippingType":[
"Free"
],
"shipToLocations":[
"AU",
"Europe"
]
}
],
"sellingStatus":[
{
"currentPrice":[
{
"#currencyId":"GBP",
"__value__":"19.99"
}
],
"convertedCurrentPrice":[
{
"#currencyId":"GBP",
"__value__":"19.99"
}
],
"sellingState":[
"Active"
],
"timeLeft":[
"P15DT14H57M21S"
]
}
],
"listingInfo":[
{
"bestOfferEnabled":[
"false"
],
"buyItNowAvailable":[
"false"
],
"startTime":[
"2014-10-13T09:36:12.000Z"
],
"endTime":[
"2015-03-12T09:41:12.000Z"
],
"listingType":[
"FixedPrice"
],
"gift":[
"false"
]
}
],
"galleryPlusPictureURL":[
"http:\/\/galleryplus.ebayimg.com\/ws\/web\/131321072184_1_5_1.jpg"
],
"condition":[
{
"conditionId":[
"1000"
],
"conditionDisplayName":[
"New with tags"
]
}
],
"isMultiVariationListing":[
"true"
],
"topRatedListing":[
"false"
]
},

Why do you need a JSONTokener? This is much easier:
JSONObject responseObject = new JSONObject(JSONResponse);
JSONArray main = responseObject.getJSONArray("findItemsAdvancedResponse");
JSONObject result = main.getJSONObject(0).getJSONArray("searchResult").getJSONObject(0);
JSONArray items = result.getJSONArray("item");

Your nextValue() method must be returning a String, which you are then trying to cast to a JSONObject. I would start by splitting the statement up into smaller statements to help isolate the issue. You are instantiating a new object, calling a method on it, and type casting all on the same line. That makes code harder to debug.
JSONTokener tokener = new JSONTokener(JSONResponse);
Object obj = tokener.nextValue();
if(obj.getClass().equals(JSONObject.class))
{
JSONObject responseObject = (JSONObject)obj;
}
else
{
System.out.println("Not a JSONObject");
}