For a student project I have to improve data quality. The first step is to request an API. Secondly, we have to edit the json structure.
This is the response from the API :
{
"lists": [
[
0,
451,
"test",
"953"
],
[
2,
1010,
"hello",
"610"
]
]
}
Now using jolt I want to have a result like that :
{
"lists": [
{
"id": 0,
"clientId": 451,
"name": "test",
"custom_value": "953"
},
{
"id": 2,
"clientId": 1010,
"name": "hello",
"custom_value": "610"
}
]
}
Currently, I can access to data values but I don't know how to separate it into array with objects.
My 'code' :
[
{
"operation": "shift",
"spec": {
"lists": {
"*": {
"*": {
"*": {
"$0": "lists"
}
}
}
}
}
}
]
Where I'm wrong and how can I edit the structure of the original array properly?
Spec
[
{
"operation": "shift",
"spec": {
"lists": {
"*": { // lists array
"0": "lists[&1].id",
"1": "lists[&1].clientId",
"2": "lists[&1].name",
"3": "lists[&1].custom_value"
}
}
}
}
]
Related
I am trying to transform the below JSON:
Input:
{
"steps": [
{
"end": "2023-01-27T09:19:29.849298Z",
"id": "1",
"start": "2023-01-27T09:18:24.59325Z",
"name": "foo"
},
{
"end": "2023-01-28T09:19:29.849298Z",
"id": "2",
"start": "2023-01-28T09:18:24.59325Z",
"name": "bar"
}
]
}
Output:
{
"steps": [
{
"end": "2023-01-27T09:19:29.849298Z",
"id": "1",
"name": "foo",
"start": "2023-01-27T09:18:24.59325Z"
},
{
"end": "2023-01-28T09:19:29.849298Z",
"id": "2",
"name": "bar",
"start": "2023-01-28T09:18:24.59325Z"
}
],
"date": [
{
"name": "startDate",
"value": "2023-01-27T09:18:24.59325Z" //steps[0].start
},
{
"name": "endDate",
"value": "2023-01-27T09:19:29.849298Z" //steps[0].end
}
]
}
I tried using the below spec:
[
{
"operation": "shift",
"spec": {
"steps": {
"*": "steps[]",
"0": {
"#startDate": "date[0].name",
"start": "date[0].value",
"end": "date[1].value",
"#endDate": "date[1].name"
}
}
}
}
]
But "*": "steps[]" only transforms the last element of the array steps. Please guide me as to what is wrong with the above spec, as I am new to jolt. Also, any pointers to the correct operations needed to achieve the above output will be greatly appreciated.
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": {
"#": "&1",
"0": {
"#startDate": "date[0].name",
"start": "date[0].value",
"#endDate": "date[1].name",
"end": "date[1].value"
}
}
}
}
]
You can use # to get the steps array and put it into the steps key with &1.
The issue with the given Jolt spec is that the transformation of the steps array elements is only happening for the last element of the array. This is because the "*": "steps[]" directive only applies to the last element of the array.
To transform all elements of the steps array, you can use the following Jolt spec:
[
{
"operation": "shift",
"spec": {
"steps": {
"0": {
"start": "date[0].value",
"end": "date[1].value"
},
"*": {
"*": "steps[&1].&",
"#startDate": "date[0].name",
"#endDate": "date[1].name"
}
}
}
}
]
One option is to conditionally pick by the existing(steps) array's indexes while walking through it such as
[
{
"operation": "shift",
"spec": {
"steps": {
"#1": "",// derive the whole value from the upper level, eg. replicate it
"0": {// the first index
"#startDate": "date[&1].name",
"start": "date[&1].value"
},
"*": {// the other index(this case there's only one)
"#endDate": "date[&1].name",// bring the value of the wrapper object'S index by using &1(going up one level the tree)
"end": "date[&1].value"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
I have a following JSON :
{
"timestamp": 1665923527,
"place": "us",
"event": "custom",
"Users": [
{
"email": "test1#test.com",
"key1": "Admin"
},
{
"email": "test2#test.com",
"key1": "Admin"
}
]
}
which I want to modify it to multiple array based on the users key dynamically. In the above example I have 2 arrays inside user. So I want the output to be:
[
{
"timestamp": 1665923527,
"place": "us",
"event": "custom",
"email": "test1#test.com",
"key1": "Admin"
},
{
"timestamp": 1665923527,
"place": "us",
"event": "custom",
"email": "test2#test.com",
"key1": "Admin"
}
]
Is it possible with Jolt?
I did search in all the documents i found, but no luck
You can walk through the attributes of the innermost array(Users) by using a shift transformation spec while copying the values of the others grabbed from after going two levels up the tree such as
[
{
"operation": "shift",
"spec": {
"Users": {
"*": {
"#(2,timestamp)": "[&1].timestamp", // go 2 levels up the tree to grab the value of timestamp by using "#(2,timestamp)", [&1] stands for going 1 level up to reach the level of "Users" array to combine all values at that level
"#(2,place)": "[&1].place",
"#(2,event)": "[&1].event",
"*": "[&1].&"
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
Edit : alternatively you can prefer the below one to make it dynamic :
[
{
// separate the attributes into two groups "x" and "y"
"operation": "shift",
"spec": {
"Users": {
"*": {
"*": {
"#": "x.&2.&"
}
}
},
"*": "y.&"
}
},
{
// combine each attributes under common index values
"operation": "shift",
"spec": {
"x": {
"*": {
"#(2,y)": "&2.&"
},
"#": "&"
}
}
},
{
// dive deep into indexes
"operation": "shift",
"spec": {
"x": {
"*": {
"*": {
"*": "[&2].&"
}
}
}
}
}
]
the demo on the site http://jolt-demo.appspot.com/ is
I have JSON Input as below
{
"data": [
{
"items": [
{
"item": "R11",
"code": "8611A"
},
{
"item": "R2",
"code": "8611B"
}
]
}
]
}
I need to extract items array as below
[
{
"item": "R11",
"code": "8611A"
},
{
"item": "R2",
"code": "8611B"
}
]
Please copy and paste the above INPUT and OUTPUT https://jolt-demo.appspot.com/#inception
I have tested [{"operation":"shift","spec":{"data":{"*":""}}}]
But it returns {"items" : [ {...}, {...} ] }
You can use a shift transformation as below
[
{
"operation": "shift",
"spec": {
"*": { //might be replaced with "data"
"*": {
"*": //might be replaced with "items"
{
"#": ""
}
}
}
}
}
]
where the asterisks wildcards represent the array data, its indexes, and the array items respectively. The "#":"" represents the items as array of objects with no key.
Alternative Spec:
[
{
"operation": "shift",
"spec": {
"data": {
"*": {
"items": ""
}
}
}
}
]
Here is the answer
[{
"operation": "shift",
"spec": {
"data": {
"0": {
"*": ""
}
}
}
}]
I am doing conversion of string to integer/double in my jolt spec. If elements does not come in request, then empty object is coming in response. Instead I do not want to pass that object itself as its empty.
Input Request Working:
{
"data": {
"first": "1",
"second": "2"
}
}
Jolt Spec:
[
{
"operation": "shift",
"spec": {
"data": {
"first": "datas.firstTag",
"second": "datas.second.secondTag"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"datas": {
"firstTag": "=toInteger",
"second": {
"secondTag": "=toInteger"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
Output if tag is there in request:
{
"datas" : {
"firstTag" : 1,
"second" : {
"secondTag" : 2
}
}
}
But when input request is like below where I do not get second tag:
{
"data": {
"first": "1"
}
}
Current Output:
{
"datas" : {
"firstTag" : 1,
"second" : { }
}
}
Getting second object as empty
"second" : { }
Expected Output:
{
"datas" : {
"firstTag" : 1
}
}
Please help to fix this issue.
Rethink in a dynamical manner by using ampersand placeholders such as
[
{
"operation": "shift",
"spec": {
"*": {
"first": "&1s.&Tag",
"*": { "#(1,&)": "&2s.&.&Tag" }
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"firstTag": "=toInteger",
"*": {
"*": "=toInteger"
}
}
}
}
]
the suffixed integers for those ampersands represents the level to go up to grab the desired value such as &1 and &2 stand for data, & without integer suffixes represent the value from the current level.
"*" wildcard represents the rest of the elements, other than first in this case(you might have third, fourth ... as other elements to be formatted like the second)
first case :
second case :
Edit : For the current case which's mentioned within the edit in order to get
{
"datas" : {
"firstTag" : "1",
"second" : {
"secondTag" : "2",
"secondNew" : "3"
}
}
}
from the input
{
"data": {
"first": "1",
"second": "2",
"secondNew": "3"
}
}
which had a new entry "secondNew": "3", you'll need to write each key explicitly within the spec such as
[
{
"operation": "shift",
"spec": {
"*": {
"first": "&1s.&Tag",
"second": "&1s.&.&Tag",
"secondNew": "&1s.second.&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": {
"firstTag": "=toInteger",
"*": {
"*": "=toInteger"
}
}
}
}
]
in order to combine the rest of the elements under common object.
For input
{
"field1": {
"array1": [
"a",
"b"
],
"array2": [
"c",
"d"
]
}
}
I want the output:
{
"objectArray": [
{
"role": "to",
"id": "a"
},
{
"role": "to",
"id": "b"
},
{
"role": "to",
"id": "c"
},
{
"role": "to",
"id": "d"
}
]
}
I currently am trying:
[
{
"operation": "shift",
"spec": {
"field1": {
"array1": {
"#to": "objectArray[].role",
"#": "objectArray[].id"
},
"array2": {
"#to": "objectArray[].role",
"#": "objectArray[].id"
}
}
}
}
]
However this seems to separate out the roles and ids into their own separate objects, after the first one.
You can use this two step of shift transformation spec.
[
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": "o[].id"
}
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": "objectArray[&1].&",
"#to": "objectArray[&1].role"
}
}
}
}
]
where you got and array for all ids in the first step, and then adding an arbitrarily chose key-value pairs within the second step while renaming the target array