I have an issue while trying to deidentify some data with DLP using an object mapper to parse the object into string - send it to DLP for deidentification - getting back the deidentified string and using the object mapper to parse the string back to the initial object. Sometimes DLP will return a string that cannot be parsed back to the initial object (it breaks the json format of the object mapper)
I use an objectMapper to parse an Address object to string like this:
Address(
val postal_code: String,
val street: String,
val city: String,
val provence: String
)
and my objectmapper will transform this object into a string eg: "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My Provence\"}" which is sent to DLP and deidentified (using LOCATION or STREET_ADDRESS detectors).
The issue is that my object mapper would expect to take back the deidentified string and parse it back to my Address object using the same json format eg:
"{\"postal_code\":\"LOCATION_TOKEN(10):asdf\",\"street\":\"LOCATION_TOKEN(10):asdf\",\"city\":\"LOCATION_TOKEN(10):asdf\",\"provence\":\"LOCATION_TOKEN(10):asdf\"}"
But there are a lot of times that DLP will return something like
"{"LOCATION_TOKEN(25):asdfasdfasdf)\",\"provence\":\"LOCATION_TOKEN(10):asdf\"}" - basically breaking the json format and i am unable to parse back the string from DLP to my initial object
Is there a way to instruct DLP infotype detectors to keep the json format, or to look for sensitive text only inside \" * \"?
Thanks
There are some options here using a custom regex and a detection ruleset in order to define a boundary on matches.
The general idea is that you require that findings must match both an infoType (e.g. STREET_ADDRESS, LOCATION, PERSON_NAME, etc.) and your custom infoType before reporting as a finding or for redaction. By requiring that both match, you can set bounds on where the infoType can detect.
Here is an example.
{
"item": {
"value": "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My Provence\"}"
},
"inspectConfig": {
"customInfoTypes": [
{
"infoType": {
"name": "CUSTOM_BLOCK"
},
"regex": {
"pattern": "(:\")([^,]*)(\")",
"groupIndexes": [
2
]
},
"exclusionType": "EXCLUSION_TYPE_EXCLUDE"
}
],
"infoTypes": [
{
"name": "EMAIL_ADDRESS"
},
{
"name": "LOCATION"
},
{
"name": "PERSON_NAME"
}
],
"ruleSet": [
{
"infoTypes": [
{
"name": "LOCATION"
}
],
"rules": [
{
"exclusionRule": {
"excludeInfoTypes": {
"infoTypes": [
{
"name": "CUSTOM_BLOCK"
}
]
},
"matchingType": "MATCHING_TYPE_INVERSE_MATCH"
}
}
]
}
]
},
"deidentifyConfig": {
"infoTypeTransformations": {
"transformations": [
{
"primitiveTransformation": {
"replaceWithInfoTypeConfig": {}
}
}
]
}
}
}
Example output:
"item": {
"value": "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My [LOCATION]\"}"
},
By setting "groupIndexes" to 2 we are indicating that we only want the custom infoType to match the middle (or second) regex group and not allow the :" or " to be part of the match. Also, in this example we mark the custom infoType as EXCLUSION_TYPE_EXCLUDE so that it does not report itself:
"exclusionType": "EXCLUSION_TYPE_EXCLUDE"
If you remove this line, anything matching your infoType could also get redacted. This can be useful for testing though - example output:
"item": {
"value": "{\"postal_code\":\"[CUSTOM_BLOCK]\",\"street\":\"[CUSTOM_BLOCK]\",\"city\":\"[CUSTOM_BLOCK]\",\"provence\":\"[CUSTOM_BLOCK][LOCATION]\"}"
},
...
Hope this helps.
i am trying to generate a class model from an APIs JSON response (List), i use some service for auto-generation, and everything seems to work fine until the response JSON start returning attributes with a null value and the other thing is that some attributes are missing when i call the APIs. EX:
response JSON:
[
{
"persone": {
"name": "To John",
"age": 15
},
"group": "ipsum"
},
{
"persone": {
"name": "To John",
"age": null
},
"group": "ipsum"
},
{
"persone": {
"name": "To John"
}
}
]
in the above example, the age attribute is null, and the group attribute is missing in the last object of the list.
My question is how do you create an appropriate class model in that case handling the null and the missing attribute?
I have a Json object like this in my Postgres DB:
{
"emails": [
{
"email": {
"id": "e8dc927f-679d-496b-85fb-465edf35c676",
"value": "hello#gmail.com"
}
},
{
"email": {
"id": "1b78758a-abc4-46ef-9de9-c999a0c8c418",
"value": "hello1#gmail.com"
}
}
],
"lastName": {
"id": "718109fd-2d00-475a-829a-c8af9a7f0067",
"value": "lastName"
},
"firstName": {
"id": "6c46a5b3-6f89-4692-a214-4943de22018d",
"value": "firstName"
},
}
And so on big json with around 1000 elements, now I want to parse and get the first 500 elements from json and make another json. what I mean by element here is anything which has Id is a element. For example firstName , LastName, email, email are the elements not the emails. I tried Jackson api but couldn't find a way how to count the elements and make a json exactly like above and return. and when I do any modifications in the first 500 elements I should save the Json with edits. Any help is much appreciated. I even tried postgres array_agg(e) function but that is only accepting only array.
I have looking to similar questions but i was not able to solve my problem .I have tried parsing my json response both into a Json array and Json object but every time , i am getting this exception of string to json object.
I guess may be the problem is with my Json response.
results = { "type": 1, "user_log": "ahsan.tahir.92_/var/www/html/2014-10-11__15-54-52__50sec.txt_res", "freq": 0.01, "coordina": [ [-37], [-9], [-20], [-12], [-22], [-9], [-22], [-15], [-25], [-7], [-20], [-12], [-20], [-9], [-25], [8], [-23], [-11], [-18], [-13], [-19], [-10], [-21], [-12], [-25], [-11], [-17], [-12], [-22], [-13], [-21], [3] ], "tot_time": 40.11, "tot_distace": 100, "stroke_each_pool": [ [16], [16] ], "tot_stroke": [ [16], [16] ], "split": [ [20.22], [19.89] ], "timing_turn": 7.04, "cycle_Rate_l": [ [2.22], [2.2680001] ], "cycle_Rate_r": [ [2.224], [2.27] ], "mean_velocity": [ [2.4727993], [2.5138261] ], "stroke_length": [ [3.125], [3.125] ], "stroke_freq": [ [79.077431], [77.419357] ], "roll_peaks": [ [-44.10043335], [55.79428101], [-61.51541138], [54.7466507], [-62.09820557], [55.01488495], [-62.48770142], [53.44023132], [-70.32449341], [51.8399353], [-65.84837341], [53.5617981], [-63.50210571], [55.9821167], [-62.37905121], [39.42669678], [-43.44207764], [63.20912933], [-59.19660187], [50.6708374], [-63.8214798], [54.57595062], [-63.31864166], [53.82037354], [-66.93650818], [52.36277008], [-65.23461151], [52.89829254], [-62.78508759], [51.17367554], [-62.87123108], [59.13114929] ], "mean_roll_dx": [ [52.475822], [54.73027] ], "mean_roll_sx": [ [61.531971], [60.950779] ], "std_roll_dx": [ [5.4471908], [4.3127728] ], "std_roll_sx": [ [7.6123171], [7.4134283] ], "mean_roll": [ [57.003899], [57.840527] ], "std_roll": [ [7.9220791], [6.6817732] ], "mean_pitch": [ [-5.5227709], [-5.2282872] ], "std_pitch": [ [-5.5227709], [-5.2282872] ], "clean_stroke_time": [ [15.92], [16.84] ], "errore": 227, "fatal_error": { "_ArrayType_": "double", "_ArraySize_": [0,0], "_ArrayData_": null } }
and i am parsing it like this :
JSONObject reader = new JSONObject(in);
JSONObject sys = reader.getJSONObject("results");
Any idea what am i missing here ?
My exception is as follows :
org.json.JSONException: Value type of type java.lang.String cannot be converted to JSONObject
There is no results field in the results string you have posted, is that the issue. In any case, please go ahead and reformat your question to more accurately reflect what you are doing, and include the error message you are seeing.
Actually, are you referring to the error message in the JSON output? That seems to be a JSON-encoded error message that was output by whatever service you called. There seems to be some issue with the data you are passing to that service.
This:
results = {rest of response here}
is not valid JSON.
Perhaps you meant to return this:
{"results": {rest of response here}}
... or just
{rest of response here}
I may have Misunderstood your question, but where are you parsing the Json response using something like
JSONObject jsonObject = (JSONObject) jsonParser.parse(strippedJSON);
where strippedJSON the response received from API.
Your reader object is your results object itself.
JSONObject resultsJSON = new JSONObject(in);
System.out.println("User Log: " + resultsJSON.get("user_log"));
You're getting the exception because there's no results object in your input JSON string i.e. you're receiving your results object without a JSON wrapper around it.
If you are using javascript
try this
var parsed_data = JSON && JSON.parse(results) || $.parseJSON(results);
In my project when i send Rest Response to Advance Rest Client It only shows Fields which Have some values and Ignores(Does not show) fields which have NULL Values or Empty values.
Part Of Code:
Gson gson=new Gson();
// firstResponse is Object which contains the values
String jsonString = gson.toJson(firstResponse);
test.saveJson(jsonString); //OR System.out.println(jsonString);
return Response.ok(firstResponse).build(); // Response to Rest Client
Response sample To return Response.ok(firstResponse).build();
Advance rest client From web project :
{
"Name": "smith",
"Properties": {
"propertyList": [
{
"ID": "072",
"Number": "415151",
"Address": "Somewhere"
},
{
"ID": "151",
"Number": "a800cc79-99d1-42f1-aeb4-808087b12c9b",
"Address": "ninink"
},
{
"ID": "269",
},
],
},
}
Now when i save this as Json String in DB or When i want to Print this to console it also prints the fiels with null or empty values:
{
"Name": "smith",
"Properties": {
"propertyList": [
{
"ID": "072",
"Number": "415151",
"Address": "Somewhere"
},
{
"ID": "151",
"Number": "a800cc79-99d1-42f1-aeb4-808087b12c9b",
"Address": "ninink"
},
{
"ID": "269",
"Number": "",
"Address": ""
},
],
},
"resultList" :[]
}
How can i print or save this JSON string same as response in rest client i.e. i dont want to print null or empty value field i just want to ignore them.
in top og entity class , try with the annotation
#JsonInclude(Include.NON_EMPTY)
this annotation don't show any empty field in your json.
Not giving you a code but here are some pointers for you:
Read the manual: How to handle NULL values
You may need to use a custom exclusion strategy
Also read this Q&A: Gson: How to exclude specific fields from Serialization without annotations