I have written a few rest services to extract data from a mysql database and display it in json format on Postman client. However for some of the response items, I get this unwanted \\r character in the response values. For example:
{
"colour": "yellow",
"deliveryCharge": 5,
"description": "Mangoes from Ratnagiri",
"keyFeatures": [
"Seedless\\\r",
"Ripe and Sweet"
],
"price": 100,
"productId": 49,
"productName": "Alfonso",
"specifications": {
"entry": [
{
"key": "Feature",
"value": "N/A"
}
]
},
"stock": 20,
"warranty": 1
}
How do I get rid of these characters in the final json response? I have tried isolating them using .replace() and .split() methods for the output strings fetched from the database, but it doesn't work.
You have two escaped character \ and \r in the string. trim method in String class can remove \r but not \. If you don't want to send these character, You will have to do it yourself before serializing the objec to to JSON.
Related
I have an issue while trying to deidentify some data with DLP using an object mapper to parse the object into string - send it to DLP for deidentification - getting back the deidentified string and using the object mapper to parse the string back to the initial object. Sometimes DLP will return a string that cannot be parsed back to the initial object (it breaks the json format of the object mapper)
I use an objectMapper to parse an Address object to string like this:
Address(
val postal_code: String,
val street: String,
val city: String,
val provence: String
)
and my objectmapper will transform this object into a string eg: "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My Provence\"}" which is sent to DLP and deidentified (using LOCATION or STREET_ADDRESS detectors).
The issue is that my object mapper would expect to take back the deidentified string and parse it back to my Address object using the same json format eg:
"{\"postal_code\":\"LOCATION_TOKEN(10):asdf\",\"street\":\"LOCATION_TOKEN(10):asdf\",\"city\":\"LOCATION_TOKEN(10):asdf\",\"provence\":\"LOCATION_TOKEN(10):asdf\"}"
But there are a lot of times that DLP will return something like
"{"LOCATION_TOKEN(25):asdfasdfasdf)\",\"provence\":\"LOCATION_TOKEN(10):asdf\"}" - basically breaking the json format and i am unable to parse back the string from DLP to my initial object
Is there a way to instruct DLP infotype detectors to keep the json format, or to look for sensitive text only inside \" * \"?
Thanks
There are some options here using a custom regex and a detection ruleset in order to define a boundary on matches.
The general idea is that you require that findings must match both an infoType (e.g. STREET_ADDRESS, LOCATION, PERSON_NAME, etc.) and your custom infoType before reporting as a finding or for redaction. By requiring that both match, you can set bounds on where the infoType can detect.
Here is an example.
{
"item": {
"value": "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My Provence\"}"
},
"inspectConfig": {
"customInfoTypes": [
{
"infoType": {
"name": "CUSTOM_BLOCK"
},
"regex": {
"pattern": "(:\")([^,]*)(\")",
"groupIndexes": [
2
]
},
"exclusionType": "EXCLUSION_TYPE_EXCLUDE"
}
],
"infoTypes": [
{
"name": "EMAIL_ADDRESS"
},
{
"name": "LOCATION"
},
{
"name": "PERSON_NAME"
}
],
"ruleSet": [
{
"infoTypes": [
{
"name": "LOCATION"
}
],
"rules": [
{
"exclusionRule": {
"excludeInfoTypes": {
"infoTypes": [
{
"name": "CUSTOM_BLOCK"
}
]
},
"matchingType": "MATCHING_TYPE_INVERSE_MATCH"
}
}
]
}
]
},
"deidentifyConfig": {
"infoTypeTransformations": {
"transformations": [
{
"primitiveTransformation": {
"replaceWithInfoTypeConfig": {}
}
}
]
}
}
}
Example output:
"item": {
"value": "{\"postal_code\":\"123ABC\",\"street\":\"Street Name\",\"city\":\"My City\",\"provence\":\"My [LOCATION]\"}"
},
By setting "groupIndexes" to 2 we are indicating that we only want the custom infoType to match the middle (or second) regex group and not allow the :" or " to be part of the match. Also, in this example we mark the custom infoType as EXCLUSION_TYPE_EXCLUDE so that it does not report itself:
"exclusionType": "EXCLUSION_TYPE_EXCLUDE"
If you remove this line, anything matching your infoType could also get redacted. This can be useful for testing though - example output:
"item": {
"value": "{\"postal_code\":\"[CUSTOM_BLOCK]\",\"street\":\"[CUSTOM_BLOCK]\",\"city\":\"[CUSTOM_BLOCK]\",\"provence\":\"[CUSTOM_BLOCK][LOCATION]\"}"
},
...
Hope this helps.
I am new in Java coding, I am trying to post a JSOn file, here is my JSON file object
{
"report": "[{\"patientId\":\"abcd-efg\",\"campaignId\":\"2\",\"phoneCallsMade\":\"[]\",\"message\":\"[]\"}, {\"patientId\":\"abcd-efg\",\"campaignId\":\"2\",\"phoneCallsMade\":\"[]\",\"message\":\"[]\"}]"
}
I am trying to remove the backslash, I tried below methods :
myJsonString.replaceAll("\\","");
myJsonString=myJsonString.replaceAll("\\\\","");
but after doing that the json format is not valid. Can someone help me please.
This is just a string not a json array. And the reason of getting invalid json post removing slashes is due to in correct mapping of double quotes. Just remove double quotes from "[ & ]" from beginning and from end and perform the operation. Your problem will be resolved.
{
"report": [{
"patientId": "abcd-efg",
"campaignId": "2",
"phoneCallsMade": "[]",
"message": "[]"
}, {
"patientId": "abcd-efg",
"campaignId": "2",
"phoneCallsMade": "[]",
"message": "[]"
}]
}
This will be the output.
I have a Java Application, which receives a very large "Javascript Object" (like JSON format) from a server (+30k characters).
Original:
String json ={
ok=true,
data= {
key1=1.0,
key2= [
1.5,
1.5,
1.5
],
key3= [
{
ts=1.526552846892E12,
msg="random_message"
},
{
ts=1.526552846892E12,
msg="random_message"
}
]
}
};
The main problem is that actually, I'm receiving something like this:
String json ={
ok=true,
data= {
key1=1.0,
key2= [
1.5,
1.5,
1.5
],
key3= [
{
ts=1.526552846892E12,
msg=random_message
},
{
ts=1.526552846892E12,
msg=random_message
}
]
}
};
Note, that I "lost" the quotes on msg.
The main problem of this is that I can't parse the JSON with JSONObject for example, cause it can't read this "incorrect" format.
Update:
Ignoring the problem that I'm receiving a "corrupted Object", I have the problem that I can't parse this because it's a JS Object (not pure JSON).
Which is the best way to add double quotes in keys and string values?
Example of formatted JSON:
String json ={
"ok":true,
"data": {
"key1":1.0,
"key2": [
1.5,
1.5,
1.5
],
"key3"= [
{
"ts":1.526552846892E12,
"msg":"random_message"
},
{
"ts":1.526552846892E12,
"msg":"random_message"
}
]
}
};
I have some files records in which are stored as plain text Json. A sample records:
{
"datasetID": "Orders",
"recordID": "rid1",
"recordGroupID":"asdf1",
"recordType":"asdf1",
"recordTimestamp": 100,
"recordPartitionTimestamp": 100,
"recordData":{
"customerID": "cid1",
"marketplaceID": "mid1",
"quantity": 10,
"buyingDate": "1481353448",
"orderID" : "oid1"
}
}
For each record, recordData may be null. If recordData is present, orderID may be null.
I write the following Avro schema to represent the structure:
[{
"namespace":"model",
"name":"OrderRecordData",
"type":"record",
"fields":[
{"name":"marketplaceID","type":"string"},
{"name":"customerID","type":"string"},
{"name":"quantity","type":"long"},
{"name":"buyingDate","type":"string"},
{"name":"orderID","type":["null", "string"]}
]
},
{
"namespace":"model",
"name":"Order",
"type":"record",
"fields":[
{"name":"datasetID","type":"string"},
{"name":"recordID","type":"string"},
{"name":"recordGroupID","type":"string"},
{"name":"recordType","type":"string"},
{"name":"recordTimestamp","type":"long"},
{"name":"recordPartitionTimestamp","type":"long"},
{"name":"recordData","type": ["null", "model.OrderRecordData"]}
]
}]
Ans finally, I use the following method to de-serialize each String record into my Avro class:
Order jsonDecodeToAvro(String inputString) {
return new SpecificDatumReader<Order>(Order.class)
.read(null, DecoderFactory.get().jsonDecoder(Order.SCHEMA$, inputString));
}
But I keep getting the exception when trying to reach the above record:
org.apache.avro.AvroTypeException: Unknown union branch customerID
at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:445)
What am I doing wrong? I am using JDK8 and Avro 1.7.7
The json input must be in the form
{
"datasetID": "Orders",
"recordID": "rid1",
"recordGroupID":"asdf1",
"recordType":"asdf1",
"recordTimestamp": 100,
"recordPartitionTimestamp": 100,
"recordData":{
"model.OrderRecordData" :{
"orderID" : null,
"customerID": "cid1",
"marketplaceID": "mid1",
"quantity": 10,
"buyingDate": "1481353448"
}
}
}
This is because of the way Avro's JSON encoding handles unions and nulls.
Take a look at this:
How to fix Expected start-union. Got VALUE_NUMBER_INT when converting JSON to Avro on the command line?
There is also an open issue regarding this:
https://issues.apache.org/jira/browse/AVRO-1582
``I am having an Json Array as response like:
[
{
"status": "Active",
"entityName": "fghgfhfghfgh",
"entityCode": 14,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "64eab9299eed9455b3683da074cf175c",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7"
}
,
{
"status": "Active",
"entityName": "ghghhguyutgh6re58rrt",
"entityCode": 13,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "912eff0613fa140c100af435c033e195",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7"
}
]
I want to split this json into two like
{
"status": "Active",
"entityName": "fghgfhfghfgh",
"entityCode": 14,
"children": [],
"attributes": [
{
"attributeValue": "500 michigan ave"
}
],
"deviceList": [],
"entityId": "64eab9299eed9455b3683da074cf175c",
"customerId": 2006546,
"type": "7dad308f82b41e02fe8959c05b631bd7" }
and the other one.I am using GSON and simplejson,when I try to remove the delimiters([ and ])the json comes as malformed one.Is there any better otpion to split the json array to two or more json strings as per the json response coming.
Is there any reason you can't parse the entire array, and then iterate over/process each element individually?
Just removing the brackets does make the JSON invalid, and splitting at the comma is going to be unreliable. The parser exists to figure out where to split the array and turn each element into an object for you.
Assuming you have some data structure defined to hold a single element once you've broken the array down, you should be able to parse the array into a list of those and step through them (or pick one out) as needed.
After that point, you can do whatever you want with the data (including formatting it back into JSON). I would definitely recommend using a proper parser to break the array down, though; it will be a lot simpler and more reliable, and should work unless you have serious performance concerns.