We Keep Coding

How to escape single quotes in ARM Template with #concat function

I have template for datafactory pipeline with destination dataset sink, which I want deploy using resource manager and Java Azure SDK:
{
"name": "[concat(parameters('factoryName'), '/', parameters('pipeline_pipelineConfiguration_pipelineTemplate_destinationDataset01'))]",
"type": "Microsoft.DataFactory/factories/datasets",
"apiVersion": "2018-06-01",
"properties": {
"linkedServiceName": {
"referenceName": "[parameters('pipeline_pipelineConfiguration_pipelineTemplate_destinationLinkedService01')]",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "DelimitedText",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"fileName": {
"value": "[concat('#concat(utcnow(\'yyyy-MM-dd\'),\'-',parameters('pipeline_pipelineConfiguration_destination'),',.txt\'')]",
"type": "Expression"
},
"container": ""
},
"columnDelimiter": ",",
"escapeChar": "\\",
"firstRowAsHeader": true,
"quoteChar": "\""
},
"schema": []
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/', parameters('pipeline_pipelineConfiguration_pipelineTemplate_DestinationLinkedService01'))]"
]
}
I get exception:
com.fasterxml.jackson.core.JsonParseException: Unrecognized character escape ''' (code 39)
most probably because of value parameter for fileName.
What would the best way to provide file name with date calculated on time of exporting data and part of the name taken from parameter ?

Use variable and make calculation of value this variable
or use replace() function.

How to create a Sql Statement from a mutable Json File using Java

I'm trying to create SQL Tables from a Json File which is written following the OpenApi Specification. Here is an example of an Input file I must convert:
"definitions": {
"Order": {
"type": "object",
"properties": {
"id": {
"type": "integer",
"format": "int64"
},
"petId": {
"type": "integer",
"format": "int64"
},
"quantity": {
"type": "integer",
"format": "int32"
},
"shipDate": {
"type": "string",
"format": "date-time"
},
"status": {
"type": "string",
"description": "Order Status",
"enum": [
"placed",
"approved",
"delivered"
]
},
"complete": {
"type": "boolean",
"default": false
}
},
"xml": {
"name": "Order"
}
},
"Category": {
"type": "object",
"properties": {
"id": {
"type": "integer",
"format": "int64"
},
"name": {
"type": "string"
}
},
"xml": {
"name": "Category"
}
},
My aim to to create two tables named "Order" and "Category" whose columns must be to ones listed in the "properties" field. I'm using Java.
The Input file is mutable, so I used Gson to read it. I managed to get an Output like this:
CREATE TABLE ORDER
COLUMNS:
id->
type: integer
format: int64
petId->
type: integer
format: int64
quantity->
type: integer
format: int32
shipDate->
type: string
format: date-time
status->
type: string
description: Order Status
Possibilities:
-placed
-approved
-delivered
complete->
type: boolean
default: false
CREATE TABLE CATEGORY
COLUMNS:
id->
type: integer
format: int64
name->
type: string
I'm stuck here, trying to convert the "type" and "format" fields into a type that can be read by PostgreSQL or MySQL. Furthermore, it is hard to work directly on the code to get a readable SQL string due the presence of nesting. So I thought it might be a good idea to work on the output and "translate" it to SQL. Is there any class\package that could help me reading a file like this? I'm trying to avoid the use of thousands IF ELSE conditions. Thank you.

Your assignment involves two phases.
One is "Parsing" the given JSON object and understanding the content
Second one is "Translating" the parsed content into a working SQL query
Here your Java program should work as a kind of Translation engine.
For parsing the JSON objects many java libraries are available.
To translate the parsed json into a SQL query you can simply use basic String manipulation methods.

Java - Optimal way to overwrite values from a subset JSON document to a superset JSON document

I have the below superset JSON and a subset JSON. In my java code, I need to parse the superset JSON and for every element in the propList which has overwrite flag set to true, I need to read the value from subset JSON which has a similar structure as superset JSON and overwrite it.
What is the optimal way to achieve this in java? The JSON document can be quite big.
Superset JSON:
{
"configList": [
{
"configElement": "elem1",
"propList": [
{
"property": "prop1",
"value": "val1",
"overwriteValueFromSubset": false
},
{
"property": "prop2",
"value": "",
"overwriteValueFromSubset": true
},
{
"property": "prop3",
"value": "val3",
"overwriteValueFromSubset": false
}
]
},
{
"configElement": "elem2",
"propList": [
{
"property": "prop1",
"value": "val1",
"overwriteValueFromSubset": false
},
{
"property": "prop2",
"value": "val2",
"overwriteValueFromSubset": false
},
{
"property": "prop3",
"value": "",
"overwriteValueFromSubset": true
}
]
}
]
}
Subset JSON:
{
"configList": [
{
"configElement": "elem1",
"propList": [
{
"property": "prop2",
"value": "new_value",
}
]
},
{
"configElement": "elem2",
"propList": [
{
"property": "prop3",
"value": "new_value",
}
]
}
]
}

Assuming you can fit all data in memory. Google "how to convert Java POJO to JSON object", (and vice versa). So the answer (without actually coding it for you) is convert it into a big Java object (graph), and process the POJOs, then then just call whatever method saves it back out as JSON. The Jackson libraries are what you want here. The object is called a 'mapper'. That should be enough tips to get you going.
https://www.mkyong.com/java/how-to-convert-java-object-to-from-json-jackson/

Parse json response of rest API and delete certain jsonObjects - JAVA

I have a json file as below which I am getting as a response from rest API:
{
"label": " MARA LEYZIN",
"ClassCode": "PROFESSIONAL",
"actvFlg": "A",
"name": "MARA LEYZIN",
"Typ": {
"label": "C_TYP_LU",
"TypCode": "PROFESSIONAL "
},
"Address": {
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item": [
{
"label": "Address",
"addrTypFk": {
"label": "C_ADDRESS_TYPE_LU",
"addrTypCd": "INDUSTRY",
"addrTypDesc": "Industry"
}
}
]
}
I am trying to parse this in Java and to remove some unwanted json objects. Like I want the following string to be replaced by blank:
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item":
To achieve this I am trying the following approach:
String jsonStr = new String(Files.readAllBytes(Paths.get(inputFile)));
System.out.println(jsonStr);
jsonStr.replaceAll("link", "");
But it is not replacing the required string with blanks. Please help me in this.

string object is immutable , so basically if do you want to replace something
System.out.println(jsonStr.replaceAll("link", "")); this will print the replaced string but it will not affect the original string, however if you do this
jsonStr=jsonStr.replaceAll("link", "");
System.out.println(jsonStr); this will print the replaced string

First of all:
Your JSON is not validate. You're missing a closing curly bracket at the end of it.
{
"label": " MARA LEYZIN",
"ClassCode": "PROFESSIONAL",
"actvFlg": "A",
"name": "MARA LEYZIN",
"Typ": {
"label": "C_TYP_LU",
"TypCode": "PROFESSIONAL "
},
"Address": {
"link": [],
"firstRecord": 1,
"pageSize": 10,
"searchToken": "multi",
"item": [{
"label": "Address",
"addrTypFk": {
"label": "C_ADDRESS_TYPE_LU",
"addrTypCd": "INDUSTRY",
"addrTypDesc": "Industry"
}
}]
}
}
Second of all you should just change order of your commands to this:
jsonStr.replaceAll("link", "");
System.out.println(jsonStr);
Important addition:
And I would suggest you to use org.json library or even better JACKSON to parse JSON files.
Here's tutorial how to use jackson and it's my warmest suggestion.
You will save a lot of time and you can do whatever you like.

org.json JsonObject.has() returning true for key that doesn't exist

Edit - Resolved:
The issued turned out to be a combination of our logging code and how json.org handles the distinction between Javascript null and Javascript undefined. Our logging code didn't print null values in objects, so although the json object I saw had no "invoice" field, the actual input had a null "invoice" field. JSONObject's .has() method reported that there was an "invoice" field, but when I tried to access it, it's value was null so it wasn't possible to access it. Replacing the .has() check with a .isNull() check (which considers both absent fields and null values) resolved the problem.
In my Google App Engine application, we parse webhook data in the form of json and go down different code paths depending on the presence of a certain key in the json object - if $.data.object.invoice exists we do some work, otherwise we don't. However, when running in production, this detection seems to be broken. I do a simple
jsonObject.getJSONObject("data").getJSONObject("object").has("invoice")
to check for the invoice key and sometimes this returns true even when the invoice key doesn't exist. I came across this error because we do make use of the invoice value when it exists, and I was getting JSONExceptions trying to access it, even though it was protected by a has() check. I added the following logging to ensure that I was doing everything correct:
boolean isAutomaticCharge = jsonObject.getJSONObject("data").getJSONObject("object").has("invoice");
boolean doesGettingStringWork;
try {
jsonObject.getJSONObject("data").getJSONObject("object").getString("invoice");
doesGettingStringWork = true;
}
catch (JSONException e) {
doesGettingStringWork = false;
}
Logger.log("The value of automatic charge is: " + isAutomaticCharge);
Logger.log("Geting the invoice worked? " + doesGettingStringWork);
Logger.log(jsonObject.getJSONObject("data").getJSONObject("object").toString());
if (isAutomaticCharge) {
handleFailedCharge(jsonObject);
}
else {
//Manual Charge
}
And in production the following data resulted in isAutomaticCharge being true and doesGettingStringWork being false:
{
"id": "evt_16vC0s1WJGsEk3Qvnstmq6fa",
"object": "event",
"api_version": "2014-03-28",
"created": 1444678274,
"data": {
"object": {
"id": "ch_16vC0r1WJGsEk3Qv5rxezNQj",
"object": "charge",
"amount": 1900,
"amount_refunded": 0,
"captured": false,
"card": {
"id": "card_16vC0k1WJGsEk3Qvl9LLEt5K",
"object": "card",
"brand": "MasterCard",
"country": "CA",
"customer": "cus_79V0ZH6qFFA4K0",
"cvc_check": "fail",
"exp_month": 9,
"exp_year": 2016,
"fingerprint": "ifxs23sixYzpKant",
"funding": "credit",
"last4": "6912",
"metadata": {},
"name": "derek#footbole.com",
"type": "MasterCard"
},
"created": 1444678273,
"currency": "usd",
"customer": "cus_79V0ZH6qFFA4K0",
"failure_code": "card_declined",
"failure_message": "Your card was declined.",
"fraud_details": {},
"livemode": true,
"metadata": {},
"paid": false,
"refunded": false,
"refunds": [],
"source": {
"id": "card_16vC0k1WJGsEk3Qvl9LLEt5K",
"object": "card",
"brand": "MasterCard",
"country": "CA",
"customer": "cus_79V0ZH6qFFA4K0",
"cvc_check": "fail",
"exp_month": 9,
"exp_year": 2016,
"fingerprint": "ifxs23sixYzpKant",
"funding": "credit",
"last4": "6912",
"metadata": {},
"name": "derek#footbole.com",
"type": "MasterCard"
},
"status": "failed"
}
},
"livemode": true,
"pending_webhooks": 2,
"request": "req_79V0v7433eb9hZ",
"type": "charge.failed"
}
When I run the code locally and feed it that json, it works as expected with isAutomaticCharge and doesGettingStringWork both being false.
I'm running version 20140107 of org.json. I declare a new JSONObject for every request, so threading shouldn't be an issue. Has anyone else had issues running org.json on Google App Engine?