converting values between to data domains - java

I have to write over a hundred integrations between many systems. This integration layer must be able to convert codes. Each system uses codes to represent business types like insurance_type, customer_type, etc. Each of them has a set of valid values. Theses values are not the same from system to system, and may even vary over time.
I start looking for data domain mapping libraries in Java. I didn't found anything suitable. I thought about: CloverETL,Pentaho ETLou GETL​ but they are all way too complex for my need or not maintain.
The goal is to put the conversion rules out of the code so they could evolve over time without the need for a new executable deployment.
I'm looking for a tool, library that would allow me to represent mapping similar to this:
{
"domains" :[
{
"name": "type police host",
"values": [
{
"code" : "0001",
"description":"Habitation",
"start_date":"2019-06-30",
"end_date":""},
{
"code" : "0002",
"description":"Automobile",
"start_date":"2019-06-30",
"end_date":""}
]
},
{
"name": "type police web",
"values": [
{
"code" : "Habitation",
"description":"Habitation",
"start_date":"2019-06-30",
"end_date":""}
]
}
],
"conversions" : [
{
"from": "type police host",
"to": "type police web",
"rules" : [
{
"from": ["0001"],
"to" : "Habitation",
"start_date":"2019-06-30",
"end_date":""},
{
"from": [ "0003","0004"],
"to" : "Deux roues",
"start_date":"2019-06-30",
"end_date":""}
]
}
]
}
From the configuration file above, I would be able to do things like convertsAsOf("2019-07-10", "type police host", "type police web", "0001") and it would return "Habitation". Any suggestion of a library that would do it?

Related

JSON Schema - Enum of Objects

I'm new to JSON schema, so bear with me. My goal is to have a JSON property that is an object. It's keys relate to each other, meaning multiple keys always have the same values together. This will probably help make it clear, it's my attempt to do this with an enum:
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"title": "Part",
"type": "object",
"properties": {
"relationship": {
"type": "object",
"enum": [
{
"code": "1",
"value": "MEMBER"
},
{
"code": "2",
"value": "SPOUSE"
},
{
"code": "3",
"value": "CHILD"
},
{
"code": "4",
"value": "STUDENT"
},
{
"code": "5",
"value": "DISABILITY_DEPENDENT"
},
{
"code": "6",
"value": "ADULT_DEPENDENT"
},
{
"code": "8",
"value": "DOMESTIC_PARTNER"
}
]
}
}
}
So using an enum like this works, even though I can't find it anywhere in the JSON Schema spec. However, the error message sucks. Normally I get the most extremely detailed error messages from schema validation, however in this case I do not.
$.part.relationship: does not have a value in the enumeration [, , , , , , ]
I'm not sure what I'm doing wrong. I'm using a Java parser for JSON Schema:
<dependency>
<groupId>com.networknt</groupId>
<artifactId>json-schema-validator</artifactId>
<version>1.0.53</version>
</dependency>
Not sure if the error message is the fault of the parser or something I'm doing bad with the schema. Help would be appreciated.
It was news to me, but according to the spec it does seem that objects are valid enum values. That said, your usage is quite unusual. I've not seen it used before.
the six primitive types ("null", "boolean", "object", "array", "number", or "string")
...
6.1.2. enum
...
Elements in the array might be of any type, including null.
Your problem is fundamentally that the library that you're using doesn't know how to convert those objects to printable strings. Even if it did give it a reasonable go, you might end up with
does not have a value in the enumeration [{"code": "1", "value":"MEMBER"}, {"code": "2" ...
which might be okay, but it's hardly amazing. If the code and value were both valid but didn't match, you might have to look quite closely at the list before you ever saw the problem.
JSON Schema in general is not very good at enforcing constraints between what it considers to be 2 unrelated fields. That's beyond the scope of it what it aims to do. It's trying to validate the structure. Dependencies between fields are business constraints, not structural ones.
I think the best thing you could do to achieve readable error messages would be to have 2 sub-properties, each with an enumeration containing 8 values; one for the codes, one for the values.
Then you'll get
$.part.relationship.code does not have a value in the enumeration [1,2,3,4 ...
or
$.part.relationship.value does not have a value in the enumeration ["MEMBER", "SPOUSE", ...
You can do some additional business validation on top of the schema validation if enforcing that constraint is important to you. Then generate your own error such as
code "1" does not match value "SPOUSE"
If code and value always have the same values relative to each other, why encode both in the JSON? Just encode a single value in the JSON and infer the other in the application.
This will be much easier to validate.

fhir.executeBundle replacing resource id...How to prevent this?

I am using this Java code to upload a resource to a FHIRstore.
The resource is as follows
{
"resourceType": "Bundle",
"id": "bundle-transaction",
"meta": {
"lastUpdated": "2018-03-11T11:22:16Z"
},
"type": "transaction",
"entry": [
{
"resource": {
"resourceType": "Patient", "id" : 123456,
"name": [
{
"family": "Smith",
"given": [
"Darcy"
]
}
],
"gender": "female",
"address": [
{
"line": [
"123 Main St."
],
"city": "Anycity",
"state": "CA",
"postalCode": "12345"
}
]
},
"request": {
"method": "POST",
"url": "Patient"
}
}
]
}
But the id i am using(123456) is getting replaced by a hexadecimal number.
This does not happen while using fhirstores.import method
Is there any way to stop executeBundle method from replacing my id...as i want to use custom id in my resource?
Any help would be appreciated.
Thank you
When you're performing a transaction, the effect is going to be the same as if you were POSTing the resources individually. On a POST, the server determines the resource id. On a regular POST, the id is just ignored or raises an error. Within a transaction, the id is used to manage resolution of references across the transaction, but the server still chooses what the id will be of the persisted resources (and updates all references accordingly). If you want to control the resource id values within a transaction, use PUT rather than POST. (Note that not all servers will allow an 'upsert' - i.e. a PUT that performs a create at a specific resource location.) For details, see http://hl7.org/fhir/http.html#upsert.

Google maps API without the map

I want to build an app with simple gui that include: "Start Point" and "End Point". The purpose of the app is to out to the screen how many meters/kilometers the rout that google maps choose, without to present the map itself. How can I do that? All what I found it's with map graphics.
You can use Distance Matrix API:
The Distance Matrix API is a service that provides travel distance and
time for a matrix of origins and destinations. The API returns
information based on the recommended route between start and end
points, as calculated by the Google Maps API, and consists of rows
containing duration and distance values for each pair.
In that case you need HttpUrlConnection or something like that for send URL request like:
https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=Washington,DC&destinations=New+York+City,NY&key=YOUR_API_KEY
and parse JSON (or XML) response like that:
{
"destination_addresses" : [ "New York, NY, USA" ],
"origin_addresses" : [ "Washington, DC, USA" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "225 mi",
"value" : 361715
},
"duration" : {
"text" : "3 hours 49 mins",
"value" : 13725
},
"status" : "OK"
}
]
}
],
"status" : "OK"
}
NB! don't forget to get API key.
Also take a look at Official and other e.g. like that tutorials and examples.

JSON parsing vs Regex based String parsing

I need to process a big JSON payload(~1MB) coming from an API, a portion of the JSON is something like this:
{
"id": "013dd2a7-fec4-4cc5-b819-f3cf16a1f820",
//more attributes
"entry_mode": "LDE",
"periods": [
{
"type": "quarter",
"id": "fe96dc03-660c-423c-84cc-e6ae535edd2d",
"number": 1,
"sequence": 1,
"scoring": {
//more attribtues
},
"events": [
{
"id": "e4426708-fadc-4cae-9adc-b7f170f5d607",
"clock": "12:00",
"updated": "2013-12-22T03:41:40+00:00",
"description": "J.J. Hickson vs. DeAndre Jordan (Blake Griffin gains possession)",
"event_type": "opentip",
"attribution": {
"name": "Clippers",
"market": "Los Angeles",
"id": "583ecdfb-fb46-11e1-82cb-f4ce4684ea4c",
"team_basket": "left"
},
"location": {
"coord_x": 572,
"coord_y": 296
},
"possession": {
"name": "Clippers",
"market": "Los Angeles",
"id": "583ecdfb-fb46-11e1-82cb-f4ce4684ea4c"
}
},
//more events
]
}
]
}
This is a nearly-realtime API that I need to process only the events, identify a set of event UUIDs, look for duplicates in the database and save new events.
I could use a JSONObject/JSONArray or use regex with string parsing to and fetch the events portion. Processing time is critical since this should be nearly-realtime and memory efficiency is important since there can be multiple payloads coming in at once.
Which one is more efficient for this use case?
Use a proper streaming JSON parser. You know what you want to pull out of the stream, you know when you can quit parsing it, so read the stream in small, manageable chunks, and quit as soon as you know you are done.
Circa 2017, I'm not aware of any browser/native JSON streaming APIs, so you'll need to find a Javascript-based streaming library. Fortunately, streaming is not a new concept, so there are a number of options already in existence:
http://oboejs.com/
https://github.com/dominictarr/JSONStream
https://github.com/creationix/jsonparse
https://github.com/dscape/clarinet
http://danieltao.com/lazy.js/demos/json/

How to specify that a JSON instance is defined by a specific JSON Schema

My question is how can I know which JSON schema to use to validate a particular JSON document? I specified the URL to the schema in the id field from it. Is this enough? Should I put the id in the JSON? I am not sure I understand how to connect a JSON to a specific JSON Schema.
Here is my schema
{
"$schema": "http://json-schema.org/draft-04/schema#",
"id": "url/schema.json",
"title": "title",
"definitions": {
"emailObject": {
"type": "object",
"properties":{
"name": {
"description": "The name of the customer",
"type": "string",
"maxLength": 200
},
"email": {
"description": "The email of the customer",
"type": "string",
"format": "email",
"maxLength": 100
}
}
}
}
To add to and clarify Tom's answer, here is an example of how you can link a JSON document to a JSON Schema. There is no standard way of doing this outside the context of an HTTP response. If that is something you need, you will have to come up with your own strategy.
GET /data/my-data HTTP/1.1
HTTP/1.1 200 OK
Content-Type: application/json
Link: </schema/my-schema> rel=describedby
{ "name": "Fake Fakerson", "email": "fake#fakerson.com" }
GET /schema/my-schema HTTP/1.1
HTTP/1.1 200 OK
Content-Type: application/schema+json
{
"$schema": "http://json-schema.org/draft-04/schema#",
"id": "url/schema.json",
"title": "title",
"definitions": {
"emailObject": {
"type": "object",
"properties":{
"name": {
"description": "The name of the customer",
"type": "string",
"maxLength": 200
},
"email": {
"description": "The email of the customer",
"type": "string",
"format": "email",
"maxLength": 100
}
}
}
}
}
According to 10.1 of the specification:
It is RECOMMENDED that instances described by a schema/profile provide
a link to a downloadable JSON Schema using the link relation
"describedby", as defined by Linked Data Protocol 1.0, section 8.1
[W3C.REC-ldp-20150226]. (emphasis mine)
This would appear to describe exactly the behaviour you require, however, a casual perusal of the Linked Data Protocol section 8.1 leaves us none the wiser:
The relationship A describedby B asserts that resource B provides a
description of resource A. There are no constraints on the format or
representation of either A or B, neither are there any further
constraints on either resource (emphasis mine)
After a quick google search, I found this question, which at first glance would appear to be duplicated by your question. However, upon deeper inspection, the question is actually about inheritance within schemas, not the referencing of a schema from it's supported instances.
One of the answers, rather intriguingly, provides a solution which draws on the JSON-Hyper-schema standard - an attempt to extend the JSON-schema standard to support the definition of application-level semantics.
The way it achieves this is by use of the links collection:
{
...
"links":[
{
"rel":"describedby",
"href":"{+fileType}"
}
]
}
It turns out that this is based on another standard RFC5988 - Web Linking which happens to be the same standard which allows us to load CSS into HTML pages.
As #Jason points out in his comment -
Your first quote, the one from the spec, is the right way to do it.
The linked data definition of describedby does not contradict the JSON
Schema spec. It's a purposefully broad definition so it can be applied
to any media type that describes data. That includes JSON Schema, XML
Schema, or anything else.
So, it would appear that including a links collection in your schema instance would be the correct way to reference the schema. So in your specific case, you could do this:
{
...
"links":[
{
"rel":"describedby",
"href":"url/schema.json" // I assume!!
}
]
}
Even though this may be correct, I don't know how many JSON parsers will respect this when resolving to an actual schema via the link.

Categories