We Keep Coding

Elasticsearch nested sort - mismatch between document and nested object used for sorting

I've been developing a new search API with AWS Elasticsearch (version 6.2) as backend.
Right now, I'm trying to support "sort" options for the API.
My mapping is as follows (unrelated fields not included):
{
"properties": {
"id": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"description": {
"type": "text"
},
"materialDefinitionProperties": {
"type": "nested",
"properties": {
"id": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"analyzer": "case_sensitive_analyzer"
},
"value" : {
"type": "nested",
"properties": {
"valueString": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
}
}
I'm attempting to allow the users sort by property value (path: materialDefinitionProperties.value.valueLong.raw).
Note that it's inside 2 levels of nested objects (materialDefinitionProperties and materialDefinitionProperties.value are nested objects).
To sort the results by the value of property with ID "PART NUMBER", my request for sorting is:
{
"fieldName": "materialDefinitionProperties.value.valueString.raw",
"nestedSort": {
"path": "materialDefinitionProperties",
"filter": {
"fieldName": "materialDefinitionProperties.id",
"value": "PART NUMBER",
"slop": 0,
"boost": 1
},
"nestedSort": {
"path": "materialDefinitionProperties.value"
}
},
"order": "ASC"
}
However, as I examined the response, the "sort" field does not match with document's property value:
{
"_index": "material-definition-index-v2",
"_type": "default",
"_id": "development_LITL4ZCNE",
"_source": {
"id": "LITL4ZCNE",
"description": [
"CPU, Intel, Cascade Lake, 8259CL, 24C, 210W, B1 Prod"
]
"materialDefinitionProperties": [
{
"id": "PART NUMBER",
"description": [],
"value": [
{
"valueString": "202-001193-001",
"isOriginal": true
}
]
}
]
},
"sort": [
"100-000018"
]
},
The document's PART NUMBER property is "202-001193-001", the "sort" field says "100-000018", which is the part number of another document.
It seems that there's a mismatch between the master document and nested object used for sorting.
This request worked well when there's only a small number of documents in the cluster. But once I backfill the cluster with ~1 million of records, the symptom appears. I've also tried creating a new ES cluster but the results are the same.
Sorting by other non-nested attributes worked well.
Did I misunderstand the concept of nested objects, or misuse the nested sort feature?
Any ideas appreciated!

This is a bug in Elasticsearch. Upgrading to 6.4.0 fixed the issue.
Issue tracker: https://github.com/elastic/elasticsearch/pull/32204
Release note: https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-6.4.0.html

Pojo/Entity corresponding to json response from Azure REST API

I have written a java class which hits the Azure REST API's and gets me the json response. for example below-
{
"value": [
{
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"kind": "Storage",
"id": "/subscriptions/cc60365e-3193-xxxx-b0f6-xxxxxxx/resourceGroups/WebServer-ubuntu/providers/Microsoft.Storage/storageAccounts/webserverubuntudiagxxx",
"name": "webserverubuntudiag4xx",
"type": "Microsoft.Storage/storageAccounts",
"location": "eastus",
"tags": {},
"properties": {
"networkAcls": {
"bypass": "AzureServices",
"virtualNetworkRules": [],
"ipRules": [],
"defaultAction": "Allow"
},
"trustedDirectories": [
"50228742-xxx-4cd2-xxx-3e50b7a62xx1"
],
"supportsHttpsTrafficOnly": false,
"encryption": {
"services": {
"file": {
"enabled": true,
"lastEnabledTime": "2018-03-26T15:02:26.8078850Z"
},
"blob": {
"enabled": true,
"lastEnabledTime": "2018-03-26T15:02:26.8078850Z"
}
},
"keySource": "Microsoft.Storage"
},
"provisioningState": "Succeeded",
"creationTime": "2018-03-26T15:02:26.6359768Z",
"primaryEndpoints": {
"blob": "https://webserverubuntudiagxx.blob.core.windows.net/",
"queue": "https://webserverubuntudiag4xx.queue.core.windows.net/",
"table": "https://webserverubuntudiagxx.table.core.windows.net/",
"file": "https://webserverubuntudiag4xx.file.core.windows.net/"
},
"primaryLocation": "eastus",
"statusOfPrimary": "available"
}
}
]
}
My questions is: Are there any model/pojo/bean already available to map this json to java object ?
I need to map the data from above json into the java beans/pojos to display into jsp pages, do I need to separately create custom pojos or there are any already in Azure libraries/sdk ? any suggestions?

swagger 2.x reference model from other file

I want to create an API with following spec.:
project-api.json
{
"Project": {
"required": [
"uuid"
],
"properties": {
"uuid": {
"type": "string",
"format": "string"
}
"Tasks": {
"type": "array",
"items": {
"$ref": "models/task.json"
}
}
}
...
task.json
{
"Task": {
"required": [
"uuid"
],
"properties": {
"user": {
"type": "string"
},
"title": {
"type": "string"
},
...
Instead of making a reference to models/tasks.json, the generated api generates:
List<ModelstasksJson>
Anyone know how to reference to models from another json file?

generating POJOs from JSON Schema for non-object types

I am trying to generate POJOs from the JSON Schema of XMBC.
I do this with jsonschema2pojo.
However, nothing gets generated. It doesn't even bring me an error.
This is a reduced sample json schema I am trying to generate from:
{
"description": "JSON-RPC API of XBMC",
"id": "http://xbmc.org/jsonrpc/ServiceDescription.json",
"methods": {
"Addons.ExecuteAddon": {
"description": "Executes the given addon with the given parameters (if possible)",
"params": [
{
"name": "addonid",
"required": true,
"type": "string"
},
{
"default": "",
"name": "params",
"type": [
{
"additionalProperties": {
"default": "",
"type": "string"
},
"type": "object"
},
{
"items": {
"type": "string"
},
"type": "array"
},
{
"description": "URL path (must start with / or ?",
"type": "string"
}
]
},
{
"default": false,
"name": "wait",
"type": "boolean"
}
],
"returns": {
"type": "string"
},
"type": "method"
}
},
"notifications": {
"Application.OnVolumeChanged": {
"description": "The volume of the application has changed.",
"params": [
{
"name": "sender",
"required": true,
"type": "string"
},
{
"name": "data",
"properties": {
"muted": {
"required": true,
"type": "boolean"
},
"volume": {
"maximum": 100,
"minimum": 0,
"required": true,
"type": "integer"
}
},
"required": true,
"type": "object"
}
],
"returns": null,
"type": "notification"
}
},
"types": {
"Addon.Content": {
"default": "unknown",
"enums": [
"unknown",
"video",
"audio",
"image",
"executable"
],
"id": "Addon.Content",
"type": "string"
}
},
"version": "6.14.3"
}
I must admin that my knowledge of JSON is very terse, maybe it is just a simple fault of mine. But can anyone help me how I can generate Java objects from such a JSON Schema?

The JSON Schema doesn't support method. JSON schema defines json data structure, it would not be used to define your methods. Most important attribute in JSON schema is properties.
It's good to generate POJO data models from a JSON schema, but not business logic. You can learn the JSON schema from those examples.

Jackson Parser for recursively parsing unknown input structure

I'm trying to parse recursively json input structure in java like the format below and trying to rewrite the same structure in another json.
Meanwhile I need to validate each & every json key/values while parsing.
{"Verbs":[{
"aaaa":"30d", "type":"ed", "rel":1.0, "id":"80", "spoken":"en", "ct":"on", "sps":null
},{
"aaaa":"31", "type":"cc", "rel":3.0, "id":"10", "spoken":"en", "ct":"off", "sps":null
},{
"aaaa":"81", "type":"nn", "rel":3.0, "id":"60", "spoken":"en", "ct":"on", "sps":null
}]}
Please advice how I can use Jackson parser JsonToken enums for reading and writing unknown json content.

You can use JSON Schema to validate your inputs.
Find the documentation for the data format, but from what I can read here, the schema would be something like this:
{
"$schema": "http://json-schema.org/schema#",
"type": "object",
"required": [ "Verbs" ],
"properties": {
"Verbs": { "type": "array", "items": { "$ref": "#/definitions/verb" } }
},
"definitions": {
"verb": {
"type": "object",
"required": [ "aaaa", "type", "rel", "id", "spoken", "ct", "sps" ],
"additionalProperties": false,
"properties": {
"aaaa": { "type": "string" },
"type": { "type": "string" },
"rel": { "type": "number" },
"id": { "type": "string", "pattern": "^[0-9]+$" },
"spoken": { "type": "string" },
"ct": { "enum": [ "on", "off" ] },
"sps": { "enum": [ null ] }
}
}
}
}
As you use Jackson, you can use this library which can validate your data for you.
Transforming your JSON after that can be done by creating a new JsonNode, for instance.