We Keep Coding

Handling Array Indexes When Searching In ElasticSearch

I have a problem with my ES queries where they fail because of having array indexes in their query strings. This happens because my following approach.
I flatten the JSON requests that I get with the following method.
private void flattenJsonRequestToMap(String currentPath, JsonNode jsonNode, Map<String, Object> map) {
if (jsonNode == null || jsonNode.isNull()) {
map.remove(currentPath);
} else if (jsonNode.isObject()) {
ObjectNode objectNode = (ObjectNode) jsonNode;
Iterator<Map.Entry<String, JsonNode>> iter = objectNode.fields();
String pathPrefix = currentPath.isEmpty() ? "" : currentPath + ".";
while (iter.hasNext()) {
Map.Entry<String, JsonNode> entry = iter.next();
flattenJsonRequestToMap(pathPrefix + entry.getKey(), entry.getValue(), map);
}
} else if (jsonNode.isArray()) {
ArrayNode arrayNode = (ArrayNode) jsonNode;
for (int i = 0; i < arrayNode.size(); i++) {
flattenJsonRequestToMap(currentPath + "[" + i + "]", arrayNode.get(i), map);
}
} else if (jsonNode.isValueNode()) {
ValueNode valueNode = (ValueNode) jsonNode;
map.put(currentPath, valueNode.asText());
} else {
LOGGER.error("JSONNNode unexpected field found during the flattening of JSON request" + jsonNode.asText());
}
}
When the Json requests have lists in them, my flattened map looks like below.
myUserGuid -> user_testuser34_ibzwlm
numberOfOpenings -> 1
managerUserGuids[0] -> test-userYspgF1_S3P6s
accessCategories[0] -> RESTRICTED
employeeUserGuid -> user_user33_m1minh
Now I construct ES Query with the following method using the above map.
public SearchResponse searchForExactDocument(final String indexName, final Map<String, Object> queryMap)
throws IOException {
BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
queryMap.forEach((name, value) -> {
queryBuilder.must(QueryBuilders.matchPhraseQuery(name, value));
LOGGER.info("QueryMap key: {} and value: {} ", name, value);
});
return this.executeSearch(indexName, queryBuilder);
}
As you can already see, it ends up executing the query below, with the array indexes in them. My mapping structure is as follows.
{
name=job,
type=_doc,
mappingData={
properties={
myUserGuid ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
numberOfOpenings ={
type=long
},
numOfUsage={
type=long
},
accessCategories ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
managerUserGuids ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
},
employeeUserGuid ={
type=text,
fields={
keyword={
ignore_above=256,
type=keyword
}
}
}
}
}
Because of the appended array index next to the name, the queries don't return any search results. How can I navigate this issue? One option I see is removing the array index using flattening the map, however I need to be able to construct a POJO object which has list for those fields in concern, using the flattened map. Would appreciate any advice/suggestions. Thanks a lot in advance.

Lists in ES are processed like just having few values for one field so if you have "accessCategories": ["foo", "bar"] this doc will match both "accessCategories": "foo" and "accessCategories": "bar" though there is no way to make a query which would match only one ("foo" but not "bar") with this data schema.
If you need to address specific items, you can unwrap list into separate fields accessCategories_0, accessCategories_1, etc. though there is a limit in Elasticsearch for total number of fields in one index.

Use parallel stream in array list with original order

I have a product, I wanna populate products in another array with the same original order, I used parallel Stream and the result was not ordered with the original order
List<Product> products = productList.getProducts();
List<ProductModelDTOV2> productModelDTOV2s = new ArrayList<>();
products.parallelStream().forEach(p -> {
try {
ProductModelDTOV2 ProductModelDTOV2 = dtoFactoryV2.populate(p, summary);
productModelDTOV2s.add(ProductModelDTOV2);
} catch (GenericException e) {
log.debug(String.format("Unable to populate Product %s", p));
}
});
return productModelDTOV2s;

It seems like this part of the code can be unordered and be run in parallel:
ProductModelDTOV2 ProductModelDTOV2 = dtoFactoryV2.populate(p, summary);
But this part must be ordered:
productModelDTOV2s.add(ProductModelDTOV2);
What you can do is to separate those two things. Do the first part in a flatMap, and the second part in forEachOrdered:
products.parallelStream().flatMap(o -> { // this block will be done in parallel
try {
return Stream.of(dtoFactoryV2.populate(p, summary));
} catch (GenericException e) {
// don't expect this message to be printed in order
log.debug(String.format("Unable to populate Product %s", p));
return Stream.of();
}
})
.forEachOrdered(productModelDTOV2s::add); // this will be done in order, non-parallel

The correct way to do this, would be to have the Stream create the list:
List<Product> products = productList.getProducts();
return products.parallelStream()
.map(p -> {
try {
return dtoFactoryV2.populate(p, summary);
} catch (GenericException e) {
log.debug("Unable to populate Product " + p);
return null;
}
})
.filter(Objects::nonNull)
.collect(Collectors.toList());

Observable not returning value as printed out

I've been trying to get an Observable to return back a Map<String,JSONObject>, but it fails to do so. The print statements show the correct return value, but not sure why this is returning an incomplete value.
#Override
#Retryable(value = {CouchbaseException.class})
public Mono<Map<String, JsonObject>> query(String query, List<String> params) {
N1qlParams n1qlParams = N1qlParams.build()
.adhoc(true);
Map<String, JsonObject> resultMap = new HashMap();
N1qlQuery n1qlQuery = N1qlQuery.parameterized(query, JsonArray.from(params), n1qlParams);
Observable<AsyncN1qlQueryResult> queryResult = bucket.async().query(n1qlQuery)
.timeout(5000, TimeUnit.MILLISECONDS)
.retryWhen(anyOf(BackpressureException.class).max(3)
.delay(Delay.exponential(TimeUnit.MILLISECONDS, 32, 2)).build())
.retryWhen(anyOf(TemporaryFailureException.class)
.max(2)
.delay(Delay.fixed(500, TimeUnit.MILLISECONDS)).build())
.onErrorResumeNext(exp -> {
return Observable.error(exp);
});
Observable<Map<String, JsonObject>> metricsMap = queryResult.flatMap(qr ->
qr.info()).map(n1qlMetrics -> {
resultMap.put("metrics", n1qlMetrics.asJsonObject());
log.info("METRICS: " + resultMap.toString());
return resultMap;
});
Observable<Map<String, JsonObject>> resultObsMap = queryResult.flatMap(asyncN1qlQueryResult ->
asyncN1qlQueryResult.errors().flatMap(error -> {
return Observable
.error(new Exception("unable to execute n1ql query " + error.toString()));
})
.switchIfEmpty(asyncN1qlQueryResult.rows())
).map(row -> {
JsonObject json = ((AsyncN1qlQueryRow) row).value();
return json;
}).map(results -> {
resultMap.put("results", results);
log.info("RESULTS: " + resultMap.toString());
return resultMap;
});
// log.info("resultObsMap: " + resultObsMap..toString());
//
Observable<Map<String, JsonObject>> retObservable = Observable
.zip(resultObsMap, metricsMap, (r, m) -> {
r.putAll(m);
return r;
});
//retObservable.subscribe(System.out::println);
return Mono.from(RxReactiveStreams.toPublisher(retObservable));
}
The 2 print statements print the below CORRECT values:
2020-07-30 09:41:19.402 INFO [cb-computations-4] com.aexp.eimp.dbtestsuite.couchbase.service.impl.CouchbaseAsyncServiceImpl - METRICS: {metrics={"executionTime":"780.964µs","resultCount":2,"resultSize":2858,"elapsedTime":"841.098µs"}}
2020-07-30 09:41:19.406 INFO [cb-computations-3]
com.aexp.eimp.dbtestsuite.couchbase.service.impl.CouchbaseAsyncServiceImpl - RESULTS: {metrics={"executionTime":"780.964µs","resultCount":2,"resultSize":2858,"elapsedTime":"841.098µs"}, results={"bucket_admin":{"name":"${__RandomString(20,abcdefghijklmnopqrstuvwxyz,)}","notificationEnrollment":{"insertTS":1503530927070,"id":"0000255BAIHSDCY","type":"ENCRYPTEDACCOUNT","issuer":"001","domain":"","apps":{"app":{"enrollmentInfoList":[{"features":[{"featureUserIdentifier":{"id":"BAIHSDCY0000255","type":"ENCRYPTEDACCOUNT","issuer":"001","domain":"CARD"},"featureInfoList":[{"featureName":"PAYMENT_RECEIVED_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"PAYMENT_REMINDER_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"STMT_NOTIFICATION_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"OFFERS","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"GOLDEN_GOOSE_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true}]}],"appVersion":"1.0","insertTS":1503530927066,"type":"Push","value":true,"deviceInfo":{"deviceType":"deviceType","deviceModel":"deviceModel","osVersion":"OSVersion","deviceId":"17BBE19B241C219C3FE766FAE1887CE2","deviceOS":"deviceOS","deviceToken":"0838273194687468409056870471453029636922734387856368468017913851"},"status":"ACTIVE"}],"applicationGroup":"INTERNAL__10","id":"com.americanexpress.mobile.wallet.carbon.notifications.messages"}}},"id":"100173"}}}
But the return value is incorrect and does not make sense - you can see here that the values in the JSON fields are missing, and the whole response seems to be incomplete.
{"metrics":{"cryptoManager":null,"empty":false,"names":["executionTime","resultCount","resultSize","elapsedTime"]},"results":{"cryptoManager":null,"empty":false,"names":["bucket_admin"]}}
We're expecting
{metrics:{"executionTime":"780.964µs","resultCount":2,"resultSize":2858,"elapsedTime":"841.098µs"}, results:{"bucket_admin":{"name":"${__RandomString(20,abcdefghijklmnopqrstuvwxyz,)}","notificationEnrollment":{"insertTS":1503530927070,"id":"0000255BAIHSDCY","type":"ENCRYPTEDACCOUNT","issuer":"001","domain":"","apps":{"app":{"enrollmentInfoList":[{"features":[{"featureUserIdentifier":{"id":"BAIHSDCY0000255","type":"ENCRYPTEDACCOUNT","issuer":"001","domain":"CARD"},"featureInfoList":[{"featureName":"PAYMENT_RECEIVED_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"PAYMENT_REMINDER_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"STMT_NOTIFICATION_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"OFFERS","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true},{"featureName":"GOLDEN_GOOSE_ALERT","attributes":[{"value":"8","key":"daysBeforeDue"}],"featurePreferenceValue":true}]}],"appVersion":"1.0","insertTS":1503530927066,"type":"Push","value":true,"deviceInfo":{"deviceType":"deviceType","deviceModel":"deviceModel","osVersion":"OSVersion","deviceId":"17BBE19B241C219C3FE766FAE1887CE2","deviceOS":"deviceOS","deviceToken":"0838273194687468409056870471453029636922734387856368468017913851"},"status":"ACTIVE"}],"applicationGroup":"INTERNAL__10","id":"com.americanexpress.mobile.wallet.carbon.notifications.messages"}}},"id":"100173"}}}
Any idea what I am doing incorrect here?

groupByTo return emptySet in Kotlin

I have string like this.
val input = "perm1|0,perm2|2,perm2|1"
Desired output type is
val output: Set<String, Set<Long>>
and desired output value is
{perm1 [], perm2 [1,2] }
Here I need empty set if value is 0. I am using groupByTo like this
val output = input.split(",")
.map { it.split("|") }
.groupByTo(
mutableMapOf(),
keySelector = { it[0] },
valueTransform = { it[1].toLong() }
)
However the output structure is like this
MutableMap<String, MutableList<Long>>
and output is
{perm1 [0], perm2 [1,2] }
I am looking for best way to get desired output without using imperative style like this.
val output = mutableMapOf<String, Set<Long>>()
input.split(",").forEach {
val t = it.split("|")
if (t[1].contentEquals("0")) {
output[t[0]] = mutableSetOf()
}
if (output.containsKey(t[0]) && !t[1].contentEquals("0")) {
output[t[0]] = output[t[0]]!! + t[1].toLong()
}
if (!output.containsKey(t[0]) && !t[1].contentEquals("0")) {
output[t[0]] = mutableSetOf()
output[t[0]] = output[t[0]]!! + t[1].toLong()
}
}

You can simply use mapValues to convert values type from List<Long> to Set<Long>
var res : Map<String, Set<Long>> = input.split(",")
.map { it.split("|") }
.groupBy( {it[0]}, {it[1].toLong()} )
.mapValues { it.value.toSet() }
And of you want to replace list of 0 with empty set you can do it using if-expression
var res : Map<String, Set<Long>> = input.split(",")
.map { it.split("|") }
.groupBy( {it[0]}, {it[1].toLong()} )
.mapValues { if(it.value == listOf<Long>(0)) setOf() else it.value.toSet() }

Note that you cannot have Set with key-value pair, result will be of type map. Below code gives sorted set in the values.
val result = "perm1|0,perm2|2,perm2|1".split(",")
.map {
val split = it.split("|")
split[0] to split[1].toLong()
}.groupBy({ it.first }, { it.second })
.mapValues { it.value.toSortedSet() }

While the other answer(s) might be easier to grasp, they build immediate lists and maps in between, that are basically discarded right after the next operation. The following tries to omit that using splitToSequence (Sequences) and groupingBy (see Grouping bottom part):
val result: Map<String, Set<Long>> = input.splitToSequence(',')
.map { it.split('|', limit = 2) }
.groupingBy { it[0] }
.fold({ _, _ -> mutableSetOf<Long>() }) { _, accumulator, element ->
accumulator.also {
it.add(element[1].toLong()))
}
}
You can of course also filter out the addition of 0 in the set with a simple condition in the fold-step:
// alternative fold skipping 0-values, but keeping keys
.fold({ _, _ -> mutableSetOf<Long>() }) { _, accumulator, element ->
accumulator.also {
val value = element[1].toLong()
if (value != 0L)
it.add(value)
}
}
Alternatively also aggregating might be ok, but then your result-variable needs to change to Map<String, MutableSet<Long>>:
val result: Map<String, MutableSet<Long>> = // ...
.aggregate { _, accumulator, element, first ->
(if (first) mutableSetOf<Long>() else accumulator!!).also {
val value = element[1].toLong()
if (value != 0L)
it.add(value)
}
}

How to handle dynamic JSON elements in Groovy

I have JSON that looks similar to this:
{"test":{"red":"on","green":"off","yellow":"on"},"test1":{"red":"off","green":"on","yellow":"off"},"test2":{"red":"on","green":"off","yellow":"off"}}
I've iterating over this with the code below:
JSONObject t = JSON.parse(params.myObject)
t.each { id, data ->
println id
println data.red
println data.green
println data.yellow
}
However, at times I can have dynamically different values in the JSON Object. For example (new color added):
{"test":{"red":"on","green":"off","yellow":"on","pink":"on"},"test1":{"red":"off","green":"on","yellow":"off","pink":"on"},"test2":{"red":"on","green":"off","yellow":"off","pink":"on"}}
Question
Is there a way to iterate over all the json without hardcoding the colors in my code?

JSONObject t = JSON.parse(params.myObject)
t.each { id, data ->
println id
data.each { prop, value ->
println prop + " = " + value
}
}

I figured it out.
I can simply just iterate over the values in similar fashion:
t.each { id, data ->
println id
data.each {id1, d ->
println id1
println d
}
}