We Keep Coding

Elastic search term query not working on a specific field

I'm new to elastic search.
So this is how the index looks:
{
"scresults-000001" : {
"aliases" : {
"scresults" : { }
},
"mappings" : {
"properties" : {
"callType" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"code" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"data" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"esdtValues" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"gasLimit" : {
"type" : "long"
},
AND MORE OTHER Fields.......
If I'm trying to create a search query in Java that looks like this:
{
"bool" : {
"filter" : [
{
"term" : {
"sender" : {
"value" : "sendervalue",
"boost" : 1.0
}
}
},
{
"term" : {
"data" : {
"value" : "YWRkTGlxdWlkaXR5UHJveHlAMDAwMDAwMDAwMDAwMDAwMDA1MDBlYmQzMDRjMmYzNGE2YjNmNmE1N2MxMzNhYjdiOGM2ZjgxZGM0MDE1NTQ4M0A3ZjE1YjEwODdmMjUwNzQ4QDBjMDU0YjcwNDhlMmY5NTE1ZWE3YWU=",
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
If I run this query I get 0 hits. If I change the field "data" with other field it works. I don't understand what's different.
How I actually create the query in Java+SpringBoot:
QueryBuilder boolQuery = QueryBuilders.boolQuery()
.filter(QueryBuilders.termQuery("sender", "sendervalue"))
.filter(QueryBuilders.termQuery("data",
"YWRkTGlxdWlkaXR5UHJveHlAMDAwMDAwMDAwMDAwMDAwMDA1MDBlYmQzMDRjMmYzNGE2YjNmNmE1N2MxMzNhYjdiOGM2ZjgxZGM0MDE1NTQ4M0A3ZjE1YjEwODdmMjUwNzQ4QDBjMDU0YjcwNDhlMmY5NTE1ZWE3YWU="));
Query searchQuery = new NativeSearchQueryBuilder()
.withFilter(boolQuery)
.build();
SearchHits<ScResults> articles = elasticsearchTemplate.search(searchQuery, ScResults.class);

Since you're trying to do an exact match on a string with a term query, you need to do it on the data.keyword field which is not analyzed. Since the data field is a text field, hence analyzed by the standard analyzer, not only are all letters lowercased but the = sign at the end also gets stripped off, so there's no way this can match (unless you use a match query on the data field but then you'd not do exact matching anymore).
POST _analyze
{
"analyzer": "standard",
"text": "YWRkTGlxdWlkaXR5UHJveHlAMDAwMDAwMDAwMDAwMDAwMDA1MDBlYmQzMDRjMmYzNGE2YjNmNmE1N2MxMzNhYjdiOGM2ZjgxZGM0MDE1NTQ4M0A3ZjE1YjEwODdmMjUwNzQ4QDBjMDU0YjcwNDhlMmY5NTE1ZWE3YWU="
}
Results:
{
"tokens" : [
{
"token" : "ywrktglxdwlkaxr5uhjvehlamdawmdawmdawmdawmdawmda1mdblymqzmdrjmmyznge2yjnmnme1n2mxmznhyjdiogm2zjgxzgm0mde1ntq4m0a3zje1yjewoddmmjuwnzq4qdbjmdu0yjcwndhlmmy5nte1zwe3ywu",
"start_offset" : 0,
"end_offset" : 163,
"type" : "<ALPHANUM>",
"position" : 0
}
]
}

Elasticsearch conditional scoring in java

I am new to Elastic Search and I wanted to ask how can I boost certain results from a query based on some conditional. Specifically, the items in Elastic Search are of the following format:
"name":"item1",
"desc":"desc1",
"brand":"brandname"
I want to make the score of certain items (those with "brand" = "nike") have lower score than the other brands. Is there a way to do that with the QueryBuilders classes that java has for elastic search?

Boosting query might be able to solve the problem. Alternatively, you can also use function_score queries for more flexibility.
Example with boosting query.
(1) Index few documents
curl -XPOST localhost:9200/idx/doc/1 -d '{"content": "foo"}' --header "Content-type:application/json"
curl -XPOST localhost:9200/idx/doc/2 -d '{"content": "bar"}' --header "Content-type:application/json"
curl -XPOST localhost:9200/idx/doc/3 -d '{"content": "foo fred bar"}' --header "Content-type:application/json"
(2) Query with a negative boost for bar to reduce score for bar
curl -XPOST localhost:9200/idx/_search?pretty -d '{
"query": {
"boosting": {
"positive": {
"match": {
"content": "foo bar fred"
}
},
"negative": {
"match": {
"content": "bar"
}
} , "negative_boost": 0.001
}
}
}' --header "Content-type:application/json"
{
"took" : 59,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 0.5619609,
"hits" : [
{
"_index" : "idx",
"_type" : "doc",
"_id" : "1",
"_score" : 0.5619609,
"_source" : {
"content" : "foo"
}
},
{
"_index" : "idx",
"_type" : "doc",
"_id" : "3",
"_score" : 0.0014472058,
"_source" : {
"content" : "foo fred bar"
}
},
{
"_index" : "idx",
"_type" : "doc",
"_id" : "2",
"_score" : 5.619609E-4,
"_source" : {
"content" : "bar"
}
}
]
}
}

To boost a certain value of a certain field (or vice versa), you can use a Function score query.
Check the official document Here.
Let me show you an example :
POST /goods/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": "shoes"
}
},
"functions": [
{
"filter": {
"term": {
"maker": "nike"
}
},
"weight": 0
}
],
"boost_mode": "multiply"
}
},
"sort": [
{
"_score": {
"order": "desc"
}
}
]
}
This query refers that "I'm looking for documents which match a keyword shoes in a name field and sort in reverse order of _score. If documents have a term nike in a maker field, however, I hope that the score is 0(cause multiplying any number by 0 equals 0)."
In Java code,
String indexName = "goods"
String nameField = "name";
String makerField = "maker";
String keyword = "shoes";
String unnecessaryMaker = "nike";
MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder(nameField, keyword);
FilterFunctionBuilder[] filterFunctions = {
new FunctionScoreQueryBuilder.FilterFunctionBuilder(
QueryBuilders.termQuery(makerField, unnecessaryMaker),
ScoreFunctionBuilders.weightFactorFunction(0)
)
};
FunctionScoreQueryBuilder functionScoreQueryBuilder = QueryBuilders.functionScoreQuery(matchQueryBuilder, filterFunctions);
functionScoreQueryBuilder.boostMode(CombineFunction.MULTIPLY);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(functionScoreQueryBuilder);
SearchRequest request = new SearchRequest();
request.indices(indexName);
request.source(searchSourceBuilder);
There are several boost_mode such as multiply(default), replace, sum, avg, max, min.
So you can apply this code to your needs.
Hope this is what you were looking for.

Highchart tooltip value should be format based on locale

For example, I want to display the below format:
Italy Format : 1.325.000
Us Format : 1,325,000
etc.
But highchart display another format using the below response.
Based on locale it shows the value in tooltip dynamically.
I am trying with below response.
Highcharts.chart('container', {
"tooltip" : {
"shared" : true
},
"legend" : {
"enabled" : true,
"reversed" : false
},
"credits" : {
"enabled" : false
},
"exporting" : {
"enabled" : false
},
"chart" : {
"zoomType" : "xy"
},
"title" : {
"text" : "Financial analytic"
},
"xAxis" : [ {
"categories" : [ "Amar", "Kiran", "Venkatesh" ],
"crosshair" : true
} ],
"yAxis" : [ {
"title" : {
"text" : "Financial"
},
"labels" : {
"format" : "${value:.2f}USD"
}
} ],
"series" : [ {
"type" : "column",
"name" : "Financial",
"data" : [ 1325000.0, 1740000.0, 1560000.0 ],
"tooltip" : {
"valueDecimals" : 2,
"valuePrefix" : "$",
"valueSuffix" : "USD"
},
"yAxis" : 0
}]
});

You need to set thousandsSep in the lang options:
Highcharts.setOptions({
lang: {
thousandsSep: ','
}
});
Live demo: http://jsfiddle.net/BlackLabel/4m79t50g/1/
API Reference: https://api.highcharts.com/gantt/lang.thousandsSep

ElasticSearch range query in a paragraph

I have a field called Description which is a text field and has data like:
This is a good thing for versions before 3.2 but bad for 3.5 and later
I want to run range query on this type of text. I know that for a field containing only Dates/Age(Numbers) or even String Ids, we can use queries like
{
"query": {
"range" : {
"age" : {
"gte" : 10,
"lte" : 20,
"boost" : 2.0
}
}
}
}
But i have a mixed field like mentioned above and I need to perform range query on that. Also, i cannot change the index structure. I can only perform queries or do some post processing after retrieving results. So anyone has any idea how to run this type of query, or even obtain my goal after getting results in the post processing? I am using Java.

I hope i fully understand what you are looking for.
I've managed to create a simple working example.
Mappings
Using char_group tokenizer:
The char_group tokenizer breaks text into terms whenever it encounters a character which is in a defined set. It is mostly useful for cases where a simple custom tokenization is desired, and the overhead of use of the pattern tokenizer is not acceptable.
Char Group Tokenizer
PUT my_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"type": "custom",
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "char_group",
"tokenize_on_chars": [
"letter",
"whitespace"
]
}
}
}
},
"mappings": {
"properties": {
"text": {
"type": "text",
"fields": {
"digit": {
"type": "text",
"analyzer": "my_analyzer"
}
}
}
}
}
}
Post a few documents
PUT my_index/_doc/1
{
"text": "This is a good thing for versions before 3.2 but bad for 3.5 and later"
}
PUT my_index/_doc/2
{
"text": "This is a good thing for versions before 5 but bad for 6 and later"
}
Search Query
GET my_index/_search
{
"query": {
"range": {
"text.digit": {
"gte": 3.2,
"lte": 3.5
}
}
}
}
Results
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "my_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"text" : "This is a good thing for versions before 3.2 but bad for 3.5 and later"
}
}
]
}
Another Search Query
GET my_index/_search
{
"query": {
"range": {
"text.digit": {
"gt": 3.5
}
}
}
}
Results
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "my_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"text" : "This is a good thing for versions before 5 but bad for 6 and later"
}
}
]
}
Analyze Query
Play with the following query till you get the desired results.
It is already compatible to your example.
This is a good thing for versions before 3.2 but bad for 3.5 and later
POST _analyze
{
"tokenizer": {
"type": "char_group",
"tokenize_on_chars": [
"letter",
"whitespace"
]
},
"text": "This is a good thing for versions before 3.2 but bad for 3.5 and later"
}
Hope this helps

Spring MongoDB Aggregation Group - Can't get the group query right

I have a document in a MongoDB, which looks like follows.
{
"_id" : ObjectId("5ceb812b3ec6d22cb94c82ca"),
"key" : "KEYCODE001",
"values" : [
{
"classId" : "CLASS_01",
"objects" : [
{
"code" : "DD0001"
},
{
"code" : "DD0010"
}
]
},
{
"classId" : "CLASS_02",
"objects" : [
{
"code" : "AD0001"
}
]
}
]
}
I am interested in getting a result like follows.
{
"classId" : "CLASS_01",
"objects" : [
{
"code" : "DD0001"
},
{
"code" : "DD0010"
}
]
}
To get this, I came up with an aggregation pipeline in Robo 3T, which looks like follows. And it's working as expected.
[
{
$match:{
'key':'KEYCODE001'
}
},
{
"$unwind":{
"path": "$values",
"preserveNullAndEmptyArrays": true
}
},
{
"$unwind":{
"path": "$values.objects",
"preserveNullAndEmptyArrays": true
}
},
{
$match:{
'values.classId':'CLASS_01'
}
},
{
$project:{
'object':'$values.objects',
'classId':'$values.classId'
}
},
{
$group:{
'_id':'$classId',
'objects':{
$push:'$object'
}
}
},
{
$project:{
'_id':0,
'classId':'$_id',
'objects':'$$objects'
}
}
]
Now, when I try to do the same in a SpringBoot application, I can't get it running. I ended up having the error java.lang.IllegalArgumentException: Invalid reference '$complication'!. Following is what I have done in Java so far.
final Aggregation aggregation = newAggregation(
match(Criteria.where("key").is("KEYCODE001")),
unwind("$values", true),
unwind("$values.objects", true),
match(Criteria.where("classId").is("CLASS_01")),
project().and("$values.classId").as("classId").and("$values.objects").as("object"),
group("classId", "objects").push("$object").as("objects").first("$classId").as("_id"),
project().and("$_id").as("classId").and("$objects").as("objects")
);
What am I doing wrong? Upon research, I found that multiple fields in group does not work or something like that (please refer to this question). So, is what I am currently doing even possible in Spring Boot?

After hours of debugging + trial and error, found the following solution to be working.
final Aggregation aggregation = newAggregation(
match(Criteria.where("key").is("KEYCODE001")),
unwind("values", true),
unwind("values.objects", true),
match(Criteria.where("values.classId").is("CLASS_01")),
project().and("values.classId").as("classId").and("values.objects").as("object"),
group(Fields.from(Fields.field("_id", "classId"))).push("object").as("objects"),
project().and("_id").as("classId").and("objects").as("objects")
);
It all boils down to group(Fields.from(Fields.field("_id", "classId"))).push("object").as("objects") that which introduces a org.springframework.data.mongodb.core.aggregation.Fields object that wraps a list of org.springframework.data.mongodb.core.aggregation.Field objects. Within Field, the name of the field and the target could be encapsulated. This resulted in the following pipeline which is a match for the expected.
[
{
"$match" :{
"key" : "KEYCODE001"
}
},
{
"$unwind" :{
"path" : "$values", "preserveNullAndEmptyArrays" : true
}
},
{
"$unwind" :{
"path" : "$values.objects", "preserveNullAndEmptyArrays" : true
}
},
{
"$match" :{
"values.classId" : "CLASS_01"
}
},
{
"$project" :{
"classId" : "$values.classId", "object" : "$values.objects"
}
},
{
"$group" :{
"_id" : "$classId",
"objects" :{
"$push" : "$object"
}
}
},
{
"$project" :{
"classId" : "$_id", "objects" : 1
}
}
]
Additionally, figured that there is no need to using $ sign anywhere and everywhere.