We Keep Coding

elasticsearch query on comparing 2 fields (using java)

I've an index in my elasticsearch and I want to have a query to compare 2 date fields.
assuming fields name are creationDate and modifiedDate. I want to get all documents which these 2 dates are the same in them.
I know it was possible to use FilteredQuery which is deprecated right now.
something like the bellowing code:
FilteredQueryBuilder query = QueryBuilders.filteredQuery(null,
FilterBuilders.scriptFilter("doc['creationDate'].value = doc['modifiedDate'].value"));
Also it's maybe possible to write manual scripts as string, but I doubt that this is the right solution. Any idea's to create the properly query would be appreciated.

Filtered query have been replaced by bool/filter queries You can do it like this:
BoolQueryBuilder bqb = QueryBuilders.boolQuery()
filter(QueryBuilders.scriptQuery("doc['creationDate'].value = doc['modifiedDate'].value"));
However, instead of using scripts at search time, you'd be better off creating a new field at indexing time that contains the information of whether creationDate and modifiedDate are the same dates. Then, you could simply check that flag at query time, it would be much more optimal and fast.
If you don't want to reindex all your data, you can update all of them with that flag, simply run an update by query like this:
POST my-index/_update_by_query
{
"script": {
"source": """
def creationDate = Instant.parse(ctx._source.creationDate);
def modifiedDate = Instant.parse(ctx._source.modifiedDate);
ctx._source.modified = ChronoUnit.MICROS.between(creationDate, modifiedDate) > 0;
""",
"lang": "painless"
},
"query": {
"match_all": {}
}
}
And then your query will simply be
BoolQueryBuilder bqb = QueryBuilders.boolQuery()
filter(QueryBuilders.termQuery("modified", "false");

spring data mongodb - aggregation with dates

Following code doesn't give the expected results, i've tried with multiple documents, those documents structured as follows.
_id: 5a7714d44c75220958e6aa01
imei:355227045347655
point: [3.143453333333333,80.10954]
heading: 0
speed:0
timestamp: 2018-02-04 19:42:36.000
point_distance: 525.25
Now i need to calculate sum of point_distance of every record match with given imei and time period. I've tried to achieve this with following code but returns nothing even if required data exists.
public Object findDistance(long imei, Date from, Date to) {
List aggregationOps = new ArrayList<>();
//operations
aggregationOps.add(match(Criteria.where("imei").is(imei)));
aggregationOps.add(match(Criteria.where("timestamp").gte(from)));
aggregationOps.add(match(Criteria.where("timestamp").lte(to)));
aggregationOps.add(group("imei").sum("point_distance").as("distance"));
aggregationOps.add(project("imei").and("distance").previousOperation());
AggregationOptions agOps = new AggregationOptions.Builder().allowDiskUse(true).cursor(new BasicDBObject()).build();
return (DistanceInfo) getMongoTemplate()
.aggregate(newAggregation(aggregationOps).withOptions(agOps), Location.class, DistanceInfo.class)
.getUniqueMappedResult();
}
Date > from and to are formatted as yyyy-MM-dd hh:mm:ss
DistanceInfo class
public class DistanceInfo {
long imei;
double distance;
}
I'm new to this mongodb stuff and no idea what did i do wrong, how can i correct this ? any help is much appreciated.

Try this. this should work
MatchOperation matchOperation = match(Criteria.where("imei").is(imei)
.and("timestamp").gte(from.getTime()).lte(to.getTime()));
GroupOperation groupOperation = group("imei").sum("point_distance").as("distance");
ProjectionOperation projectionOperation = project().andExpression("imei").as("imei")
.andExpression("distance").as("distance");
Aggregation aggregation = newAggregation(matchOperation, groupOperation, projectionOperation);
AggregationResults<DistanceInfo> results = mongoTemplate.aggregate(aggregation, "location", DistanceInfo.class);
return results.getMappedResults();

Elasticsearch query through Java API

I am using the following elasticsearch query to fetch the details,
{
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"bool": {
"should": [
{"match": {
"val": "GET"
}}]
}
}
}
}
}
It is working fine and given the result as required.
I want to execute the same query through java and get the same results and tried the following,
getClient().prepareSearch(esIndex)
.setQuery(QueryBuilders.queryStringQuery(QUERY)).execute().actionGet();
It is not returning the data and throw some query format wrong exception as well.
Is there any Java API available using which the same query can be executed?
NOTE: There is a possibility available to create the boolquery and aggregation builders in java api and execute the same. I am just curious to find a way to execute this query directly through elasticsearch java api

If you really want to use the Query String Query, your query has to follow Query String Syntax:
getClient().prepareSearch(esIndex)
.setQuery(QueryBuilders.queryStringQuery("val: \"GET\""))
.execute()
.actionGet();
As already stated, you should construct your query by using the provided QueryBuilders instead of strings. This will keep your code clean and readable even for complex queries.
getClient().prepareSearch(esIndex)
.setQuery(QueryBuilders.boolQuery()
.should(QueryBuilders.matchQuery("val", "GET"))
.execute()
.actionGet();

BoolQueryBuilder bool = boolQuery();
bool.must(QueryBuilders.matchAllQuery());
bool.filter(QueryBuilders.boolQuery().should(QueryBuilders.matchQuery("Val", "GET")));
AggregationBuilder agg = AggregationBuilders.terms("").field("");
SearchResponse reponse = getClient().prepareSearch().setIndices("indexName").setTypes("indexType")
.setQuery(bool).addAggregation(agg).execute().actionGet();

you should use boolQuery() when you construct your QueryBuilder:
QueryBuilder qb = boolQuery()
.must(termQuery("content", "test1"))
.must(termQuery("content", "test4"))
.mustNot(termQuery("content", "test2"))
.should(termQuery("content", "test3"))
.filter(termQuery("content", "test5"));
Official docs:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-compound-queries.html

MongoDB Date query JAVA

Im trying to do a query to get all the values from my DB wich each have a date. One example:
leadTime: [
{
date: ISODate("2014-03-19T23:00:00Z"),
value: 25.8
},
{
date: ISODate("2014-03-20T23:00:00Z"),
value: 31.299999999999997
},
{
date: ISODate("2014-03-21T23:00:00Z"),
value: 34.4
}
]
enter code here
My code is:
DBObject query=new BasicDBObject("group",group);
DBObject serieData= col.findOne(query,new BasicDBObject(serie, 1));
if (serieData != null) {
List<DBObject> data = (List<DBObject>) serieData.get(serie);
for (DBObject item : data) {
result.add(new HistoryData((Date) item.get("date"),(Double) item.get("value")));
}
Now I want to get the values that the date is bigger than a date that I pass by parameter. The query I did is this:
DBObject query=new BasicDBObject("group",group)
.append("date", new BasicDBObject("$gte", parameterDate))
;
But I always receive the result empty, can you help me? sorry for my english and than

Assuming that leadTime is a field in your documents, your query has to look like this
DBObject query=new BasicDBObject("group",group)
.append("leadTime.date", new BasicDBObject("$gte", parameterDate))
;
The way you did it, MongoDB would have searched for a date field in your document root:
{ _id: "Foo",
date: ISODate("2014-03-19T23:00:00Z"),
[...]
}
Queries in MongoDB don't make a difference if the queried field is a single value or an array, so using the dot notation on a field which holds an array of subdocuments is perfectly valid.

What you want to do is not possible with a simple query.
But if you still want to do it in mongodb you need to use the aggregation framework, with something like that :
db.<col_name>.aggregate( [ { $unwind : "$group" }, { $match : {'group.date': { $gte : parameterDate } } ] )
this a js command, but you should be able to translate it easly in Java Driver (you can also add a $project operation to just return needed fields).

mongodb java $in query with $regex

I am trying to write $in query with $regex in mongo+java. It's not working in mongo shell either. What I mean is I don't get any results but no query parse error either.
Here's the final query I got from Java Debugger at the line where I say collection.find(finalQuery)
{"$and": [
{"$or": [
{"country": "united states"}
]},
{"businesses": {
"$in": [
{"$regex": "^.*cardinal.*health.*$"},
{"$regex": "^.*the.*hartford.*$"}
]
}}
]}
Java Code snipet for Above query:
Set<Pattern> businesses = new HashSet<Pattern>();
for(String st: srchTerms) {
businesses.add(Pattern.compile("^"+st.trim()+"$"));
}
srchTermQuery.append("businesses", new BasicDBObject("$in", businesses));
However, following query works in mongo shell but I don't know how to write it into java:
{"registering_organization": {
"$in": [
/^.*cardinal.*health.*$/,
/^.*the.*hartford.*$/
]
}}
Java Code add double quotes around regex if we try to define it as a string.

The behavior you're seeing might be a bug, however as an alternative you can write your query like this
Pattern pattern = Pattern.compile("(^aaa$)|(^bbb$)");
srchTermQuery.append("businesses", pattern);
Not pretty but it seem to do the trick

You're not going to be able to convert:
{"businesses" : {
"$in":[
/^.*cardinal.*health.*$/,
/^.*the.*hartford.*$/
]
}}
directly into a Java regex. This is not a bug, it's because the Java driver uses $regex format when creating regex queries to avoid any ambiguity.
The $regex documentation states that
db.collection.find({field: /acme.*corp/});
db.collection.find({field: {$regex: 'acme.*corp'}});
So your Java-generated query of:
{"businesses": {
"$in": [
{"$regex": "^.*cardinal.*health.*$"},
{"$regex": "^.*the.*hartford.*$"}
]
}}
is exactly equivalent of the query you were trying to convert:
{"businesses": {
"$in": [
/^.*cardinal.*health.*$/,
/^.*the.*hartford.*$/
]
}}
In summary, the Java you've written is already the correct way to convert the query you wanted. I've run it in my own test and it returns the expected results.
Perhaps if you included some sample documents that you expect to be returned by the query we could help further?

I had a need to list all keys beginning with a specified string. The following worked for me in CLI:
db.crawlHTML.count({"_id": /^1001/})
The following was the Java implementation:
public List<String> listKeysLike(DB mongoDB, String likeChars) throws Exception {
DBCollection dbCollection = this.getHTMLCollection(mongoDB, TESTPROD);
List<String> keyList = new ArrayList<String>();
BasicDBObject query = new BasicDBObject();
String queryString = "^" + likeChars.trim() ; // setup regex
query.put("_id", java.util.regex.Pattern.compile(queryString));
DBCursor cursor = dbCollection.find(query);
while (cursor.hasNext()) { // _id used as the primary key
BasicDBObject obj = (BasicDBObject) cursor.next();
String tempString = obj.getString("_id");
keyList.add(tempString);
} // while
return keyList;
}
NB: The "TESTPROD" just tells me which of two databases I should be using.

You have to use mongodb regex notation rather than putting it in a string
db.somecollection.find({records: {$in: [/.*somestring.*/]}})