I have a requirement where I need to update a field in elastic search for multiple ids. Currently I am using XcontentBuilder and passing an Id along with field name but it's a for loop that's why time complexity becomes horrible if I pass multiple Ids. Is there a way where I can do same operation in batches?
My Code is like this:
UpdateRequest updateRequest = new UpdateRequest();
updateRequest.index("index");
updateRequest.type("_doc");
updateRequest.id("1");
updateRequest.doc(jsonBuilder()
.startObject()
.field("gender", "male")
.endObject());
client.update(updateRequest).get();
Id is a dynamic field and for each Id I am running a loop using above code.
I have not tested this but you can check If this can be of any help
Option 1:
List<String> ids = new ArrayList<>();
ids.add("1");
ids.add("2");
for (String id : ids) {
UpdateRequest updateRequest = new UpdateRequest();
updateRequest.index("index");
updateRequest.type("_doc");
updateRequest.id("1");
updateRequest.doc(jsonBuilder().startObject().field("gender", "male").endObject());
bulkRequest.add(updateRequest);
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
Option 2: Using Script
List<String> ids = new ArrayList<>();
ids.add("1");
ids.add("2");
Map<String, Object> params = new HashMap<>();
String scriptCode = "ctx._source.gender=params.gender";
params.put("gender", "male");
BulkRequestBuilder bulkRequest = client.prepareBulk();
for(String id : ids) {
UpdateRequestBuilder updateRequestBuilder = client.prepareUpdate("index", "type", id)
.setScript(new Script(ScriptType.INLINE, "painless", scriptCode, params));
bulkRequest.add(updateRequestBuilder);
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
Related
I'm searching in multiple fields, and I want to get results if the record matches a specific value (entry.getValue()) or the String "ALL"
Here is my code, but it's not working.
SearchRequest searchRequest = new SearchRequest(MY_INDEX);
final BoolQueryBuilder booleanQuery = QueryBuilders.boolQuery();
searchRequest.source().query(booleanQuery);
final BoolQueryBuilder booleanQuery= QueryBuilders.boolQuery();
for (Map.Entry<String, String> entry : params.entrySet()) {
booleanQuery.should(QueryBuilders.termsQuery(entry.getKey(), entry.getValue(), "ALL");
}
I'm using JDK 11 and ES 7.1
Here is a sample code written for country index which is searching for data provided in map. Customize it according to your needs.
//using map for country
Map<String, String> map = new HashMap<>();
map.put("country" , "FRANCE");
map.put("countryCode", "FR");
//List of should queries this will go in should clause of bool query
List<Query> shouldQueryList = new ArrayList<>();
for (Map.Entry<String, String> entry :map.entrySet()) {
//list of terms to match i.e value from map and all.
List<FieldValue> list = Arrays.asList(FieldValue.of(entry.getValue()), FieldValue.of("ALL"));
//Terms query
Query query = new Query.Builder().terms(termsQueryBuilder -> termsQueryBuilder
.field(entry.getKey())
.terms(termQueryField -> termQueryField
.value(list))).build();
shouldQueryList.add(query);
}
try {
//running search from elastic search java client 7.16.3
SearchResponse<Country> response = elasticsearchClient.search(searchRequest -> searchRequest
.query(qBuilder -> qBuilder
.bool(boolQueryBuilder -> boolQueryBuilder
//using should query list here
.should(shouldQueryList)))
, Country.class);
response.hits().hits().forEach(a -> {
//Print matching country name in console
System.out.println(a.source().getCountry());
});
} catch (IOException e) {
log.info(e.getMessage());
}
Above code will generate query like this :
{"query":{"bool":{"should":[{"terms":{"country":["FRANCE","ALL"]}},{"terms":{"countryCode":["FR","ALL"]}}]}}}
this is my previous question - how to insert data in elastic search index
index mapping is as follows
{
"test" : {
"mappings" : {
"properties" : {
"name" : {
"type" : "keyword"
},
"info" : {
"type" : "nested"
},
"joining" : {
"type" : "date"
}
}
}
how can i check the data of field is already present or not before uploading a data to the index
Note :- I dont have id field maintained in index. need to check name in each document if it is already present then dont insert document into index
thanks in advance
As you don't have a id field in your mapping, you have to search on name field and you can use below code to search on it.
public List<SearchResult> search(String searchTerm) throws IOException {
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
MatchQueryBuilder multiMatchQueryBuilder = new
MatchQueryBuilder(searchTerm, "firstName");
searchSourceBuilder.query(matchQueryBuilder);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = esclient.search(searchRequest, RequestOptions.DEFAULT);
return getSearchResults(searchResponse);
}
Note, as you have keyword field instead of match you can use termquerybuilder
And it uses the utility method to parse the searchResponse of ES, code of which is below:
private List<yourpojo> getSearchResults(SearchResponse searchResponse) {
RestStatus status = searchResponse.status();
TimeValue took = searchResponse.getTook();
Boolean terminatedEarly = searchResponse.isTerminatedEarly();
boolean timedOut = searchResponse.isTimedOut();
// Start fetching the documents matching the search results.
//https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-search
// .html#java-rest-high-search-response-search-hits
SearchHits hits = searchResponse.getHits();
SearchHit[] searchHits = hits.getHits();
List<sr> sr = new ArrayList<>();
for (SearchHit hit : searchHits) {
// do something with the SearchHit
String index = hit.getIndex();
String id = hit.getId();
float score = hit.getScore();
//String sourceAsString = hit.getSourceAsString();
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
String firstName = (String) sourceAsMap.get("firstName");
sr.add(userSearchResultBuilder.build());
}
I have a DynamoDb table named school-data in AWS. Below is the existing code to get all the school with a school's name:
private DynamoDBQueryExpression<School> createQueryBySchoolName(String schoolName) {
String matchSchoolName = "schoolName = :schoolName";
Map<String, AttributeValue> schoolNames = new HashMap<>();
schoolNames.put(":schoolName", new AttributeValue().withS(schoolName));
return new DynamoDBQueryExpression<School>()
.withIndexName("schoolName-index")
.withKeyConditionExpression(matchSchoolName)
.withExpressionAttributeValues(schoolNames)
.withConsistentRead(false);
}
The above query works fine. But Now I need to fetch all schools with a particular school name and their address . So, below are the 3 columns in the table:
id schoolName details
The data in details column looks like below:
{
"zone": "North",
"type": "Convent",
"address": {
"id": "138",
"street1": "123 Street",
"street2": "456 Road"
}
}
So, I need to fetch all the schools with name 'ABC' and address street1 as '123 Street'. So, I updated the above query as below:
private DynamoDBQueryExpression<School> createQueryBySchoolName(String schoolName) {
String matchSchoolName = "schoolName = :schoolName";
Map<String, AttributeValue> schoolNames = new HashMap<>();
schoolNames.put(":schoolName", new AttributeValue().withS(schoolName));
schoolNames.put(":streetName", new AttributeValue().withS("123 Street"));
return new DynamoDBQueryExpression<School>()
.withIndexName("schoolName-index")
.withKeyConditionExpression(matchSchoolName)
.withFilterExpression("details.address.street1 = :streetName")
.withExpressionAttributeValues(schoolNames)
.withConsistentRead(false);
}
But, this returns no data. Can you please let me know what am I doing wrong?
You need to set HashKeyValues for the DynamoDBQueryExpression and add the page limit
private DynamoDBQueryExpression<School> createQueryBySchoolName(String schoolName) {
String matchSchoolName = "schoolName = :schoolName";
Map<String, AttributeValue> schoolNames = new HashMap<>();
schoolNames.put(":schoolName", new AttributeValue().withS(schoolName));
schoolNames.put(":streetName", new AttributeValue().withS("123 Street"));
return new DynamoDBQueryExpression<Voucher>()
.withHashKeyValues(schoolName)
.withIndexName("schoolName-index")
.withKeyConditionExpression(matchSchoolName)
.withFilterExpression("details.address.street1 = :streetName")
.withExpressionAttributeValues(schoolNames)
.withConsistentRead(false)
.withLimit(pPageSize);
}
.withHashKeyValues(schoolName) will only accept a Hash Key object as shown in this image
Is it possible to add 2 custom fields (that are not exist at my collections), into aggregate group query, using java with mongoDB? My approach is below but it doesn't work
Double custom_calculated_field1 = 0d;
Double custom_calculated_field2 = 0d;
Document customGroup_fields = new Document();
customGroup_fields.append("customer_id", "$customer_id");
//custom_calculated fields
customGroup_fields.append("custom_calculated_field1", custom_calculated_field1);
customGroup_fields.append("custom_calculated_field2", custom_calculated_field2);
AggregateIterable<Document> customers_results = customerID.aggregate(
Arrays.asList(
Aggregates.match(values),
Aggregates.group(customGroup_fields)
));
This is a problem with the way you structure your aggregation. You can read more about it on the Mongo aggregation docs here: https://docs.mongodb.com/manual/aggregation/
Effectively you assume that your pipeline knows what operations you are trying to execute. A pipeline however works very similar to what a unix pipe | does. It executes any command and passes the output to the next. So you need to tell it that your match is a match and the group is a group. I have an example for both Document and DBObject, where DBObject is the more up to date approach I believe. Consider this example:
#Test
public void testAggProjection() {
DBCollection collection = template.getCollection("test-collection");
DBObject ob = new BasicDBObject(ImmutableMap.of("test", "value1", "something", "here"));
DBObject ob2 = new BasicDBObject(ImmutableMap.of("test", "value1", "hello", "world"));
DBObject ob3 = new BasicDBObject(ImmutableMap.of("test", "value2", "other", "fields"));
WriteResult insert = collection.insert(ob, ob2, ob3);
Assert.assertEquals(3, insert.getN());
DBObject match = new BasicDBObject("$match", new BasicDBObject("test", "value1")); // match the test value
Map<String, Object> grouping = new HashMap<>();
grouping.put("_id", "$test");
grouping.put("my_count", new BasicDBObject("$sum", 1));
DBObject group = new BasicDBObject("$group", grouping); // group by
AggregationOutput aggregate = collection.aggregate(match, group);
System.out.println(aggregate.results());
DBObject next = aggregate.results().iterator().next();
Assert.assertTrue(next.containsField("_id"));
Assert.assertTrue(next.containsField("my_count"));
// with documents raw
MongoClient mc = (MongoClient) client;
MongoCollection<Document> collection2 = mc.getDatabase(template.getDb().getName()).getCollection("test-collection");
Document dob = new Document(ImmutableMap.of("test", "value1", "something", "here"));
Document dob2 = new Document(ImmutableMap.of("test", "value1", "hello", "world"));
Document dob3 = new Document(ImmutableMap.of("test", "value2", "other", "fields"));
collection2.insertMany(Arrays.asList(dob, dob2, dob3));
long count = collection2.count();
Assert.assertEquals(3, count);
Document match2 = new Document("$match", new Document("test", "value1"));
Document group2 = new Document("$group", ImmutableMap.of("_id", "$test", "my_count", new Document("$sum", 1)));
AggregateIterable<Document> aggregate2 = collection2.aggregate(Arrays.asList(match2, group2));
Document next2 = aggregate2.iterator().next();
Assert.assertTrue(next2.get("_id") != null);
Assert.assertTrue(next2.get("my_count") != null);
}
This is a working unit test. What this does is (in both cases):
Create a new collection and insert 3 documents.
Aggregate them with a match, matching 2 documents, and a group where we count the objects we found.
The important bits for you are here:
Document match2 = new Document("$match", new Document("test", "value1"));
Document group2 = new Document("$group", ImmutableMap.of("_id", "$test", "my_count", new Document("$sum", 1)));
AggregateIterable<Document> aggregate2 = collection2.aggregate(Arrays.asList(match2, group2));
In my match2 and group2 documents, I define what stage the aggregation is executing as $match and $group.
The resulting Json is then:
Document{{_id=value1, my_count=2}}
I hope that helps!
Please note that the _id field inside the group is mandatory.
Artur
I am trying to use map with Bulk Insert Api of ElasticSearch Java Api
public void bulkInsert(List<Map<String,String>> listOfObjects ){
BulkRequestBuilder bulkRequest = client.prepareBulk();
Iterator<Map<String,String>> itr = listOfObjects.iterator();
if (itr.hasNext()){
Map<String,String> document = itr.next();
bulkRequest.add(client.prepareIndex(index, type)
.setSource(document));
}
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {
System.out.println(bulkResponse.buildFailureMessage());
}
}
And I am calling this with
Map<String,String> jsonMap = new HashMap<String,String>();
jsonMap.put("name", fullName.toString());
jsonMap.put("file", file);
List<Map<String,String>> listOfObjects = new ArrayList<Map<String,String>>();
listOfObjects.add(jsonMap);
indexService.bulkInsert(listOfObjects);
I am getting following exception
The number of object passed must be even but was [1]
Ok I got the fix :
Use Map<String, Object> instead of Map <String,String>
Map<String,Object> jsonMap = new HashMap<String,Object>();
jsonMap.put("name", fullName.toString());
jsonMap.put("file", file);
List<Map<String,Object>> listOfObjects = new ArrayList<Map<String,Object>>();
listOfObjects.add(jsonMap);
indexService.bulkInsert(listOfObjects);
From ES java api;
Using Map
Map is a key:values pair collection. It represents a JSON structure:
Map<String, Object> json = new HashMap<String, Object>();
json.put("user","kimchy");
json.put("postDate",new Date());
json.put("message","trying out Elasticsearch");