Updating elasticsearch entities with bulk - java

I have this database data as below (ES 7.xx) version
{
"id":"1234",
"expirationDate":"17343234234",
"paths":"http:localhost:9090",
"work":"software dev",
"family":{
"baba":"jams",
"mother":"ela"
}
},
{
"id":"00021",
"expirationDate":"0123234",
"paths":"http:localhost:8080",
"work":"software engi",
"family":{
"baba":"stev",
"mother":"hela"
}
}
how can i update the entity which its expirationDate smaller than current Time? to be the current time for example:
the id 00021 is expired because its expiration date is smaller than today then it should updated to current time.
something like void updateExpiredEntity(List<ids> ids,Long currentTime) suing void bulkUpdate(List<UpdateQuery> queries, BulkOptions bulkOptions, IndexCoordinates index);
Please provide me some code implementation
is it correct like this?
public void update(UUID id,Long currentDate) {
UpdateQuery updateQuery = UpdateQuery.builder(id.toString()).withRouting("expirationDate=currentDate")
.build();
elasticsearchTemplate.bulkUpdate(List.of(updateQuery), IndexCoordinates.of("index"));
}
}

If you are using Elasticsearch 7.xx, I will assume that you have use Spring Data Elasticsearch version 4.0.x that comes with Spring boot 2.3.x. Since it's the version that support Elasticsearch 7.xx.
There're many things that have change in this Spring Data Elasticsearch version. Update document by query is one of them. Unlike before that we autowired ElasticsearchTemplate, we now have to use ElasticsearchRestTemplate and RestHighLevelClient instead.
In your case if you might want to use RestHighLevelClient to update document by query. Assume that you stored expirationDate as number mapping type in seconds unit then the code that you have asked for should look like this.
public class ElasticsearchService {
#Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
#Autowired
private RestHighLevelClient highLevelClient;
public void updateExpireDateDemo() throws IOException {
String indexName = "test";
Date currentDate = new Date();
Long seconds = (Long) (currentDate.getTime() / 1000);
UpdateByQueryRequest request = new UpdateByQueryRequest(indexName);
request.setQuery(new RangeQueryBuilder("expirationDate").lte(seconds));
Script updateScript = new Script(
ScriptType.INLINE, "painless",
"ctx._source.expirationDate=" + seconds + ";",
Collections.emptyMap());
request.setScript(updateScript);
highLevelClient.updateByQuery(request, RequestOptions.DEFAULT);
}
}
I'm not quite get why you really need to use the bulkUpdate but if that's the case then. You have to query the record that need to be update from Elasticsearch to get and id of each document first. Then you can update with list of UpdateQuery. So your code will look like this.
#Service
public class ElasticsearchService {
#Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
public void updateExpireDateByBulkDemo() throws IOException {
String indexName = "test";
Date currentDate = new Date();
Long seconds = (Long) (currentDate.getTime() / 1000);
List<UpdateQuery> updateList = new ArrayList();
RangeQueryBuilder expireQuery = new RangeQueryBuilder("expirationDate").lte(seconds);
NativeSearchQuery query = new NativeSearchQueryBuilder().withQuery(expireQuery).build();
SearchHits<Data> searchResult = elasticsearchRestTemplate.search(query, Data.class, IndexCoordinates.of(indexName));
for (SearchHit<Data> hit : searchResult.getSearchHits()) {
String elasticsearchDocumentId = hit.getId();
updateList.add(UpdateQuery.builder(elasticsearchDocumentId).withScript("ctx._source.expirationDate=" + seconds + ";").build());
}
if (updateList.size() > 0) {
elasticsearchRestTemplate.bulkUpdate(updateList, IndexCoordinates.of(indexName));
}
}
}
However, this only update first page of the search result. If you require to update every record matched your query then you have to use searchScroll method in oder to get every document id instead.

Related

Is there any way to write custom or native queries in Java JPA (DocumentDbRepository) while firing a query to azure-cosmosdb?

Connected to azure-cosmosdb and able to fire default queries like findAll() and findById(String Id). But I can't write a native query using #Query annotation as the code is not considering it. Always considering the name of the function in respository class/interface. I need a way to fire a custom or native query to azure-cosmos db. ?!
Tried with #Query annotation. But not working.
List<MonitoringSessions> findBySessionID(#Param("sessionID") String sessionID);
#Query(nativeQuery = true, value = "SELECT * FROM MonitoringSessions M WHERE M.sessionID like :sessionID")
List<MonitoringSessions> findSessions(#Param("sessionID") String sessionID);
findBySessionID() is working as expected. findSessions() is not working. Below root error came while running the code.
Caused by: org.springframework.data.mapping.PropertyReferenceException: No property findSessions found for type MonitoringSessions
Thanks for the response. I got what I exactly wanted from the below link. Credit goes to Author of the link page.
https://cosmosdb.github.io/labs/java/technical_deep_dive/03-querying_the_database_using_sql.html
public class Program {
private final ExecutorService executorService;
private final Scheduler scheduler;
private AsyncDocumentClient client;
private final String databaseName = "UniversityDatabase";
private final String collectionId = "StudentCollection";
private int numberOfDocuments;
public Program() {
// public constructor
executorService = Executors.newFixedThreadPool(100);
scheduler = Schedulers.from(executorService);
client = new AsyncDocumentClient.Builder().withServiceEndpoint("uri")
.withMasterKeyOrResourceToken("key")
.withConnectionPolicy(ConnectionPolicy.GetDefault()).withConsistencyLevel(ConsistencyLevel.Eventual)
.build();
}
public static void main(String[] args) throws InterruptedException, JSONException {
FeedOptions options = new FeedOptions();
// as this is a multi collection enable cross partition query
options.setEnableCrossPartitionQuery(true);
// note that setMaxItemCount sets the number of items to return in a single page
// result
options.setMaxItemCount(5);
String sql = "SELECT TOP 5 s.studentAlias FROM coll s WHERE s.enrollmentYear = 2018 ORDER BY s.studentAlias";
Program p = new Program();
Observable<FeedResponse<Document>> documentQueryObservable = p.client
.queryDocuments("dbs/" + p.databaseName + "/colls/" + p.collectionId, sql, options);
// observable to an iterator
Iterator<FeedResponse<Document>> it = documentQueryObservable.toBlocking().getIterator();
while (it.hasNext()) {
FeedResponse<Document> page = it.next();
List<Document> results = page.getResults();
// here we iterate over all the items in the page result
for (Object doc : results) {
System.out.println(doc);
}
}
}
}

How do you execute a MongoDB query stored as string in Java?

I'm kind of new to the MongoDB Java driver and I was wondering how you could execute a query stored as a string. Is this the best way to execute them, or what would be a better approach?
I've stumbled across the piece of the below on another stackoverflow thread, but haven't been able to get anything useful out of it. The output does not contain the result of the query at all.
The code I'm running right now:
#Test
public void testExecuteStoredQueries() {
String code = "db.getCollection('users').find({})";
final BasicDBObject command = new BasicDBObject();
String formattedCode = String.format("function() { return %s ; }", code);
System.out.println("Formatted code:");
System.out.println(formattedCode);
command.put("eval", formattedCode);
Document result = DbEngine.getInstance().getDatabase().runCommand(command);
System.out.println(result.toJson());
}
Summarized output:
{
"retval": {
"_mongo": "....",
"_db": "...",
"_collection": "...",
"_ns": "cezy.users",
"_query": {},
"_fields": null,
"_limit": 0,
"_skip": 0,
"_batchSize": 0,
"_options": 0,
"_cursor": null,
"_numReturned": 0,
"_special": false
},
"ok": 1
}
I use morphia when i have to deal with objects. As when you retrieve the data from MongoDb, for the long values you get extended Json instead of Json Response. Parsing Extended Json could be a trouble and might break the code. As Gson doesn't support the conversion from Extended Json to Json.
private void createDatastore(boolean createIndexes) {
Morphia morphia = new Morphia();
morphia.map(classname.class);
datastore = morphia.createDatastore(mongoClient, databaseName);
if (createIndexes) {
datastore.ensureIndexes();
}
}
#Override
public Datastore getDatastore() {
return this.datastore;
}
#Test
public void testExecuteStoredQueries() {
String code = "db.getCollection('users').find({})";
String formattedCode = String.format("function() { return %s ; }", code);
final BasicDBObject basicObject = new BasicDBObject(new BasicDBObject("$in", formattedCode));
Query<ClassName> query = getDatastore().createQuery(<Classname>.class).filter("_eval", basicObject);
List<Classname> List = query.asList();
//if you want to access each object and perform some task
List.forEach((cursor) -> {
//perform your task
});
}
Removing the function creation and adding ".toArray()" pretty much solved the problem.
#Test
public void testExecuteStoredQueries() {
String code = "db.users.find({}).toArray();";
final BasicDBObject command = new BasicDBObject();
command.put("eval", code);
Document result = DbEngine.getInstance().getDatabase().runCommand(command);
System.out.println(result.toJson());
assertNotNull(result.get("retval"));
}
The array is in the "retval" field of the response.

Manage Simultaneous calls to a REST Service without Duplicate an ID in JAVA

I´m build a REST service through Spring and Swagger with CMIS Protocol in Maven. My services works well until the moment when I´m doing simultaneous calls through Jmeter to stress the system. Context: the service get an ID from an ID creator to generate a node in Alfresco. The id cannot be repeated in the system. this id creator take the last created node and sum +1. I tried with Thread.sleep and TimeUnit. The Jmeter Answer in 10 calls in one sec that the Node was created 10 times with the same ID.
Extract of current code: Controller:
#ApiOperation(value = "Crea un Tipo de documento")
#RequestMapping(value = "/create2", method = RequestMethod.POST)
ResponseMessage createTypeDocument2(#RequestParam String description) throws InterruptedException {
return typeDocumentService.createTypeDocument(description);
}
Service:
#Transactional
public ResponseMessage createTypeDocument(String description) throws InterruptedException {
Session session = obtieneSesion();
int id = this.idCreator2();
int aux = 0;
if(searchDocuments(id)==null) {
aux=0;
}else {
aux = Integer.parseInt(searchDocuments(id).getProperty("bc:id").getValueAsString());
}
ResponseMessage rm = new ResponseMessage();
TimeUnit.MILLISECONDS.sleep(new Random().nextInt(500));
Thread.sleep(1000);
if(id != aux ) {
SimpleDateFormat sdf = new SimpleDateFormat("SSS");
Date date = new Date();
String nombre = sdf.format(date.getTime());
DocumentTypeDTO docFilter = new DocumentTypeDTO();
Folder root = (Folder) session.getObjectByPath("/DataList-BCH");
HashMap<String, Object> metadata = new HashMap<String, Object>();
metadata.put(PropertyIds.OBJECT_TYPE_ID, "***************");
metadata.put(PropertyIds.NAME, nombre);
metadata.put("bc:id", id);
metadata.put("bc:available", activo);
metadata.put("bc:description", description);
Document newDoc = root.createDocument(metadata, null, null);
docFilter.setId( Integer.parseInt(newDoc.getProperty("bc:id").getValueAsString()));
docFilter.setDescription(newDoc.getProperty("bc:description").getValueAsString());
docFilter.setUuid(newDoc.getId().replaceAll(";1.0", "").replace("workspace://SpacesStore/", ""));
rm.setMensaje("Exitoso");
rm.setCodigo(200);
rm.setObjeto(docFilter);
return rm;
}else {
rm.setCodigo(-1);
rm.setMensaje("La ID ya existe");
return rm;
}
}
SearchDocument method if the object is not found return null. I try multiple options without results, for that reason I need your help to resolve it. Thanks in Advice
RESOLVED IN STACKOVERFLOW IN SPANISH:
RESOLVED

Spring Rest Controller date calculation

Below is a generalized version of my Spring RestController implementation. It's purpose is to respond to HTTP requests at a specific base URI ("/rest/my/uri/to/resource") with a resource based on the URI ID input. It works fine, however due to the structure of the data it returns I have had to add an optional date param. I have the controller calculate today's date and use that in my database query when the user does not supply one in the URI.
My question to you is if I use the todaySqlDate variable for each response where the user does not supply a date as I am below, will it recalculate the date each time it responds to a request? Obviously if the user inputs a date like
/rest/my/uri/to/resource/identifier/666?date=2016-03-15
this will not be an issue. My concern is that when the date is not included, a la
/rest/my/uri/to/resource/identifier/666
it will use the default date. Will the default date stop being current if the service is left running for more than 24 hours?
#RestController
#RequestMapping(value = "/rest/my/uri/to/resource")
public class ResourceController
{
private static final Logger LOGGER = LoggerFactory.getLogger(ResourceController.class);
#Autowired
private ResourceService resourceService;
public String todaySqlDate = getTodaySqlDate();
#RequestMapping(value = "/identifier/{id}", method = RequestMethod.GET, produces = {MediaType.APPLICATION_JSON_VALUE, MediaType.APPLICATION_XML_VALUE})
public Resource getResource(#PathVariable("id") String id,
#RequestParam(value="date", defaultValue="", required=false) String resourceDate)
throws InvalidParameterException, DataNotFoundException, Exception
{
LOGGER.trace("In ResourceController.getResouce() with {}", id);
try
{
if(!isValidIdentifier(id))
{
throw new InvalidParameterException("id is not valid: " + id);
}
if(resourceDate.isEmpty())
{
resourceDate = todaySqlDate;
}
else if(!isValidSqlDateString(resourceDate))
{
throw new InvalidParameterException("resourceDate is present but not valid: " + resourceDate);
}
ResourceList resourceList = resourceService.getResource(id, resourceDate);
if (resourceList == null)
{
LOGGER.trace("No resources found for given input");
throw new DataNotFoundException("ResourceList for " + id + " not found");
}
return resourceList;
}
catch (Exception e)
{
LOGGER.error(e.getMessage(), e);
throw e;
}
}
public String getTodaySqlDate()
{
java.util.Date utilDate = new java.util.Date();
java.sql.Date sqlDate = new java.sql.Date(utilDate.getTime());
return sqlDate.toString();
}
}
Yes, every new request will be handled by a new separate instance (thread) and hence it will re-calculate the date every time.
You can have a look at Spring/REST Documentation for more information.
Useful Link:
How are Threads allocated to handle Servlet request?

spring data mongodb query document

I am facing this issue(getting null response) when i am trying to Query in Java using
I need to based on placed time stamp range and releases desc and status.
// My document as follows:
<ordersAuditRequest>
<ordersAudit>
<createTS>2013-04-19 12:19:17.165</createTS>
<orderSnapshot>
<orderId>43060151</orderId>
<placedTS>2013-04-19 12:19:17.165</placedTS>
<releases>
<ffmCenterDesc>TW</ffmCenterDesc>
<relStatus>d </relStatus>
</releases>
</ordersAudit>
</ordersAuditRequest>
I am using following query but it returns null.
Query query = new Query();
query.addCriteria(Criteria.where("orderSnapshot.releases.ffmCenterDesc").is(ffmCenterDesc)
.and("orderSnapshot.releases.relStatus").is(relStatus)
.andOperator(
Criteria.where("orderSnapshot.placedTS").gt(orderPlacedStart),
Criteria.where("orderSnapshot.placedTS").lt(orderPlacedEnd)
)
);
I can't reproduce your problem, which suggests that the issue is with the values in the database and the values you're passing in to the query (i.e. they're not matching). This is not unusual when you're trying to match dates, as you need to make sure they're stored as ISODates in the database and queried using java.util.date in the query.
I have a test that shows your query working, but I've made a number of assumptions about your data.
My test looks like this, hopefully this will help point you in the correct direction, or if you give me more feedback I can re-create your problem more accurately.
#Test
public void shouldBeAbleToQuerySpringDataWithDates() throws Exception {
// Setup - insert test data into the DB
SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd' 'hh:mm:ss.SSS");
MongoTemplate mongoTemplate = new MongoTemplate(new Mongo(), "TheDatabase");
// cleanup old test data
mongoTemplate.getCollection("ordersAudit").drop();
Release release = new Release("TW", "d");
OrderSnapshot orderSnapshot = new OrderSnapshot(43060151, dateFormat.parse("2013-04-19 12:19:17.165"), release);
OrdersAudit ordersAudit = new OrdersAudit(dateFormat.parse("2013-04-19 12:19:17.165"), orderSnapshot);
mongoTemplate.save(ordersAudit);
// Create and run the query
Date from = dateFormat.parse("2013-04-01 01:00:05.000");
Date to = dateFormat.parse("2014-04-01 01:00:05.000");
Query query = new Query();
query.addCriteria(Criteria.where("orderSnapshot.releases.ffmCenterDesc").is("TW")
.and("orderSnapshot.releases.relStatus").is("d")
.andOperator(
Criteria.where("orderSnapshot.placedTS").gt(from),
Criteria.where("orderSnapshot.placedTS").lt(to)
)
);
// Check the results
List<OrdersAudit> results = mongoTemplate.find(query, OrdersAudit.class);
Assert.assertEquals(1, results.size());
}
public class OrdersAudit {
private Date createdTS;
private OrderSnapshot orderSnapshot;
public OrdersAudit(final Date createdTS, final OrderSnapshot orderSnapshot) {
this.createdTS = createdTS;
this.orderSnapshot = orderSnapshot;
}
}
public class OrderSnapshot {
private long orderId;
private Date placedTS;
private Release releases;
public OrderSnapshot(final long orderId, final Date placedTS, final Release releases) {
this.orderId = orderId;
this.placedTS = placedTS;
this.releases = releases;
}
}
public class Release {
String ffmCenterDesc;
String relStatus;
public Release(final String ffmCenterDesc, final String relStatus) {
this.ffmCenterDesc = ffmCenterDesc;
this.relStatus = relStatus;
}
}
Notes:
This is a TestNG class, not JUnit.
I've used SimpleDateFormat to create Java Date classes, this is just for ease of use.
The XML value you pasted for relStatus included spaces, which I have stripped.
You showed us the document structure in XML, not JSON, so I've had to assume what your data looks like. I've translated it almost directly into JSON, so it looks like this in the database:
{
"_id" : ObjectId("51d689843004ec60b17f50de"),
"_class" : "OrdersAudit",
"createdTS" : ISODate("2013-04-18T23:19:17.165Z"),
"orderSnapshot" : {
"orderId" : NumberLong(43060151),
"placedTS" : ISODate("2013-04-18T23:19:17.165Z"),
"releases" : {
"ffmCenterDesc" : "TW",
"relStatus" : "d"
}
}
}
You can find what yours really looks like by doing a db.<collectionName>.findOne() call in the mongoDB shell.

Categories