I am trying to find a person by first or last name all having the same building_id.
For example someone will put in a name in a textfield which may be the first or last name. They'll all have the same building id. I cannot seem to find this in the documentation for elastic search.
My result data looks like this:
{
took: 9,
timed_out: false,
_shards: {
total: 5,
successful: 5,
failed: 0
},
hits: {
total: 56,
max_score: 1,
hits: [
{
_index: "jdbc",
_type: "jdbc",
_id: "5",
_score: 1,
_source: {
first_name: "some first name",
last_name: "some last name",
building_id: 1
}
},
...
...
One way is to use a term filter with a multi_match query. To illustrate, first create some docs:
curl -XPUT "http://localhost:9200/test_index"
curl -XPUT "http://localhost:9200/test_index/person/1" -d'
{
"first_name": "Bob",
"last_name": "Jones",
"building_id": 1
}'
curl -XPUT "http://localhost:9200/test_index/person/2" -d'
{
"first_name": "Bill",
"last_name": "Smith",
"building_id": 1
}'
curl -XPUT "http://localhost:9200/test_index/person/3" -d'
{
"first_name": "Joe",
"last_name": "Williams",
"building_id": 2
}'
curl -XPUT "http://localhost:9200/test_index/person/4" -d'
{
"first_name": "John",
"last_name": "Taylor",
"building_id": 2
}'
curl -XPUT "http://localhost:9200/test_index/person/5" -d'
{
"first_name": "Taylor",
"last_name": "Johnson",
"building_id": 2
}'
Then, if you want to search for all the "Taylor"s in Building 2, execute the following:
curl -XPOST "http://localhost:9200/test_index/person/_search" -d'
{
"query": {
"filtered": {
"query": {
"multi_match": {
"query": "taylor",
"fields": [
"first_name",
"last_name"
]
}
},
"filter": {
"term": {
"building_id": 2
}
}
}
}
}'
and you will get back the results:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.94125634,
"hits": [
{
"_index": "test_index",
"_type": "person",
"_id": "5",
"_score": 0.94125634,
"_source": {
"first_name": "Taylor",
"last_name": "Johnson",
"building_id": 2
}
},
{
"_index": "test_index",
"_type": "person",
"_id": "4",
"_score": 0.5906161,
"_source": {
"first_name": "John",
"last_name": "Taylor",
"building_id": 2
}
}
]
}
}
Here is a runnable example (you will need Elasticsearch installed on your machine and running at localhost:9200):
http://be6c2e3260c3e2af000.qbox.io/gist/cc47b232ec60f09b93906a5c80028c1317dbdf28
Related
I'm trying to validate the response schema with karate but facing issue with array.
Attaching the response and feature as well as my schema.json.
Response -
{
"page": 1,
"per_page": 6,
"total": 12,
"total_pages": 2,
"data": [
{
"id": 3,
"email": "emma.wong#reqres.in",
"first_name": "Emma",
"last_name": "Wong",
"avatar": "https://reqres.in/img/faces/3-image.jpg"
},
{
"id": 4,
"email": "eve.holt#reqres.in",
"first_name": "Eve",
"last_name": "Holt",
"avatar": "https://reqres.in/img/faces/4-image.jpg"
},
{
"id": 5,
"email": "charles.morris#reqres.in",
"first_name": "Charles",
"last_name": "Morris",
"avatar": "https://reqres.in/img/faces/5-image.jpg"
},
{
"id": 6,
"email": "tracey.ramos#reqres.in",
"first_name": "Tracey",
"last_name": "Ramos",
"avatar": "https://reqres.in/img/faces/6-image.jpg"
}
],
"support": {
"url": "https://reqres.in/#support-heading",
"text": "To keep ReqRes free, contributions towards server costs are appreciated!"
}
}
Scenario: Get all Users and validate schema
Given url getUrl
When method Get
Then status 200
And print response
Then match response == '#object'
* string jsonSchemaExpected = read('file:src/test/resources/features/sample/responseSchema.json')
And print response.data.length
And match response == jsonSchemaExpected
responseSchema.json
{
"page": "#number",
"per_page": "#number",
"total": "#number",
"total_pages": "#number",
"data": "#[] #object",
"support": "#object"
}
The only observation I have is if you cast to a string, you won't be able to do any matching.
Instead of * string jsonSchemaExpected do * def jsonSchemaExpected.
I'm trying to convert following Mongo shell aggregation query into Spring Data MongoDB Aggregation:
db.getCollection('Orders').aggregate([
{ $group: {
"_id": {"book":"$bookingId"},
"docs": {$push: '$$ROOT'}
}
},
{$project: {
latestOrd: {
$filter: {
input: "$docs",
as: "item",
cond: { $eq: ["$$item.bookingVersion", { $max: "$docs.bookingVersion" }] }
}
}
}
},
{ $unwind: "$latestOrd" },
{ $replaceRoot:{newRoot:"$latestOrd"}}
])
Query fetches all orders with highest booking version (for example one bookingId may have many documents with version 3).
This query works fine in Mongo shell but I have issue with Spring Data version of it:
Aggregation.group("bookingId").push(Aggregation.ROOT).as("docs");
Aggregation.project().and(filter("docs")
.as("item")
.by(valueOf("item.bookingVersion")
.equalToValue(AccumulatorOperators.Max.maxOf("docs.bookingVersion"))))
.as("latestOrd");
Aggregation.unwind("latestOrd");
Aggregation.replaceRoot("latestOrd");
Spring generates Mongo query similar to one that I've provided above except $max accumulator:
{ "$max" : "$$docs.bookingVersion" }
for some reason it adds double dollar sign instead of single dollar sign and as a result I have following error:
'Use of undefined variable: docs' on server 127.0.0.1:27017
I'm using spring-boot-starter 2.1.0.RELEASE and 4.2 version of Mongo server.
I appreciate any help on this.
Input documents:
[
{
"_id": "5f847811ebcd1a51a0196736",
"status": "REJECTED",
"bookingId": "1",
"bookingVersion": 4,
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196735",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 4,
"docNumber": "7",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196734",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 3,
"docNumber": "6",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196738",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 1,
"docNumber": "8",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196737",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 2,
"docNumber": "9",
"operation": "CREATE"
}
]
Expected output:
[
{
"_id": "5f847811ebcd1a51a0196736",
"status": "REJECTED",
"bookingId": "1",
"bookingVersion": 4,
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196735",
"status": "CREATED",
"bookingId": "1",
"bookingVersion": 4,
"docNumber": "7",
"operation": "CREATE"
},
{
"_id": "5f847811ebcd1a51a0196737",
"status": "CREATED",
"bookingId": "2",
"bookingVersion": 2,
"docNumber": "9",
"operation": "CREATE"
}
]
Issue might be with 2.1.0.RELEASE version of spring boot but upgrading it is not an option for me.
To make this work I've created AggregationExpression instead of AccumulatorOperators.Max.maxOf("docs.bookingVersion") part.
Here is a method for it:
private AggregationExpression buildMaxExpression() {
return new AggregationExpression() {
#Override
public Document toDocument(final AggregationOperationContext context) {
return new Document("$max", "$docs.bookingVersion");
}
};
}
I am working with Elasticsearch recently, and I meet a problem that don't know how to solve it.
I have a Json like:
{
"objects": [
"object1": {
"id" : "12345",
"name":"abc"
},
"12345"
]
}
Object2 is a reference of object1, when I trying to saving(or called indexing) into elastic search, it says:
"org.elasticsearch.index.mapper.MapperParsingException: failed to parse"
After I google I found that because object1 is an object, but object 2 is considered as a string.
We cannot change our json in our project, so in this case how can I save it in the elasticsearch?
Thanks for any help and suggestion.
How do you do that?
I run this command and it works.
PUT test/t1/1
{
"objects": {
"object1": {
"id" : "12345",
"name":"abc"
},
"object2": "12345"
}
}
and the result is:
{
"_index": "test",
"_type": "t1",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"created": true
}
UPDATE 1
Depending on your requirements one of these may solve your problem:
PUT test/t1/2
{
"objects": [
{
"object1": {
"id": "12345",
"name": "abc"
}
},
{
"object2": "12345"
}
]
}
PUT test/t1/2
{
"objects": [
{
"object1": {
"id": "12345",
"name": "abc"
},
"object2": "12345"
},
{
...
}
]
}
I'm try to make a bulk update
Method: Post
Url: /customer/external/_bulk
Json Body:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }
Id 1 is updated but id 2 didnt update. I dont know why?
Response is here:
{
"took": 138,
"errors": false,
"items": [
{
"index": {
"_index": "customer",
"_type": "external",
"_id": "1",
"_version": 15,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": false,
"status": 200
}
}
]
}
As #Val mentioned, you should be having the new line character \n at the end of the last line in your json body:
{"index":{"_id":"1"}}
{"name": "John Doe" }
{"index":{"_id":"2"}}
{"name": "Jane Doe" }\n
as per mentioned in bulk_api. Hope it helps!
I am currently in the process of attempting to update an ElasticSearch document via the Java API. I have a groovy script with the following code:
static updateRequestById(String agencyIndex, String type, String id, def policy) {
UpdateRequest updateRequest = new UpdateRequest()
updateRequest.docAsUpsert(true);
updateRequest.parent("agentNumber");
updateRequest.index(agencyIndex)
updateRequest.type(type)
updateRequest.id(id)
updateRequest.doc("policies", policy)
elasticsearchClient.update(updateRequest).get()
}
The problem with I am having is that I want to update an array within the following document:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "int-b-agency",
"_type": "jacket",
"_id": "99808.1.27.09_4644",
"_score": 1,
"_source": {
"agentNumber": "99808.1.27.09",
"fileNumber": "4644",
"policies": [
{
"agentNumber": "99808.1.27.09",
"fileNumber": "4644",
"policyNumber": "2730609-91029084",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "110 Allan Ct ",
"ppcity": "Jacksonville",
"ppstate": "FL",
"ppzip": "32226",
"ppcounty": "Duval",
"policytype": "",
"status": "Active",
"effectiveDate": "2015-04-01T00:00:00-05:00",
"formType": "BASIC OWNERS - ALTA Owners Policy 06_306_FL - FL Original Rate",
"rateCode": "FLOR",
"rateCodeDesc": "FL Original Rate",
"policyTypeCode": "1",
"policyTypeCodeDesc": "BASIC OWNERS",
"amount": 200000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "\\\\10.212.61.206\\FNFCenter\\legacy_jacket_pdfs\\2015_4_FL6465\\Policy_2730609-91029084.pdf",
"legacyPolicy": "true",
"associatedPolNbr": null
}
]
}
}
]
}
}
In the document above I have a document that has an array called "policies" with a single object. I want to be able to update the "policies" array with additional objects. The end result should look something like the following:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "int-b-agency",
"_type": "jacket",
"_id": "41341.1.81.38_41340103",
"_score": 1,
"_source": {
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policies": [
{
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policyNumber": "8122638-91036874",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "1800 Smith St ",
"ppcity": "sicklerville",
"ppstate": "PA",
"ppzip": "08105",
"ppcounty": "Dauphin",
"policytype": "",
"status": "Active",
"effectiveDate": "2016-02-01T00:00:00-06:00",
"formType": "TestData",
"rateCode": "PASALERATE",
"rateCodeDesc": "Sale Rate - Agent",
"policyTypeCode": "26",
"policyTypeCodeDesc": "SALE OWNERS",
"amount": 180000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "SomeLocation1",
"legacyPolicy": "true",
"associatedPolNbr": null
},
{
"agentNumber": "41341.1.81.38",
"fileNumber": "41340103",
"policyNumber": "8122638-91036875",
"checkNumber": "0",
"checkAmount": 0,
"createdOn": null,
"createdBy": "traxuser621",
"propertyTypeCode": "",
"propertyTypeDesc": "1-4 FAMILY RESIDENTIAL",
"ppaddress": "1800 Smith St ",
"ppcity": "sicklerville",
"ppstate": "PA",
"ppzip": "08105",
"ppcounty": "Dauphin",
"policytype": "",
"status": "Active",
"effectiveDate": "2016-02-01T00:00:00-06:00",
"formType": "Test Data",
"rateCode": "PASALERATE",
"rateCodeDesc": "Sale Rate - Agent",
"policyTypeCode": "26",
"policyTypeCodeDesc": "SALE OWNERS",
"amount": 180000,
"hoiAgentNumber": "",
"proForma": false,
"pdfLocation": "SomeLocation2",
"legacyPolicy": "true",
"associatedPolNbr": null
}
]
}
}
]
}
}
What am I doing wrong?
You can use a scripted update:
Put your new policy in a parameter, for example policy
Use a script like the following :
if (!ctxt._source.policies) { ctxt._source.policies = [] }
ctxt._source.policies += policy
See this documentation : https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html
Updates in inverted indexes are deletes and replacements of documents. There is no in-place update like you find in a db. ES uses Lucene under the hood which in-turn implements a kick-ass inverted index.