I'm new with Elasticsearch, so don't know how to start properly with the following task.
I have an index with documents that contain 2 types of fields:
address: the string including city and street;
houses: the list of houses' numbers (integers).
In usual case I could search this documents by followin query:
(1)
GET /_search
{
"query":{
"bool":{
"should": [
{"match": {"address": "Gotham Fourteenth street"}},
{"match": {"houses": 33}}
]
}
}
}
My goal is to match such records by single string, like:
(2)
GET /_search
{
"query":{
"bool":{
"should": [
{"match": {"address": "Gotham Fourteenth street 33"}}
]
}
}
}
or even:
curl -X GET 'http://localhost:9200/_search?q=Gotham+Fourteenth+street+33'
i.e. convert the query (2) to (1), that is cut the house number '33' from 'address' and put it as 'houses' match-parameter to the same query before search performed.
I think I could create a plugin in Java that would extract house number from 'address' (parsing is not a problem) and add an extra parameter 'houses' with this value.
Thus my problem is:
how in my plugin I can programmatically add an extra match-parameter 'houses' to my query before search performed?
how to cut the trailing house-number token from 'address' parameter?
curl -X GET 'http://localhost:9200/_search?q=address:"Gotham Fourteenth street"
curl -X GET 'http://localhost:9200/_search?q=address:"Gotham Fourteenth street" AND (houses=13)
Also checkout wildcards
In the end I've developed a plugin for ElasticSearch 2.4.2 and added a REST-action class (derived from BaseRestHandler) with the following handleRequest() method:
#Override
protected void handleRequest(RestRequest request, RestChannel channel, final Client client) throws Exception {
// Get the address parameter
String address = request.param("address");
// ... Code that parses address and extracts house number ...
int house = ...
// Now send both parameters as query to search engine
SearchResponse sr = client
.prepareSearch("myindex")
.setTypes("mytype")
// Query all shards, DFS==calculate global words' frequencies
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
// Address string without (cutted) house number
.setQuery(QueryBuilders.matchQuery("address", address))
// And extracted house number as second filtering parameter
.setPostFilter(QueryBuilders.termQuery("houses", house))
// Starting position of the first returning hit
.setFrom(0)
// Max number of hits
.setSize(10)
// Explain hit score
.setExplain(true)
.get();
// Convert the search response to rest response and send it
BytesRestResponse sr_rest = search2restResponse(sr);
channel.sendResponse(sr_rest);
}
}
The method search2restResponse() mentioned above converts SearchResponse to REST-response and looks as following:
private BytesRestResponse search2restResponse(SearchResponse sr) {
SearchHit[] searchHits = sr.getHits().getHits();
StringBuilder builder = new StringBuilder();
builder.append("[");
for (int i = 0; i < searchHits.length; i++) {
if (i > 0) {
builder.append(",");
}
builder.append(searchHits[i].getSourceAsString());
}
builder.append("]");
String res_json = builder.toString();
return new BytesRestResponse(RestStatus.OK, res_json);
}
Related
How to POST the following JSON structure using Apache OLINGO client? What is the best way to build up this structure? Are there any examples?
{
"itemNumber": "ITEM1"
"lines": [
{
componentNumber": "COMPONENT1"
},
{
componentNumber": "COMPONENT2"
}
]
}
The following Java example using the Olingo Client works for me to post the following JSON structure:
{
"itemNumber": "ITEM1"
"lines": [
{
componentNumber": "COMPONENT1"
},
{
componentNumber": "COMPONENT2"
}
]
}
Java code using OLINGO CLient
public void deepInsertExample(){
//Initiate the ODATA client
ODataClient client = ODataClientFactory.getClient();
client.getConfiguration();
getClient().getObjectFactory();
//Initiate the Client Object Factory
ClientObjectFactory factory = getClient().getObjectFactory();
//Create Line Item 1
ClientEntity lineItem1 = factory.newEntity(new FullQualifiedName("ODATA.LineItem"));
lineItem1.getProperties()
.add(factory.newPrimitiveProperty("componentNumber", factory.newPrimitiveValueBuilder().buildString("COMPONENT2")));
//Create Line Item 2
ClientEntity lineItem2 = factory.newEntity(new FullQualifiedName("ODATA.LineItem"));
lineItem2.getProperties()
.add(factory.newPrimitiveProperty("componentNumber", factory.newPrimitiveValueBuilder().buildString("COMPONENT1")));
//Initiate the entity set
ClientEntitySet entitySet = factory.newEntitySet();
//Add Line Item 1 and Line Item 2 to the Enity
entitySet.getEntities().add(lineItem1);
entitySet.getEntities().add(lineItem2);
//Create the Lines LInk
ClientLink linesLink = factory.newDeepInsertEntitySet("Lines", entitySet);
ClientComplexValue complexValueCreate = factory.newComplexValue("Lines");
complexValueCreate.getNavigationLinks().add(linesLink);
//Create the Item object
ClientEntity item = factory.newEntity(new FullQualifiedName("ODATA.Item"));
item.getProperties()
.add(factory.newPrimitiveProperty("itemNumber", factory.newPrimitiveValueBuilder().buildString("ITEM1")));
//Add the Lines(Entity Set) link to Item Object
item.addLink(linesLink);
//Post the Item
URI absoluteUri = client.newURIBuilder("URL").build();
ODataEntityCreateRequest<ClientEntity> request = client.getCUDRequestFactory()
.getEntityCreateRequest(absoluteUri, item);
request.setAccept("application/json;odata.metadata=minimal");
request.execute();
}
You will have to specify the the NavigationPropertyName in the deep part. So for your sample payload it would look like
{
"itemNumber": "ITEM1",
"lines": {
"componentNumber":"COMPONENT1",
"componentNumber":"COMPONENT2",
}
}
You can refer to this post in SO to get details about n level nesting
The above answer asuumes that your NavigationProperty is named lines, you can substitute it with the right name by looking at service/$metadata
The answer assumes that you are trying to do deep inserts in a odata2 service, for OData4 the concept remains the same but syntax might vary a bit. Please refer to the payload descripted in documentation in case of OData 4
I'm trying to get the order number where a transactionId is equal to another variable I have in my code. My tolls.booths collection looks like this
In my code,
def boothsException = booths.find([ "pings.loc.transactionId": tollEvent.id, "pings.loc.order":1] as BasicDBObject).iterator()
println boothsException
I am getting boothsException = DBCursor{collection=DBCollection{database=DB{name='tolls'}
I would like to essentially say get where transactionId = 410527376 and give me that order number in boothsException (5233423).
This is using MongoDB Java Driver v3.12.2.
The code extracts the value from the returned cursor. I am using newer APIs, so you will find some differences in class names.
int transId = 410527376; // same as tollEvent.id
MongoCursor<Document> cursor = collection
.find(eq("pings.loc.transactionId", transId))
.projection(fields(elemMatch("pings.loc.transactionId"), excludeId()))
.iterator();
while (cursor.hasNext()) {
Document doc = cursor.next();
List<Document> pings = doc.get("pings", List.class);
Integer order = pings.get(0).getEmbedded(Arrays.asList("loc","order"), Double.class).intValue();
System.out.println(order.toString()); // prints 5233423
}
NOTES:
The query with projection gets the following one sub-document from the pings array:
"pings" : [
{
"upvote" : 575,
"loc" : {
"type" : "2dsphere",
"coordinates" : [ .... ],
"transactionId" : 410527376,
"order" : 5233423
},
...
}
]
The remaining code with looping the cursor is to extract the order value from it.
The following are the imports used with the find method's filter and projection:
import static com.mongodb.client.model.Filters.*;
import static com.mongodb.client.model.Projections.*;
While processing the DialogFlow Response object, I get the below given string as textPayload. If this is a Json string, I can easily convert it to a JSONObject and then extract the values. However, could not convert this to a Json Object. How do I get the values for the keys in this string? What is a good way to parse this string in Java?
String to be processed
Dialogflow Response : id: "XXXXXXXXXXXX"
lang: "en"
session_id: "XXXXX"
timestamp: "2020-04-26T16:38:26.162Z"
result {
source: "agent"
resolved_query: "Yes"
score: 1.0
parameters {
}
contexts {
name: "enaccaccountblocked-followup"
lifespan: 1
parameters {
}
}
metadata {
intent_id: "XXXXXXXXXXXX"
intent_name: "EN : ACC : Freezing Process - Yes"
end_conversation: true
webhook_used: "false"
webhook_for_slot_filling_used: "false"
is_fallback_intent: "false"
}
fulfillment {
speech: "Since you have been permanently blocked, please request to unblock your account"
messages {
lang: "en"
type {
number_value: 0.0
}
speech {
string_value: "Since you have been permanently blocked, please request to unblock your account."
}
}
}
}
status {
code: 200
error_type: "success"
}
Convert it to valid json, then map using one of the many libraries out there.
You'll only need to:
replace "Dialogflow Response :" with {
add } to the end
add commas between attributes, ie
at the end of every line with a ":"
after "}", except when the next non-whitespace is also "}"
Jackson (at least) can be configured to allow quotes around attribute names as optional.
Deserializing to a Map<String, Object> works for all valid json (except an array, which this isn't).
If I understand you correctly the issue here is that the keys do not have quotations marks, hence, a JSON parser will reject this.
Since the keys all start on a new line with some white-space and all end with a colon : you can fix this easily with a regular expression.
See How to Fix JSON Key Values without double-quotes?
You can then parse it to a Map via
Map<String, Object> map
= objectMapper.readValue(json, new TypeReference<Map<String,Object>>(){});
(but I assume you are aware of this).
Create a class for TextPayload object like this.
public class TextPayload {
private int session_id;
private String lang;
private String timestamp;
private String[] metadata ;
//Other attributes
//getters setters
}
Then using an ObjectMapper extract the values from textpayload like this:
ObjectMapper mapper = new ObjectMapper();
TextPayload textPayload = mapper.readValue(output, User.class);
To utilize ObjectMapper and hands on with it follow this
you can use the nodejs package parse-dialogflow-log to parse the textResponse string.
replace "Dialogflow Response :" with "{"
add "}" to the end
run the package on the result and you'll get a nice json.
{"serviceName":"Legal Entity account for given input account.","requestTime":1545426348945,"responseTime":1545426348949,"timeTaken":4,"responseCode":0,"responseMessage":"Success","pageSize":100,"pageNumber":0,"accounts":{"transferDate":1549429200000,"migrationWave":"5","searchedLEAccount":{"accountNumber":"41477514","cbdNumber":"12345678","bic":"CHASGBXxX","poolAccount":"Y","sweepMasterAccount":"Y","status":"DORMANT","branchId":"000000071","branchName":"LONDON","leAccountType":"OLD"},"linkedLEAccount":{"accountNumber":"6541245045","cbdNumber":"854321","bic":"CHASLUY","status":"DORMANT","branchId":"000000055","branchName":"S.A","leAccountType":"NEW"}}}
I am trying to grab all accountNumber and validate if they are numbers. What am I doing wrong?
When method Post
Then status 200
And match response != null
And match response contains {serviceName: 'Legal Entity account for given input account.' }
And match response.accounts.searchedLEAccount contains { accountNumber: '#notnull' }
And match response.accounts.searchedLEAccount contains { accountNumber: '#present' }
And match response.accounts.searchedLEAccount contains { accountNumber: '#number' }
In one line:
* match each $..accountNumber == '#regex \\d+'
Tip: read the docs carefully and understand Json-Path.
Here's the full example which you can paste into a new Scenario and see working:
* def response =
"""
{
"serviceName":"Legal Entity account for given input account.",
"requestTime":1545426348945,
"responseTime":1545426348949,
"timeTaken":4,
"responseCode":0,
"responseMessage":"Success",
"pageSize":100,
"pageNumber":0,
"accounts":{
"transferDate":1549429200000,
"migrationWave":"5",
"searchedLEAccount":{
"accountNumber":"41477514",
"cbdNumber":"12345678",
"bic":"CHASGBXxX",
"poolAccount":"Y",
"sweepMasterAccount":"Y",
"status":"DORMANT",
"branchId":"000000071",
"branchName":"LONDON",
"leAccountType":"OLD"
},
"linkedLEAccount":{
"accountNumber":"6541245045",
"cbdNumber":"854321",
"bic":"CHASLUY",
"status":"DORMANT",
"branchId":"000000055",
"branchName":"S.A",
"leAccountType":"NEW"
}
}
}
"""
* match each $..accountNumber == '#regex \\d+'
I use the following code to post :
try {
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(VERIFY_PAYMENT_ACTIONURL);
// key is the parameter
// MERCHANT_KEY is the value
method.addParameter("form", "2");
method.addParameter("key", MERCHANT_KEY.trim());
method.addParameter("command", VERIFY_PAYMENT_COMMAND.trim());
method.addParameter("hash", hash);
method.addParameter("var1", transactionID.trim());
method.addParameter("salt", ALGORIHTM_SHA512_SALT_KEY.trim());
int statusCode = client.executeMethod(method);
if (statusCode != -1) {
in = method.getResponseBodyAsStream();
}
String text = method.getResponseBodyAsString();
System.out.println("text : "+text);
method.releaseConnection();
} catch (Exception e) {
e.printStackTrace();
}
and I get the response text as follows :
a:3:{s:6:"status";i:1;s:3:"msg";s:44:"1 out of 1 Transactions Fetched Successfully";s:19:"transaction_details";a:1:{i:2298597;a:15:{s:8:"mihpayid";s:18:"403993715508098532";s:10:"request_id";N;s:12:"bank_ref_num";N;s:3:"amt";s:5:"53.77";s:4:"disc";s:4:"0.00";s:4:"mode";s:2:"CC";s:7:"PG_TYPE";s:4:"AXIS";s:7:"card_no";s:16:"512345XXXXXX2346";s:12:"name_on_card";s:3:"emu";s:4:"udf2";s:1:"0";s:7:"addedon";s:19:"2013-06-03 17:34:42";s:6:"status";s:7:"failure";s:14:"unmappedstatus";s:6:"failed";s:12:"Merchant_UTR";N;s:10:"Settled_At";N;}}}
now I want to extract the above output and put it into map like "transactionid" and other details like the following format
array('status' => '1',
'msg' => 'Transaction Fetched Successfully',
'transaction_details' =>
array(
'mihpayid' => Transaction ID,
'request_id' => Request ID,
'bank_ref_num' => Bank Reference Number,
'amt' => Amount
'disc' => Discount
'mode' => Transaction Mode (NB for Netbanking, CC for credit card, DC for Debit card, "-" for
'status' => Transaction Status
unknown)
)
);
I really do not find any common way to extract the above output. will some one help to do this?
or is it possible for me to convert the above to JSON and put them in to a map like the above?
thanks
I don't recognise it but are you sure it's not a well know format?
To turn it into JSON you'd need to
1)remove every instance of a character that is followed by a colon (and the colon)
2)replace every other semi-colons with a comma
3)replace all the other semi-colons with colons
And you'd need to do all that while taking into account that strings could contain any of those things.
From comments:
The string is in PHP serialised form. This library parses it: https://code.google.com/p/serialized-php-parser/