I am trying to use ES as the index for my MongoDB. I've managed to integrate them successfully, but I find the search API rather complex and confusing. The Java API is not too helpful either.
I am able to find exact matches, but how can I get this result? Here is my code:
Node node = nodeBuilder().node();
SearchResponse sr = node.client().prepareSearch()
.addAggregation(
AggregationBuilders.terms("user").field("admin2san")
.subAggregation(AggregationBuilders.terms("SPT").field("64097"))
)
.execute().actionGet();
SearchHit[] results = sr.getHits().getHits();
List<Firewall> myfirewall = results.getSourceAsObjectList(Firewall.class);
for (Firewall info : myfirewall) {
System.out.println("search result is " + info);
}
I'm not quite sure I understood your question.
If you want to print the result of your searchResponse according to your example it should be something like this :
SearchHit[] results = sr.getHits().getHits();
for(SearchHit hit : results){
String sourceAsString = hit.getSourceAsString();
if (sourceAsString != null) {
Gson gson = new GsonBuilder().setDateFormat(dateFormat)
.create();
System.out.println( gson.fromJson(sourceAsString, Firewall.class));
}
}
I'm using Gson to convert from the Json response to the FireWall(POJO).
I hope it's what you were looking for.
response.getHits().getHits()[0].getSourceAsMap() you could try somwthing like this
Related
My project had a new reqiurement of migrating all datas which are currently in postresql to elasticsearch. Successfully migrated all my datas, but I am stuck with writing java code to search for some datas in elastic search index.
Sample structure of a hit in index is attached in the below image:
I need to find average of activity.attentionLevel from the index.
I wrote something like below query to find average:
GET proktor-activities/_search
{
"aggs" : {
"avg_attention" : {
"avg" : {
"script" : "Float.parseFloat(doc['activity.attentionLevel.keyword'].value)" }
}
}
}
please help me to find java equivalent code for doing the same.
thanks in advance
Using Elastic's RestHighLevel API would be something like this:
// Create the client
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(new HttpHost("localhost", 9200, "http")));
// Specify the aggregation request
SearchRequest searchRequest = new SearchRequest("proktor-activities");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.aggregation(AggregationBuilders
.avg("avg_attention")
.field("activity.attentionLevel"));
searchRequest.source(searchSourceBuilder);
// Execute the aggreagation with Elasticsearch
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
// Read the aggregation result
Aggregations aggregations = response.getAggregations();
Avg averageAttention = aggregations.get("avg_attention");
double result = averageAttention.getValue();
client.close(); // get done with the client
More information here.
I am developing a project that integrates Java and Elasticsearch.
And I am using scroll api because of searching a large amount of data.
I want to see unique result( like distint in oracle).
How to remove duplicate search result in elasticsearch?
I searched, but couldn't find the Java version.
My code is like this (this is a just sample code):
final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
SearchRequest searchRequest = new SearchRequest("posts");
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(matchQuery("title", "Elasticsearch"));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
while (searchHits != null && searchHits.length > 0) {
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(scroll);
searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();
}
Is there any way to search the data on the elastic without duplication?
Scroll will return all the documents in elasticsearch. You cannot perform distinct operation in elasticsearch
There are two ways to resolve your issue
Field collapsing
It does a group by on different fields and return top 1 document
Terms Aggregation
{
"aggs": {
"t": {
"terms": {
"script": "doc['title.keyword'] + ' '+doc['description.keyword']"
}
}
}
}
Scroll is intended to fetch large number of documents and above options are not for bulk data. So you need to perform distinct operation client side(outside of elastic search)
I'm using the Java API for ElasticSearch. I'm attempting to highlight my fields but it's not working. The correct results that match the search term are being returned, so there is content to highlight, but it simply won't do it. I set my SearchResponse and HighlightBuilder like this:
QueryBuilder matchQuery = simpleQueryStringQuery(searchTerm);
...
HighlightBuilder highlightBuilder = new HighlightBuilder()
.postTags("<highlight>")
.preTags("</highlight>")
.field("description");
SearchResponse response = client.prepareSearch("mediaitems")
.setTypes("mediaitem")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(matchQuery) // Query
.setFrom(from)
.setSize(pageSize)
.setExplain(true)
.highlighter(highlightBuilder)
.get();
and in my JSON->POJO code, I check to see which fields have been highlighted, but the returned Map is empty.
Arrays.stream(hits).forEach((SearchHit hit) -> {
String source = hit.getSourceAsString();
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
try {
MediaItem mediaItem = objectMapper.readValue(source, MediaItem.class);
mediaItemList.add(mediaItem);
} catch (IOException e) {
e.printStackTrace();
}
});
Why on earth is my highlighting request being ignored?
Any help is greatly appreciated.
You have to set the highlighted field in HighlightBuilder.
For example:
HighlightBuilder.Field field = new HighlightBuilder.Field(fieldName);
highlightBuilder.field(field);
I saw you are using simple query string query, so you can do the following:
Your query string: fieldname: searched text
So for example your query string is the following:
price: >2000 && city: Manchaster
With this query string you specified the fields in the query too.
Now highlighter should work.
I am learning Amazon Cloud Search but I couldn't find any code in either C# or Java (though I am creating in C# but if I can get code in Java then I can try converting in C#).
This is just 1 code I found in C#: https://github.com/Sitefinity-SDK/amazon-cloud-search-sample/tree/master/SitefinityWebApp.
This is 1 method i found in this code:
public IResultSet Search(ISearchQuery query)
{
AmazonCloudSearchDomainConfig config = new AmazonCloudSearchDomainConfig();
config.ServiceURL = "http://search-index2-cdduimbipgk3rpnfgny6posyzy.eu-west-1.cloudsearch.amazonaws.com/";
AmazonCloudSearchDomainClient domainClient = new AmazonCloudSearchDomainClient("AKIAJ6MPIX37TLIXW7HQ", "DnrFrw9ZEr7g4Svh0rh6z+s3PxMaypl607eEUehQ", config);
SearchRequest searchRequest = new SearchRequest();
List<string> suggestions = new List<string>();
StringBuilder highlights = new StringBuilder();
highlights.Append("{\'");
if (query == null)
throw new ArgumentNullException("query");
foreach (var field in query.HighlightedFields)
{
if (highlights.Length > 2)
{
highlights.Append(", \'");
}
highlights.Append(field.ToUpperInvariant());
highlights.Append("\':{} ");
SuggestRequest suggestRequest = new SuggestRequest();
Suggester suggester = new Suggester();
suggester.SuggesterName = field.ToUpperInvariant() + "_suggester";
suggestRequest.Suggester = suggester.SuggesterName;
suggestRequest.Size = query.Take;
suggestRequest.Query = query.Text;
SuggestResponse suggestion = domainClient.Suggest(suggestRequest);
foreach (var suggest in suggestion.Suggest.Suggestions)
{
suggestions.Add(suggest.Suggestion);
}
}
highlights.Append("}");
if (query.Filter != null)
{
searchRequest.FilterQuery = this.BuildQueryFilter(query.Filter);
}
if (query.OrderBy != null)
{
searchRequest.Sort = string.Join(",", query.OrderBy);
}
if (query.Take > 0)
{
searchRequest.Size = query.Take;
}
if (query.Skip > 0)
{
searchRequest.Start = query.Skip;
}
searchRequest.Highlight = highlights.ToString();
searchRequest.Query = query.Text;
searchRequest.QueryParser = QueryParser.Simple;
var result = domainClient.Search(searchRequest).SearchResult;
//var result = domainClient.Search(searchRequest).SearchResult;
return new AmazonResultSet(result, suggestions);
}
I have already created domain in Amazon Cloud Search using AWS console and uploaded document using Amazon predefine configuration option that is movie Imdb json file provided by Amazon for demo.
But in this method I am not getting how to use this method, like if I want to search Director name then how do I pass in this method as because this method parameter is of type ISearchQuery?
I'd suggest using the official AWS CloudSearch .NET SDK. The library you were looking at seems fine (although I haven't look at it any detail) but the official version is more likely to expose new CloudSearch features as soon as they're released, will be supported if you need to talk to AWS support, etc, etc.
Specifically, take a look at the SearchRequest class -- all its params are strings so I think that obviates your question about ISearchQuery.
I wasn't able to find an example of a query in .NET but this shows someone uploading docs using the AWS .NET SDK. It's essentially the same procedure as querying: creating and configuring a Request object and passing it to the client.
EDIT:
Since you're still having a hard time, here's an example. Bear in mind that I am unfamiliar with C# and have not attempted to run or even compile this but I think it should at least be close to working. It's based off looking at the docs at http://docs.aws.amazon.com/sdkfornet/v3/apidocs/
// Configure the Client that you'll use to make search requests
string queryUrl = #"http://search-<domainname>-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.cloudsearch.amazonaws.com";
AmazonCloudSearchDomainClient searchClient = new AmazonCloudSearchDomainClient(queryUrl);
// Configure a search request with your query
SearchRequest searchRequest = new SearchRequest();
searchRequest.Query = "potato";
// TODO Set your other params like parser, suggester, etc
// Submit your request via the client and get back a response containing search results
SearchResponse searchResponse = searchClient.Search(searchRequest);
I'm trying to use Lucene 4.8.1's SearchAfter methods to implement paging of search results in a web application.
A similar question has been asked before, but the accepted answer given there does not work for me:
Stack Overflow Question: Lucene web paging
When I create a Lucene ScoreDoc from scratch in this way to use as an argument for SearchAfter:
ScoreDoc sd = new ScoreDoc(14526, 0.0f);
TopDocs td = indexSearcher.searchAfter(sd, query, null, PAGEHITS);
I get this exception:
java.lang.IllegalArgumentException: after must be a FieldDoc
This appears contrary to the documentation. But in any case, when I create a Field Doc instead, I get:
java.lang.IllegalArgumentException: after.fields wasn't set
after.fields is an Object array, so I can hardly set that with information I can pass in a URI!
I cannot find any working code examples using SearchAfter. My original plan was obviously to create a new ScoreDoc as the previous question suggests. Can anybody suggest what I might be doing wrong, or link to any working code examples of SearchAfter?
Thanks!
I don't believe you can create a scoredoc and then pass it to searchAfter. You need to use the ScoreDocs returned from a previous search.
can you have a try.
#Test
public void searchAfter() {
Object[] objects = new Object[]{"1"};
List<Map<String, Object>> data = new ArrayList<Map<String, Object>>();
boolean type = true;
while (type) {
SearchHits searchHits = searchAfter(objects);
SearchHit[] hits = searchHits.getHits();
if (hits != null && hits.length > 0){
objects = hits[hits.length-1].getSortValues();
if (hits.length < size) type = false;
for (SearchHit hit : hits) {
data.add(hit.getSourceAsMap());
System.out.println(JsonUtil.objectToJson(hit.getSourceAsMap()));
}
}
}
Iterator<Map<String, Object>> iterator = data.iterator();
while (iterator.hasNext()) {
System.out.println(iterator.next().toString());
}
System.out.println(data.size() + "-----------------");
}
public SearchHits searchAfter(Object[] objects) {
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.query(QueryBuilders.termQuery("age", "33"));
sourceBuilder.size(size);
sourceBuilder.sort("account_number", SortOrder.ASC);
sourceBuilder.searchAfter(objects);
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("bank");
searchRequest.source(sourceBuilder);
ActionFuture<SearchResponse> response = elasticsearchTemplate.getClient().search(searchRequest);
SearchHits searchHits = response.actionGet().getHits();
return searchHits;
}