querying on field missing from some document in elasticsearch - java

I have a special need where in I need to query elastic search for a field say F1.
F1 is not there in all documents . My search should be for missing F1 documents and F1 with some value.
F1 is missing or F1 = 'A1'
I am not sure if this is possible in elastic search. Here is my query and I know that is not right query. Would appreciate somebody could correct it or the Java program.
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"bool": {
"should": {
"terms": {
"F1": [
"Some Value"
]
}
}
}
},
{
"missing": {
"field": "F1"
}
}
]
}
}
}
}
}
This is my java code building the query.
public QueryBuilder getQueryBuilder() {
QueryBuilder qb;
//what do we want to return if there is no request? Nothing or all.
if (filters == null || filters.isEmpty()) {
qb = QueryBuilders.matchAllQuery();
} //build our filtered query using given filters in our request
else {
Set<String> keys = filters.keySet();
BoolFilterBuilder boolFilterBuilder = FilterBuilders.boolFilter();
for (String key : keys) {
Set<Object> values = filters.get(key);
if (values == null || values.toString().isEmpty()) {
continue; //Ignore nothing to do.
}
if (key.equalsIgnoreCase(Constants.MISSING_FILTER)) {
Iterator i = values.iterator();
while (i.hasNext()) {
boolFilterBuilder.must(FilterBuilders.missingFilter((String) i.next()));
}
} else {
boolFilterBuilder.must(buildShouldQuery(key, values));
}
}
qb = QueryBuilders.filteredQuery(createSearchQueryBuilder(), boolFilterBuilder);
}
return qb;
}


Try whit this.
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"should": [
{ "terms": { "F1": ["Some Value"] },
{ "missing": { "field": "F1" } }
]
}
}
}
}
}

Related

Convert mongo Query to Java ArrayList<Bson>

I need help in converting below mongo query to an ArrayList so that I can execute the query in spring boot app. Having tough time to convert it in java using Aggregate classe's match, group, etc. method.
private List aggregateInspectionWorkOrders(ArrayList operations) {
MongoCollection mongoCollection = db().getCollection(Constants.DB.MyTable, InspectionWorkOrder.class);
return mongoCollection.aggregate(operations).into(new ArrayList<>());
}
db.mytable.aggregate([
{
$match: {
$and: [
{
deleted: { $eq: false }
},
{
status: {
$in: ['O', 'IP']
}
}
]
}
},
{
$group: {
_id: {
sub: '$sub',
pre: '$pre'
},
location: { $addToSet: { mile: '$mp'}}
}
},
{
$group: {
_id: {
sub: '$_id.sub'
},
location: {
$addToSet: {
pre: '$_id.pre',
from: {
$min: '$location.mp'
},
to: {
$max: '$location.mp'
}
}
}
}
},
{
$project: {
_id: 0,
sub: '$_id.sub',
location: '$location'
}
}
]).pretty()

Elasticsearch: sort by a numberic field

I'm coding a console app to Insert data from JSOIN files into Elasticsearch 7.5.1. The _id field should be an int and act like auto-increment. To make that happen, before inserting I get the last ID inserted and increment 1.
My problem is that I'm having trouble getting the last ID, because the ordering is happening on a string. This is what I mean: if you have 10 items and the last ID is 10, when querying and sorting it will return 9.
This is my query when using Postman:
GET my_index/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort": [{
"_id": {
"order": "desc"
}
}
]
}
And my Java function using their client:
private static String getLastElasticSearchId(String index)
{
RestHighLevelClient client = getElasticSearchClient();
SearchRequest searchRequest = new SearchRequest(index);
SearchSourceBuilder b = new SearchSourceBuilder();
b.query(QueryBuilders.matchAllQuery());
b.sort(new FieldSortBuilder("_id").order(SortOrder.DESC));
b.from(0);
b.size(1);
searchRequest.source(b);
try {
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = searchResponse.getHits();
if(hits.getTotalHits().value > 0){
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
return hit.getId();
}
}
else {
return "0";
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
client.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return null;
}
Both return 9 as the last ID, even having another item in the index with 10 as ID.
I cannot apply string padding in the IDs as answered here.
How can I achieve what I need?

Try this:
GET my_index/_search
{
"size": 1,
"query": {
"match_all": {}
},
"sort" : {
"_script" : {
"type" : "number",
"script" : {
"lang": "painless",
"source": "Integer.parseInt(doc['_id'].value)"
},
"order" : "desc"
}
}
}
Hope this helps

how to parse this nested json response?

I am getting json response in this format.
{
"apiGroups":
{
"Affiliate":
{
"listingsAvailable":
{
"Beauty_Personal_Care":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:586:821655440?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Eyewear":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:473:662748456716?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Real_Estate":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:897:673143570606?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Jewellery":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:6:315773046?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Furniture":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:580:1894930153?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Tweens_Boys":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:814:934253466?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Automobiles":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:1145:639299259208?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Home_Improvement":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:864:624389489778?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"The_Designer_Studio":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:924:655684426383?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
"Fashion_Jewellery":
{
"listingVersions":
{
"v1":
{
"get": "http://affiliate-feeds.snapdeal.com/feed/api/category/v1:1113:672114192240?expiresAt=1446085800024&signature=civtucyhsbufsjzjvqfa"
}
}
},
I need to get categories like beauty personal care, eye ware and their respective urls in get field.How can i loop through this and get the.So far i tried like this and dont no how to proceed next.Can anybody give me suggestions how to parse this json?
json = jParser.getJSONFromUrl(response);
JSONObject api = json.getJSONObject("apiGroups");
JSONObject affiliate = api.getJSONObject("Affiliate");
JSONObject list = affiliate.getJSONObject("listingsAvailable");

You can read the documentation about the JSONObject class in Android.
In this documentation, you will find the method keys that will "Returns an iterator of the String names in this object."
So you just have to call this method and use the iterator.
Iterator<String> keysIterator = jsonObject.keys();
String key;
while (keysIterator.hasNext()) {
key = keysIterator.next();
//use the key to retrieve the data form jsonObject
}
However, if you are the one generating this json, you may consider changing it a bit. The data in the listingsAvailable should probably be in an array.

Recursively parsing JSON via JSONObject to fetch value against specific keys

I have a complex JSON as below which I need to parser recursively. The end result of recursion is Map> type of object where key is the audience - name value and the inner map is Text-key, Title-value.
This is just a part of the complete JSON.
"sections": {
"1": {
"1": {
"1": {
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
}
},
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
}
},
"2": {
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
}
},
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
},
"2": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
}
},
"2": {
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
},
"anchor":"xxx"
},
"3": {
"1": {
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"tag": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
},
"styleHint": {
"tag": {
"name": "xxx",
"title": "xxx",
"id": "xxx"
}
}
}
},
"title": "xxx",
"text": "xxx",
"tags": {
"audience": {
"1": {
"name": "xxx",
"title": "xxx",
"id": "xxxx"
}
},
"styleHint": {
"1": {
"name": "xx",
"title": "xxx",
"id": "xxxx"
}
}
}
}
}
I used JSONObject for this only to realise very late that iteration happens in reverse order :(
I tried to parse the whole structure recursively and reverse it to my benefit. BUt the order is going haywire :( :( mainly because of the text, title, snippet which follows the 2nd text,title and has 2 audience names. The text and title of that part get skipped due to which the whole order is compromised
Please help !! my current implementation is as below
private Map<String, Map<String, String>> parseTextAndTitle(JSONObject json,
Map<String, Map<String, String>> ttMap, String article,
List<String> usrGrp) throws JSONException {
logger.info("Entering method..");
String userGroup = null;
Map<String, String> titleAndText = new LinkedHashMap<String, String>();
Map<String, String> currMap = new LinkedHashMap<String, String>();
Map<String, String> tempMap = new LinkedHashMap<String, String>();
Iterator<String> keys = json.sortedKeys();
while (keys.hasNext()) {
String key = keys.next();
JSONObject value = null;String firstKey = null;
String text = null;String title = null;
int length = 0;
try {
value = json.getJSONObject(key);
if (key.equalsIgnoreCase(STYLEHINT) || key.equalsIgnoreCase(ANCHOR)
|| key.equalsIgnoreCase(INLINE)) {
continue;
}
if (key.equals(TEXT)) {
text = json.getString(key);
text = removeHtmlTag(text);
logger.debug("TEXT RETRIEVED:" + text);
if(text != null) {
titleAndText.put(text, "");
}
else
logger.debug("Text not retrieved!!");
}
if (key.equals(TITLE)) {
title = json.getString(TITLE);
title = appendNewline(title);
logger.debug("TITLE RETRIEVED:" + title);
if (title != null) {
for (Map.Entry<String, String> iter : titleAndText
.entrySet())
firstKey = iter.getKey();
if(firstKey != null) {
titleAndText.put(firstKey, title);
}
else
logger.debug("NO key present in textAndTitle Map!!");
}
}
if (key.equals(AUDIENCE_TAG)) {
try {
length = value.length();
for (int i = 0; i < length; i++) {
userGroup = (String) value.getJSONObject(
String.valueOf(i + 1)).get(NAME);
logger.debug("USERGROUP RETRIEVED:" + userGroup);
usrGrp.add(userGroup);
}
} catch (Exception e) {
userGroup = (String) value.getJSONObject(TAG).get(NAME);
logger.debug("USERGROUP RETRIEVED:" + userGroup);
usrGrp.add(userGroup);
}
}
else{
parseTextAndTitle(value, ttMap, article, usrGrp);
}
} catch (Exception e) {
logger.debug("value not a JSON Object..rather an element");
// Extract the text values
if (key.equals(TEXT)) {
text = json.getString(key);
text = removeHtmlTag(text);
logger.debug("TEXT RETRIEVED:" + text);
if(text != null) {
titleAndText.put(text, "");
}
else
logger.debug("Text not retrieved!!");
}
if (key.equals(TITLE)) {
title = json.getString(TITLE);
title = appendNewline(title);
logger.debug("TITLE RETRIEVED:" + title);
if (title != null) {
for (Map.Entry<String, String> iter : titleAndText
.entrySet())
firstKey = iter.getKey();
if(firstKey != null) {
titleAndText.put(firstKey, title);
}
else
logger.debug("NO key present in textAndTitle Map!!");
}
}
}
if (!(usrGrp.isEmpty()) && !(titleAndText.isEmpty())
&& title != null) {
if(usrGrp.size() > 1)
{
for(int i=0;i<usrGrp.size();i++)
{
//If user group already present, extract current text,title map
//If not put usergroup as key, text,title map as value
if (ttMap.containsKey(usrGrp.get(i))) {
currMap = ttMap.get(usrGrp.get(i));
if (currMap.isEmpty()) {
ttMap.put(usrGrp.get(i), titleAndText);
} else {
currMap = ttMap.get(usrGrp.get(i));
for (Map.Entry<String, String> entry : currMap
.entrySet()) {
tempMap.put(entry.getKey(),
(String) entry.getValue());
}
for (Map.Entry<String, String> ttEntry : titleAndText
.entrySet()) {
tempMap.put(ttEntry.getKey(),
(String) ttEntry.getValue());
}
ttMap.put(usrGrp.get(i),tempMap);
// titleAndText = new LinkedHashMap<String, String>();
tempMap = new LinkedHashMap<String, String>();
}
}
else {
ttMap.put(usrGrp.get(i), titleAndText);
}
}
titleAndText.clear();
}
else
{
if (ttMap.isEmpty())
{
tempMap = titleAndText;
ttMap.put(usrGrp.get(0), tempMap);
}
else {
currMap = ttMap.get(usrGrp.get(0));
if (currMap.isEmpty()) {
ttMap.put(usrGrp.get(0), titleAndText);
}else {
currMap = ttMap.get(usrGrp.get(0));
for (Map.Entry<String, String> entry : currMap
.entrySet()) {
tempMap.put(entry.getKey(),
(String) entry.getValue());
}
for (Map.Entry<String, String> ttEntry : titleAndText
.entrySet()) {
tempMap.put(ttEntry.getKey(),
(String) ttEntry.getValue());
}
ttMap.put(usrGrp.get(0),tempMap);
titleAndText.clear();
}
}
}
usrGrp.clear();
}
}
logger.info("Exiting method..");
return ttMap;
}

Modified #sklimkovitch code to get it working in some complex Json Structure...
public void loopThroughJson(Object input) throws JSONException {
if (input instanceof JSONObject) {
Iterator<?> keys = ((JSONObject) input).keys();
while (keys.hasNext()) {
String key = (String) keys.next();
if (!(((JSONObject) input).get(key) instanceof JSONArray))
if (((JSONObject) input).get(key) instanceof JSONObject) {
loopThroughJson(((JSONObject) input).get(key));
} else
System.out.println(key + "=" + ((JSONObject) input).get(key));
else
loopThroughJson(new JSONArray(((JSONObject) input).get(key).toString()));
}
}
if (input instanceof JSONArray) {
for (int i = 0; i < ((JSONArray) input).length(); i++) {
JSONObject a = ((JSONArray) input).getJSONObject(i);
loopThroughJson(a);
}
}
}

package Test.json;
import java.util.Iterator;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;
public class App {
public static void main(String[] args) {
String str = "{\"a\":\"1\", \"b\":\"2\", \"c\":[{\"d\":\"4\"},{\"e\":\"5\"},{\"f\":[{\"g\":\"6\"},{\"h\":\"7\"}]}], \"i\":8}";
try {
loopThroughJson(new JSONObject(str));
} catch (JSONException e) {
e.printStackTrace();
}
}
public static void loopThroughJson(Object input) throws JSONException {
if (input instanceof JSONObject) {
Iterator<?> keys = ((JSONObject) input).keys();
while (keys.hasNext()) {
String key = (String) keys.next();
if (!(((JSONObject) input).get(key) instanceof JSONArray))
System.out.println(key + "=" + ((JSONObject) input).get(key));
else
loopThroughJson(new JSONArray(((JSONObject) input).get(key).toString()));
}
}
if (input instanceof JSONArray) {
for (int i = 0; i < ((JSONArray) input).length(); i++) {
JSONObject a = ((JSONArray) input).getJSONObject(i);
Object key = a.keys().next().toString();
if (!(a.opt(key.toString()) instanceof JSONArray))
System.out.println(key + "=" + a.opt(key.toString()));
else
loopThroughJson(a.opt(key.toString()));
}
}
}
}
Output:
a=1
b=2
d=4
e=5
g=6
h=7
i=8

Instead of
while (keys.hasNext()) {
<blah blah>
if (key.equalsIgnoreCase(STYLEHINT) || key.equalsIgnoreCase(ANCHOR)
|| key.equalsIgnoreCase(INLINE)) {
continue;
}
if (key.equals(TEXT)) {
<blah blah>
}
if (key.equals(TITLE)) {
....
One can simply code:
text = json.getString(TEXT);
<deal with text>
title = json.getString(TITLE);
<etc>
If it's possible that the some of the key values are not there, simply test for their absence with has before fetching them.
Since STYLEHINT, ANCHOR, and INLINE are ignored, simply don't fetch them.
To handle the screwy layout of the JSON, do this:
if (json.has("title")) {
<extract title/text/tags/stylehint as described above>
}
else {
Iterator<String> keys = json.sortedKeys();
while (keys.hasNext()) {
// Note that "key" must be "1", "2", "3"...
String key = keys.next();
value = json.getJSONObject(key);
<recursively call method using "value">
}
}

Found a solution to the ordering..ditched JSONObject API and used gson JsonObject instead
private Map<String, List<String>> parseJsonSection(
Map<String, List<String>> retTextMap, JsonObject jsonObject,
String lastKey, StringBuffer tt, List<String> ttext)
throws ParseException, JSONException {
for (Entry<String, JsonElement> entry : jsonObject.entrySet()) {
String key = entry.getKey();
Object value = entry.getValue();
logger.debug("Key:" + key + "\n" + value.toString());
if (key.equalsIgnoreCase(STYLEHINT) || key.equalsIgnoreCase(INLINE)
|| key.equalsIgnoreCase(ANCHOR))
continue;
if (key.equalsIgnoreCase(TEXT)) {
tt.append(value.toString());
ttext.add(tt.toString());
}
if (key.equalsIgnoreCase(TITLE) && tt.length() == 0) {
tt = new StringBuffer();
tt.append(value.toString() + "-");
}
if (key.equalsIgnoreCase(NAME)) {
logger.debug("Value of usergrp:" + value.toString());
String usrGrp = value.toString();
if (retTextMap.isEmpty()) {
if (tt.toString() != null) {
List<String> temp = new ArrayList<String>();
temp = ttext;
retTextMap.put(usrGrp, temp);
}
return retTextMap;
} else if (retTextMap.get(usrGrp) != null) {
List<String> temp = retTextMap.get(value.toString());
if (!temp.contains(tt.toString()))
temp.add(tt.toString());
retTextMap.put(usrGrp, temp);
} else if (retTextMap.get(usrGrp) == null) {
if (tt != null) {
List<String> temp = new ArrayList<String>();
temp.add(tt.toString());
retTextMap.put(usrGrp, temp);
return retTextMap;
}
}
}
if (value instanceof JsonObject) {
parseJsonSection(retTextMap, (JsonObject) value, key, tt, ttext);
}
}
return retTextMap;
}

extract url from facebook FQL

I am stuck here and would like to extract using java the second link of the facebook query below
{
"data": [
{
"attachment": {
"media": [
{
"photo": {
"images": [
{
"src": "https://fbcdn-photos-h-a.akamaihd.net/hphotos-ak-prn2/1508634_699393523428883_996610253_s.png"
},
{
"src": "https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-prn2/s720x720/1508634_699393523428883_996610253_n.png"
}
]
}
}
]
}
}
]
}
my code below, is obviously not working
try
{
List<JsonObject> queryResults = facebookClient.executeFqlQuery(query, JsonObject.class);
if(!queryResults.isEmpty())
{
JsonObject facebookPosturl_J = queryResults.get(0);
facebook_post = facebookPosturl_J.getString("src");
}
}
catch (Exception e){logger.warn("Unexpected error", e);}

Try calling:
facebookPosturl_J.getJsonArray("data").getJsonObject(0).getJsonObject("attachment").getJsonArray("media").getJsonObject(0).getJsonObject("photo").getJsonArray("images").getJsonObject(1).getString("src")

Categories