How to access nested json in elasticsearch? - java

I have following json:
metadata: {
authors: [
],
links: [
{
href: "http://www.latimes.com/opinion/readersreact/la-le-1028-wednesday-meat-cancer-20151028-story.html#navtype=outfit",
value: "Why hot dogs and bacon aren't as dangerous as cigarettes"
},
{
href: "http://www.latimes.com/opinion/readersreact/la-le-1028-wednesday-porter-ranch-lausd-20151028-story.html#navtype=outfit",
value: "LAUSD school in Porter Ranch shows the importance of parent involvement"
},
{
href: "http://www.latimes.com/opinion/readersreact/la-le-1028-wednesday-billboards-20151028-story.html#navtype=outfit",
value: "Maine and Vermont show L.A. what life is like without billboards"
},
{
href: "http://www.latimes.com/opinion/readersreact/la-le-1028-wednesday-broad-beach-20151028-story.html#navtype=outfit",
value: "Malibu beach-front homeowners, meet King Canute"
}
]
},
I would like to search only for metadata.links.value in elasticsearch:
requestBuilder.setQuery(QueryBuilders.matchQuery("metadata.links.value", "Malibu"));
But unfortunately this doesn't work. I get 0 hits when i enter a value.
What am i doing wrong?
Update:
Here is my complete code
public List<ArticleExtraction> search(String searchQuery, SearchProvider searchProvider) {
TransportClient client = searchProvider.getClient();
Map<String, String> query = new HashMap<>();
ArrayList<String> singleQuery = new ArrayList<>();
if (searchQuery.length() > 0 && searchQuery.contains(":")) {
String[] queries = searchQuery.split(",");
for (String q : queries) {
String[] jsonQuery = q.split(":");
query.put(jsonQuery[0], jsonQuery[1]);
}
} else {
String[] queries = searchQuery.split(",");
for (String q : queries) {
singleQuery.add(q);
}
}
SearchRequestBuilder requestBuilder = client.prepareSearch("crawlbot")
.setTypes("item")
.setSize(100);
for (Map.Entry<String, String> e : query.entrySet()) {
requestBuilder.setQuery(QueryBuilders.matchQuery(e.getKey(), e.getValue()));
}
for (String q : singleQuery) {
requestBuilder.setQuery(QueryBuilders.queryStringQuery(q));
}
SearchResponse response = requestBuilder.execute().actionGet();
List<ArticleExtraction> articles = new ArrayList<>();
SearchHit[] hits = response.getHits().getHits();
for (SearchHit hit : hits) {
String sourceAsString = hit.getSourceAsString();
if (sourceAsString != null) {
JsonObject json = new JsonParser().parse(sourceAsString).getAsJsonObject();
if (json.has("article")) {
Gson gson = new Gson();
articles.add(gson.fromJson(json.get("article"), ArticleExtraction.class));
}
}
}
return articles;
Explanation:
The input of the searchQuery could be something like:
metadata.links.value:malibu
Or if it is a singlequery: malibu
I made some code so both queries can get accepted
Mappings (sry if this gets big)
mappings: {
item: {
properties: {
article: {
properties: {
description: {
type: "string"
},
description_html: {
type: "string"
},
entities: {
properties: {
count: {
type: "long"
},
meta: {
type: "object"
},
name: {
type: "string"
},
type: {
type: "string"
}
}
},
favicon_url: {
type: "string"
},
images: {
properties: {
colors: {
properties: {
color: {
type: "long"
}
}
},
entropy: {
type: "double"
},
height: {
type: "long"
},
url: {
type: "string"
},
width: {
type: "long"
}
}
},
keywords: {
properties: {
label: {
type: "string"
},
score: {
type: "double"
}
}
},
language: {
type: "string"
},
metadata: {
properties: {
authors: {
properties: {
name: {
type: "string"
}
}
},
links: {
properties: {
href: {
type: "string"
},
value: {
type: "string"
}
}
},
twitter: {
type: "string"
}
}
},
provider_display: {
type: "string"
},
provider_name: {
type: "string"
},
provider_url: {
type: "string"
},
published: {
type: "string"
},
published_long: {
type: "long"
},
summary: {
type: "string"
},
title: {
type: "string"
},
url: {
type: "string"
}
}
},
id: {
properties: {
_inc: {
type: "long"
},
_machine: {
type: "long"
},
_new: {
type: "boolean"
},
_time: {
type: "long"
}
}
},
job: {
properties: {
api: {
type: "long"
},
crawl_depth: {
type: "long"
},
max_pages: {
type: "long"
},
name: {
type: "string"
},
status: {
type: "long"
},
url: {
type: "string"
},
userid: {
type: "long"
}
}
},
query: {
properties: {
match: {
properties: {
name: {
type: "string"
}
}
}
}
}
}
}
},

metadata is contained within the article root object.
Therefore your query should be constructed as:
QueryBuilders.matchQuery("article.metadata.links.value"‌​, "Malibu");

Related

Mongodb - aggregate match within attribute value

MongoDB Data:
{
"_id" : ObjectId("123"),
"attr" : [
{
"nameLable" : "First Name",
"userEnteredValue" : [
"Amanda"
],
"rowNumber":"1"
},
{
"nameLable" : "Last Name",
"userEnteredValue" : [
"Peter"
],
"rowNumber":"1"
},
{
"nameLable" : "First Name",
"userEnteredValue" : [
"Sandra"
],
"rowNumber":"2"
},
{
"nameLable" : "Last Name",
"userEnteredValue" : [
"Peter"
],
"rowNumber":"2"
}
]
}
Matching (First Name equals "Amanda" && Last Name equals "Peter") -> Match should happen within rowNumber so that i will get rowNumber1 record but now i am getting both rows as "Peter" happens to be in both "rowNumber" attribute.
Criteria Code:
Criteria cr = Criteria.where("attr").elemMatch(Criteria.where("nameLable").is(map.get("value1")).and("userEnteredValue").regex(map.get("value2").trim(), "i"); //Inside loop
AggregationOperation match = Aggregation.match(Criteria.where("testId").is("test").andOperator(cr.toArray(new Criteria[criteria.size()])));
DB Query for above search Criteria Match:
db.Col1.aggregate([
{
"$match":{
"testId":"test",
"$and":[
{
"attr":{
"$elemMatch":{
"nameLable":"First Name",
"userEnteredValue":{
"$regex":"Amanda",
"$options":"i"
}
}
}
},
{
"attr":{
"$elemMatch":{
"nameLable":"Last Name",
"userEnteredValue":{
"$regex":"Peter",
"$options":"i"
}
}
}
}
]
}
}
]
)
Please let me know how can we do match within "rowNumber" attribute.
Let me start by recommending you reconsider your document structure, I do not know your product but this structure is very unique and definitely makes most "simple" access patterns I can think of to very cumbersome to execute. This will be noticeable in my answer.
So the current query you have just required 2 separate elements in the array exist, as you mentioned you want the same rowNumber, due to the document structure this isn't really queryable, we will have to first use your query to match "potential" matching documents. At that point we can filter our the matched rows and see if we have both a first name and a last name matching.
Finally we could filter out the none matching rows from the result, here is the pipeline:
db.collection.aggregate([
{
"$match": {
"testId": "test",
"$and": [
{
"attr": {
"$elemMatch": {
"nameLable": "First Name",
"userEnteredValue": {
"$regex": "Amanda",
"$options": "i"
}
}
}
},
{
"attr": {
"$elemMatch": {
"nameLable": "Last Name",
"userEnteredValue": {
"$regex": "Peter",
"$options": "i"
}
}
}
}
]
}
},
{
$addFields: {
goodRows: {
"$setIntersection": [
{
$map: {
input: {
$filter: {
input: "$attr",
cond: {
$and: [
{
$eq: [
"$$this.nameLable",
"First Name"
]
},
{
"$regexMatch": {
"input": {
"$arrayElemAt": [
"$$this.userEnteredValue",
0
]
},
"regex": "Amanda",
"options": "i"
}
}
]
}
}
},
in: "$$this.rowNumber"
}
},
{
$map: {
input: {
$filter: {
input: "$attr",
cond: {
$and: [
{
$eq: [
"$$this.nameLable",
"Last Name"
]
},
{
"$regexMatch": {
"input": {
"$arrayElemAt": [
"$$this.userEnteredValue",
0
]
},
"regex": "Peter",
"options": "i"
}
}
]
}
}
},
in: "$$this.rowNumber"
}
}
]
}
}
},
{
$match: {
$expr: {
$gt: [
{
$size: "$goodRows"
},
0
]
}
}
},
{
$addFields: {
attr: {
$filter: {
input: "$attr",
cond: {
$in: [
"$$this.rowNumber",
"$goodRows"
]
}
}
}
}
}
])
Mongo Playground

How to get single field in mongodb query?

I have data like this:
{ id : 1,
book: "Flash",
chapters: [
{
chap_no: "1",
sub_chapter: [
{sub_no: 1, description: "<description>"
},
{sub_no: 2, description: "<description>"
},
]
}
]
}
i want to show one field like this base on book -> chapter_no -> sub_no
{
sub_no: 2, description: "<description>"
}
in mongodb query.
$match
$unwind
$unwind
$match
$replaceRoot
db.collection.aggregate([
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
"chapters.sub_chapter.sub_no": 2
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])
mongoplayground
you can make like this
db.collection.aggregate([
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$unwind": "$chapters"
},
{
"$unwind": "$chapters.sub_chapter"
},
{
"$match": {
$and: [
{
"book": "Flash3"
},
{
"chapters.chap_no": "2"
},
{
"chapters.sub_chapter.sub_no": "1"
}
]
}
},
{
"$replaceRoot": {
"newRoot": "$chapters.sub_chapter"
}
}
])

How to find distinct value with count of other array inside dhe document in MongoDB

I want to get distinct categoryCode and categoryName while this document also contains list of accessories I want to count of this accessories in the response.
Sample data:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessories":[{"a_id":1},{"a_id":2}]
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessories":[{"a_id":1},{"a_id":2},{"a_id":3}]
},
{
"categoryCode":"categoryCode1",
"categoryName":"categoryNam1",
"accessories":[{"a_id":1},{"a_id":2}]
}
]
Expected result:
[
{
"categoryCode":"categoryCode1",
"categoryName":"categoryName1",
"accessoriesCount":2
},
{
"categoryCode":"categoryCod2",
"categoryName":"categoryName2",
"accessoriesCount":3
}
]
https://mongoplayground.net/p/q6AZOaTwo5a
db.collection.aggregate([
{
"$group": {
"_id": {
categoryCode: "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$addToSet": "$accessories"
}
}
},
{
"$project": {
categoryCode: "$_id.categoryCode",
categoryName: "$_id.categoryName",
accessoriesCount: {
$size: "$accessories"
},
_id: 0
}
}
])
Query
group to have the distinct values
$push the accesories arrays (we dont have $concat accumulator)
reduce those arrays to union them, keep only the distinct members, and take the count.
Test code here
db.collection.aggregate([
{
"$group": {
"_id": {
"categoryCode": "$categoryCode",
"categoryName": "$categoryName"
},
"accessories": {
"$push": "$accessories"
}
}
},
{
"$set": {
"accessoriesCount": {
"$size": {
"$reduce": {
"input": "$accessories",
"initialValue": [],
"in": {
"$setUnion": [
"$$value",
"$$this"
]
}
}
}
}
}
},
{
"$project": {
"_id": 0,
"categoryCode": "$_id.categoryCode",
"categoryName": "$_id.categoryName",
"accessoriesCount": "$accessoriesCount"
}
}
])

How to converted dataTable from request Spring Boot

i'm learning programming and want use DataTables and i have a problem. I don't know how to converted records...
I have code in spring boot
#RequestMapping(path="/seriess", method=RequestMethod.GET)
public Page<SeriesDao> showSeries(#RequestParam(defaultValue="0") int page)
{
Page<SeriesDao> sss = seriesRepository.findAll(new PageRequest(page, 20));
return sss;
}
#RequestMapping("/showSeries")
public ModelAndView model(){
ModelAndView model = new ModelAndView("showSeries");
return model;
}
Next I have json when i go to localhost:8080/seriess, i didn't copy all result (20)
{"content":[{"id":41,"name":"Average Weekly Earnings of All Employees: Total Private in Corpus Christi, TX (MSA)","file":"SMU48185800500000011.csv","cassid":"1d2e556b-031e-4c6f-aec4-981c4e907324","categoryid":3,"datefrom":"2006-12-31","dateto":"2016-09-30","frequency":5,"markers":null,"unit":"$ per Week","feed":"Macroeconomic_And_Major_Markets","createdate":1476567529000,"changedate":1483919401000}........and next 19 records ]
"last":false,"totalPages":25,"totalElements":488,"size":20,"number":0,"sort":null,"first":true,"numberOfElements":20}
This is my java script code. I have to correct convert this data from url, but i don't know how...
$(document).ready( function () {
$('#dataTable').DataTable({
"processing": true,
"serverSide": true,
"ajax": {
"url": "seriess",
"dataSrc" : ""
}
},
"columns": [
{ "data": "id"},
{ "data": "name" },
{ "data": "file" },
{ "data": "cassid" },
{ "data": "categoryid" },
{ "data": "datefrom" },
{ "data": "dateto" },
{ "data": "frequency" },
{ "data": "markers" },
{ "data": "unit" },
{ "data": "feed" },
{ "data": "createdate" },
{ "data": "changedate" }
]
})});
Change
"ajax": {
"url": "seriess",
"dataSrc" : ""
}
}
to
"ajax": {
"url": "/seriess",
"dataSrc" : "content"
}
}

Mongo Shell Query in Java

Is there any easy way of firing Mongo query in Java??
db.Test.aggregate(
[
{
'$match':
{
'o': { '$gt': [] }
}
},
{
'$project': {
'uid': 1,
'o': 1
}
},
{
'$project': {
'_id': 0,
'uid': 1,
o: {
$filter: {
input: "$o",
as: "item",
cond: {
$and: [
{
$lt: [ "$$item.ad", 0 ]
},
{
$lt: [ "$$item.at", 0 ]
}
]
}
}
}
}
},
{
'$match': {
'o': { '$gt': []}
}
},
{
$project: {
uid: 1,
"mids": "$o.mid"
}
},
{
$unwind: "$mids"
},
{
$group: {
_id: {
uid: "$uid",
mid: "$mids"
},
count: { $sum: 1 }
}
},
{
$project: {
_id: 0,
uid: "$_id.uid",
mid: "$_id.mid",
count: 1
}
}
]
);
Is http://jongo.org serve the purpose for complex queries?
As an alternative, you can use the Java driver's Document.parse() method. You can supply a JSON string to the method (following MongoDB's extended JSON formatting), and it will return a parsed BSON document for you.
Please see http://mongodb.github.io/mongo-java-driver/3.5/javadoc/org/bson/Document.html#parse-java.lang.String- for the method's documentation.

Categories