I am trying to learn QueryDSL in order to return the results of a single search term from Postgres:
#GetMapping("/product/contains/{matchingWords}")
public List<ProductModel> findByTitleContaining(#PathVariable String matchingWords) {
QProductModel product = QProductModel.productModel;
JPAQuery<?> query = new JPAQuery<>(em);
List<ProductModel> products = query.select(product)
.from(product)
.where(product.title.toLowerCase()
.contains(matchingWords.toLowerCase()))
.fetch();
return products;
}
But I also want to search for any number of search terms, for example, say this is my list of search terms divided by the plus symbol:
String[] params = matchingWords.split("[+]");
How can I dynamically create contains(params[0]) AND/OR contains(params[1] AND/OR ... contains(params[n]) using either QueryDSL or any Java/Spring query framework? I see QueryDSL has a predicate system, but I still don't understand how to dynamically create a query based on a variable number of parameters searching in the same column.
I figured it out. It's a little non-intuitive, but using BooleanBuilder and JPAQuery<?> you can create a dynamic series of boolean predicates, which return a list of matches.
Example:
QProductModel product = QProductModel.productModel;
JPAQuery<?> query = new JPAQuery<>(//entity manager goes here//);
// example list of search parameters separated by +
String[] params = matchingWords.split("[+]");
BooleanBuilder builder = new BooleanBuilder();
for(String param : params) {
builder.or(product.title.containsIgnoreCase(param));
}
// then you can put together the query like so:
List<ProductModel> products = query.select(product)
.from(product)
.where(builder)
.fetch();
return products;
My sample request
{
"requestModel":{
"CUSTID": "100"
},
"returnParameters":[
{
"name":"NETWORK/NETID",
"datatype":"String",
"order":"asc",
"sequence":1
},
{
"name":"INFODATA/NAME",
"datatype":"String",
"order":"asc",
"sequence":1
},
{
"name":"SOURCE/SYSTEM",
"datatype":"int",
"order":"asc",
"sequence":2
},
]
}
Sample Response
Below is my dynamically generated Map format of json response[Response parameters will be different each time based on the request params],
"responseModel":{
"documents": [
{
"NETWORK":[
{"NETID":"1234"},
{"ACT":"300"}
],
"SOURCE": {
"SYSTEM":"50"
},
"INFODATA":{
"NAME":"PHIL"
}
},
{
"NETWORK":[
{"NETID":"1234"},
{"ACT":"300"}
],
"SOURCE": {
"SYSTEM":"100"
},
"INFODATA":{
"NAME":"PHIL"
}
}
]
}
Problem Statement
I need to do multi level sorting based on the "returnParameters" in the request which is dynamic...
"order" indicates ascending (or) descending and sequence indicates the the priority for ordering like (group by in sql query)
Code
Map<String,Object> documentList = new HashMap<String,Object>();
JSONObject jsonObject= new JSONObject(response.getContent());
response.getContent() -> is nothing but it contains the above json response in Map format.
Now I converting the map to list of json object
JSONArray jsonArray= (JSONArray)jsonObject.get("documents");
ArrayList<JSONObject> list = new ArrayList<>();
for(int i=0;i<jsonArray.length();i++){
list.add((JSONObject) jsonArray.get(i));
}
Collections.sort(list, new ResponseSorter());
public class ResponseSorter implements Comparator<JSONObject> {
#Override
public int compare(JSONObject o1,JSONObject o2){
String s1= (String)((JSONObject) o1.get("NETWORK")).get("NETID");
String s2= (String)((JSONObject) o2.get("NETWORK")).get("NETID");
int i1=Integer.parseInt(s1);
int i2=Integer.parseInt(s2);
return i1-i2;
}
}
I'm stuck here to proceed further. Created one for Integer comparator, .Should I create for each dataType? also
I need to dynamically construct the composite comparator by parsing the "retunrParameters" , below sample is hard coded, how to create dynamically??
(String)((JSONObject) o1.get("NETWORK")).get("NETID"); -> this should be dynamically framed , since "returnParameters" are also dynamic in nature.[NETWORK & NETID may not be come in another request],so my comparator should be capable enough to frame the keys in runtime
Would anyone able to assist me to create composite comparator in runtime for sorting?
NOTE:- Java Pojo cannot be created as the response is dynamic nature
In your case a simple comparator that's provided with the sort parameters might be easier to understand than a bunch of nested comparators.
Basically you'd do something like this:
class ReturnParameterComparator implements Comparator<JSONObject> {
private List<ReturnParameter> params; //set via constructor
public int compare( JSONObject left, JSONObject right) {
int result = 0;
for( ReturnParameter p : params ) {
//how exactly you get those values depends on the actual structure of your data and parameters
String leftValueStr = left.get( p );
String rightValueStr = right.get( p );
switch( p.datatype ) {
case "String":
result = String.compare( leftValueStr, rightValueStr );
break;
case "int":
//convert and then compare - I'll leave the rest for you
}
//invert the result if the order is descending
if( "desc".equals(p.order ) {
result += -1;
}
//the values are not equal so return the order, otherwise continue with the next parameter
if( result != 0 ) {
return result;
}
}
//at this point all values are to be considered equal, otherwise we'd have returned already (from the loop body)
return 0;
}
}
Note that this is just a stub to get you started. You'll need to add quite a few things:
how to correctly use the parameters to extract the values from the json objects
how to convert the data based on the type
how to handle nulls, missing or incompatible data (e.g. if a value should be sorted as "int" but it can't be parsed)
Adding all those would be way too much for the scope of this question and depends on your data and requirements anyway.
EDITED after additional questions in comments and additional info in description
You have a couple of steps you need to do here to get to the solution:
You want to have the sorting be dynamic based on the value of the property sequence in the request. So you need to parse the names of those returnParameters and put them in order. Below I map them to a List where each String[] has the name and order (asc/desc). The list will be ordered using the value of sequence:
List<String[]> sortParams = params.stream() // params is a List<JSONObject>
.filter(json -> json.containsKey("sequence")) // filter those that have "sequence" attribute
.sorted( sequence ) // sorting using Comparator called sequence
.map(jsonObj -> new String[]{jsonObj.get("name").toString(), jsonObj.get("order").toString()} )
.collect(Collectors.toList());
Before this you'll map the objects in the returnParameters array in the request to a List first.Then the stream is processed by 1. filtering the JSONObjects to only keep those that have prop sequence, 2. sorting the JSONObjects using comparator below. 3. from each JSONObject get "name" & "order" and put them in a String[], 4. generate a list with those Arrays. This list will be ordered in the order of attributes with priority 1 first, then priority 2, etc, so it will be ordered in the same way you want the JSONObjects ordered in the end.
Comparator<JSONObject> sequence = Comparator.comparingInt(
jsonObj -> Integer.valueOf( jsonObj.get("sequence").toString() )
);
So for your example, sortParams would look like: List( String[]{"NETWORK/NETID", "asc"}, String[]{""INFODATA/NAME", "asc"}, String[]{"SOURCE/SYSTEM", "asc"} )
Then you need to write a method that takes two params: a JSONObject and a String (the path to the property) and returns the value of that property. Originally I advised you to use JSONAware interface and then figure out the sub-class, but let's forget about that for now.
I am not going to write this method for you. Just keep in mind that .get(key) method of JSON.Simple always yields an Object. Write a method with this signature:
public String findSortValue(JSONObject doc, String path){
// split the path
// find the parent
// cast it (parent was returned as an Object of type Object)
// find the child
return value;
}
Write a generic individual comparator (that compares values of just one sort attribute at a time) and figures out if it's an Int, Date or regular String. I would write this as a regular method so it'll be easier to combine everything later on. Since you had so many questions about this I've made an example:
int individualComparator(String s1, String s2){
int compResult = 0;
try{
int numeric1 = Integer.parseInt(s1);
int numeric2 = Integer.parseInt(s2);
compResult = numeric1 - numeric2; // if this point was reached both values could be parsed
} catch (NumberFormatException nfe){
// if the catch block is reached they weren't numeric
try{
DateTime date1 = DateTime.parse(s1);
DateTime date2 = DateTime.parse(s2);
compResult = date1.compareTo(date2); // compareTo method of joda.time, the library I'm using
} catch (IllegalArgumentException iae){
//if this catch block is reached they weren't dates either
compResult = s1.compareTo(s2);
}
}
return compResult;
};
Write an overall Comparator that combines everything
Comparator<JSONObject> overAllComparator = (jsonObj1, jsonObj2) -> {
List<String[]> sortValuesList = sortParams.stream()
.map(path -> new String[]{ findValueByName(jsonObj1, path), findValueByName(jsonObj2, path) } )
.collect(Collectors.toList());
//assuming we always have 3 attributes to sort on
int comp1 = individualComparator(sortValuesList.get(0)[0], sortValuesList.get(0)[1]);
int comp2 = individualComparator(sortValuesList.get(1)[0], sortValuesList.get(1)[1]);
int comp3 = individualComparator(sortValuesList.get(2)[0], sortValuesList.get(2)[1]);
int result = 0;
if (comp1 != 0){
result = comp1;
} else if (comp2 != 0){
result = comp2;
} else{
result = comp3;
}
return result;
};
This Comparator is written lambda-style, for more info https://www.mkyong.com/java8/java-8-lambda-comparator-example/ .
First it takes the ordered list of sortParams we made in step 1 and for each returns an array where position 0 has the value for jsonObj1, and position 1 has the value for jsonObj2 and collects it in sortValuesList. Then for each attribute to sort on, it get the result of the individualComparatormethod. Then it goes down the line and returns as result of the overall comparison the first one that doesn't result in 0 (when a comparator results in 0 both values are equal).
The only thing that's missing now is the asc/desc value from the request. You can add that by chainingint comp1 = individualComparator(sortValuesList.get(0)[0], sortValuesList.get(0)[1]); with a simple method that takes an int & a String and multiplies the int by -1 if the String equals "desc". (Remember that in sortParams we added the value for order on position 1 of the array).
Because the first list we made, sortParams was ordered based on the priority indicated in the request, and we always did evertything in the order of this list, the result is a multi-sort in this order. It is generic & will be determined dynamically by the contents of returnParams in the request. You can apply it to your list of JSONObjects by using Collections.sort()
My suggestion: learn about:
Comparator.comparing which allows you to build your comparator by specifying the key extractor
Comparator.thanComparing which allows you to chain multiple comparators. The comparators later in the chain are called only if predecessors say the objects are equal
A tutorial if you need one: https://www.baeldung.com/java-8-comparator-comparing
I'm trying to use a SimpleQuery to get distinct result from my solr collection, but even after setting my StatsOption with calcDistinct true, I can't get the result I want.
BTW I'm using spring-data-solr-2.1.4.RELEASE.
SampleCode:
Field field = new SimpleField("fieldName");
StatsOptions statsOptions = new StatsOptions().addField(field).setCalcDistinct(true);
SimpleQuery query = new SimpleQuery("*:*").setStatsOptions(statsOptions);
StatsPage<MyClass> statsPage = solrTemplate.queryForStatsPage(query, MyClass.class);
FieldStatsResult statsResult = statsPage.getFieldStatsResult(field);
Collection<Object> distinctValues = statsResult.getDistinctValues();
Set<String> result = distinctValues.stream().map((i) -> i.toString()).collect(Collectors.toSet());
return result;
After trying the above code, all I get is the max, min, count, but no results for distinct totals or distinct values.
What am I doing wrong in this sample?
Looks like your disctintValues collection is also an implementation of EmptyList which means that there are no values in the response.
Check if your query returns any result, first.
StatsOptions statsOptions = new StatsOptions().addField(field).setSelectiveCalcDistinct(true).setCalcDistinct(true);
The processStatsOptions method in org.springframework.data.solr.core.DefaultQueryParser is described as follows :
Boolean selectiveCountDistincts = statsOptions.isSelectiveCalcDistincts(field);
if (selectiveCountDistincts != null) {
solrQuery.add(selectiveCalcDistinctParam, String.valueOf(selectiveCountDistincts.booleanValue()));
}
I am getting data for a particular user id from 14 tables as shown below. As part of data, I am extracting user_id, record_name and record_value and then I get timestamp from record_name (by splitting on it) and then populate my TreeMap with key as timestamp and value as record_value.
After that I am extracting 100 most recent record_value from valueTimestampMap and then populating it in my recordValueHolder LinkedList.
In my case 100 most recent means by looking at the timestamp not the way they are coming.
Below is my code -
public List<String> getData(String userId) {
List<String> recordValueHolder = new LinkedList<String>();
Map<Long, String> valueTimestampMap = new TreeMap<Long, String>(Collections.reverseOrder());
for (int tableNumber = 0; tableNumber < 14; tableNumber++) {
String sql = "select * from table_" + tableNumber + " where user_id='" + userId + "';";
SimpleStatement query = new SimpleStatement(sql);
query.setConsistencyLevel(ConsistencyLevel.QUORUM);
ResultSet res = session.execute(query);
Iterator<Row> rows = res.iterator();
while (rows.hasNext()) {
Row r = rows.next();
String user_id = r.getString("user_id"); // get user id
String record_name = r.getString("record_name"); // get record name
String record_value = r.getString("record_value"); // get record value
long timestamp = Long.parseLong(record_name.split("\\.")[1]);
// populate my tree map
valueTimestampMap.put(timestamp, record_value);
}
}
// now extract 100 most recent record_value since
// valueTimestampMap is already sorted basis on key in
// descending order
for (Map.Entry<Long, String> entry : valueTimestampMap.entrySet()) {
if (recordValueHolder.size() > 99)
break;
recordValueHolder.add(entry.getValue());
}
return recordValueHolder;
}
I am sorting TreeMap in descending order of the keys by using Collections.reverseOrder() so that I have most recent timestamps at the top and then I can simply extract 100 most recent record_value from it and that's what my above code does.
Problem Statement:-
I have 100 most recent record_value in recordValueHolder List. Now I also need to find out which tableNumber each record_value out of 100 came from and what was the record_name for that record_value as well?
So I was thinking to make a data structure something like below which can hold 100 most recent record_value along with their tableNumber, record_name and timestamp as well.
public class RecordValueTimestampTableHolder {
private long timestamp;
private String recordName;
private String recordValue;
private Integer tableNumber;
// setters and getters
}
So the size of List<RecordValueTimestampTableHolder> should be 100. Is this possible to do with my current setup? I am not able to understand how to make this work?
Now my return data type of getData method will change and instead of returning List<String>, now it will return List<RecordValueTimestampTableHolder> which will have 100 most recent record_values along with other values as well.
Instead of using a TreeMap<Long, String>, use TreeMap<Long, RecordValueTimestampTableHolder>
Instead of using
valueTimestampMap.put(timestamp, record_value);
use:
valueTimestampMap.put(timestamp, new RecordValueTimestampTableHolder(timestamp, record_name, record_value, tableNumber));
Of course, this means you will have to add a constructor to RecordValueTimestampTableHolder that accepts the four parameters and assigns them to the internal fields.
As you said recordValueHolder will have to be defined as a List<RecordValueTimestampTableHolder> and this will also have to be the return type from this method.
Filling it will be exactly like you fill it now. Though personally I'd use valueTimestampMap.values() to iterate.
int i = 0;
for (RecordValueTimestampTableHolder item : valueTimestampMap.values()) {
recordValueHolder.add(item);
if (++i == 100)
break;
}
Consider a collection of objects having fields like:
{
id: // String
type: //Integer
score: //Double value
}
I would like to query on collection using type and for returned documents divide their scores by their maximum. Consider following query oject:
DBObject searchQuery = new BasicDBObject("type", 2);
collection.find(searchQuery);
With above query it'll return some documents. I want to get maximum of scores among all those documents and then divide all those documents' score by obtained maximum.
How can I do this??
I could find maximum using aggregation as follows:
String propertyToOperateOn = "score";
DBObject match = new BasicDBObject("$match", searchQuery);
DBObject groups = new BasicDBObject("_id", null);
DBObject operation = new BasicDBObject("$max", "$" + propertyToOperateOn);
groups.put("maximum", operation);
DBObject apply = new BasicDBObject("$group", groups);
AggregationOutput output = mongoConstants.IAScores.aggregate(match, apply);
Here output will contain the maximum value. But then how can I update (divide) all documents' scores by this maximum??
I hope there could be better way to do this task, but I'm unable to get it as I'm very much new to mongodb (or any database as such).
This is technically the same issue as "mongodb: java: How to update a field in MongoDB using expression with existing value", but I'll repeat the answer:
At the moment, MongoDB doesn't allow you to update the value of a field according to an existing value of a field. Which means, you can't do the following SQL:
UPDATE foo SET field1 = field1 / 2;
In MongoDB, you will need to do this in your application, but be aware that this is no longer an atomic operation as you need to read and then write.