I need to sort on a date-field type, which name is "mod_date".
It works like this in the browser adress-bar:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+desc
But I am using a phpSolr client which sends an URL to Solr, and the url sent is this:
fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc
// This wont work and is echoed after this in php:
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
This wont work, I dont know why!
Everything else works fine, all right fields are returned. But the sort doesn't work.
Any ideas?
Thanks
BTW: The field "mod_date" contains something like:
2010-03-04T19:37:22.5Z
EDIT:
First I use PHP to send this to a SolrPhpClient which is another php-file called service.php:
require_once('../SolrPhpClient/Apache/Solr/Service.php');
$solr = new Apache_Solr_Service('localhost', 8983, '/solr/');
$results = $solr->search($querystring, $p, $limit, $solr_params);
$solr_params is an array which contains the solr-parameters (q, fq, etc).
Now, in service.php:
$params['version'] = self::SOLR_VERSION;
// common parameters in this interface
$params['wt'] = self::SOLR_WRITER;
$params['json.nl'] = $this->_namedListTreatment;
$params['q'] = $query;
$params['sort'] = 'mod_date desc'; // HERE IS THE SORT I HAVE PROBLEM WITH
$params['start'] = $offset;
$params['rows'] = $limit;
$queryString = http_build_query($params, null, $this->_queryStringDelimiter);
$queryString = preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=', $queryString);
if ($method == self::METHOD_GET)
{
return $this->_sendRawGet($this->_searchUrl . $this->_queryDelimiter . $queryString);
}
else if ($method == self::METHOD_POST)
{
return $this->_sendRawPost($this->_searchUrl, $queryString, FALSE, 'application/x-www-form-urlencoded');
}
The $results contain the results from Solr...
So this is the way I need to get to work (via php).
This code below (also on top of this Q) works but thats because I paste it into the adress bar manually, not via the PHPclient. But thats just for debugging, I need to get it to work via the PHPclient:
http://localhost:8983/solr/select/?&q=bmw&sort=mod_date+des // Not via phpclient, but works
UPDATE (2010-03-08):
I have tried Donovans codes (the urls) and they work fine.
Now, I have noticed that it is one of the parameters causing the 'SORT' not to work.
This parameter is the "wt" parameter. If we take the url from top of this Q, (fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&version=1.2&wt=json&json.nl=map&q=%2A%3A%2A&start=0&rows=5&sort=mod_date+desc), and just simply remove the "wt" parameter, then the sort works.
BUT the results appear differently, thus making my php code not able to recognize the results I believe. Donovan would know this I think. I am guessing in order for the PHPClient to work, the results must be in a specific structure, which gets messed up as soon as I remove the wt parameter.
Donovan, help me please...
Here is some background what I use your SolrPhpClient for:
I have a classifieds website, which uses MySql. But for the searching I am using Solr to search some indexed fields. Then Solr returns an array of ID:numbers (for all matches of the search criteria). Then I use those ID:numbers to find everything in a MySql db and fetch all other information (example is not searchable information).
So simplified: Search -> Solr returns all matches in an array of ID:nrs -> Id:numbers from Solr are the same as the Id numbers in the MySql db, so I can just make a simple match agains every record with the ID matching the ID from the Solr results array.
I don't use Faceting, no boosting, no relevancy or other fancy stuff. I only sort by the latest classified put, and give the option to users to also sort on the cheapest price. Nothing more.
Then I use the "fq" parameter to do queries on different fields in Solr depending on category chosen by users (example "cars" in this case which in my language is "Bilar").
I am really stuck with this problem here...
Thanks for all help
As pointed out in the stack overflow comments, your browser query is different than your php client based query - to remove that from the equation you should test with this corrected. To get the same results as the browser based query you're php code should have looked something like this:
$solr = new Apache_Solr_Client(...);
$searchOptions = array(
'sort' => 'mod_date desc'
);
$results = $solr->search("bmw", 0, 10, $searchOptions);
Instead, I imagine it looks more like:
$searchOptions = array(
'fq' => 'category:"Bilar" + car_action:Sälje',
'sort' => 'mod_date desc'
)
$solr->search("\*:*", 0, 10, $searchOptions);
What I expect you to see is that php client results will be the same as the browser based results, and I imagine the same would happen if you did it the opposite way - take your current parameters from the php client and applied them correctly to the browser based query.
Now onto your problem, you don't see documents sorted properly.
I would try this query, which is the equivalent of the php client based code:
http://localhost:8983/solr/select/?&q=%2A%3A%2A&fq=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
versus this query, which moves the filter query into the main query:
http://localhost:8983/solr/select/?&q=+category%3A%22Bilar%22+%2B+car_action%3AS%C3%A4ljes&sort=mod_date+desc
and see if there is a difference. If there is, then it might be a bug in how results from cached filtered queries are used and sorted by solr - which wouldn't be a problem with the client, but the solr service itself.
Hope this gets you closer to an anser.
Use session's values for save sort parameters.
The quick answer in case someone is attempting to sort via solr-php-client:
$searchOptions = array('sort' => 'field_date desc');
Ditch the + sign that you would usually put on the URL. It took me a while as well to figure it out, I was encoding it and putting it all over the place...
It's possible it's related to the json.nl=map parameter. When the response is set to JSON with wt=json and json.nl=map, facets are not sorted as expected with the facet.sort or f.<field_name>.facet.sort=count|index options.
e.g. with facet.sort=count and wt=json only, I get:
"dc_coverage": [
"United States",
5,
"19th century",
1,
"20th century",
1,
"Detroit (Mich.)",
1,
"Pennsylvania",
1,
"United States--Michigan--Detroit",
1,
"United States--Washington, D.C.",
1
]
But with facet.sort=count, wt=json, and json.nl=map as an option, you can see the sorting is lost:
"dc_coverage": {
"19th century": 1,
"20th century": 1,
"Detroit (Mich.)": 1,
"Pennsylvania": 1,
"United States": 5,
"United States--Michigan--Detroit": 1,
"United States--Washington, D.C.": 1
}
There is more information here about formatting the JSON response when using json.nl=map: https://cwiki.apache.org/confluence/display/solr/Response+Writers#ResponseWriters-JSONResponseWriter
Related
I'm using Java GMail API and everything is working good for sending e-mails, collecting data from my profile, etc.
The only problem is that, while I can get the Signature for my 0-th element of the list of SendAs aliases, I can't get the Display Name: it returns an empty String. Both work for the other aliases (get(1) and subsequent numbers). It seems that the problem is on 0, tried on different authenticated users with Name set and it remains the same.
ListSendAsResponse aliases = service.users().settings().sendAs().list("me").execute();
SendAs mimmo = aliases.getSendAs().get(0);
actualsign = mimmo.getSignature();
sendername = mimmo.getDisplayName();
In Gmail API, there are two different ways to retrieve alias(es):
getSendAs() and SendAs.Get(java.lang.String userId,java.lang.String sendAsEmail)
The first one returns you a list of all alias, the second returns you one alias ressource - the one with the specified userId and sendAsEmail parameters.
If what you want to do it to retrieve the first element of the getSendAs() response, you should do it with getSendAs()[0] and not with the Java method get.
Sample:
SendAs mimmo = aliases.getSendAs()[0];
System.out.println(mimmo.getDisplayName());
It is always useful to test with the Try this API what response a method returns. Thereby, the userId can be set to me.
I have a SearchQuery I've constructed like this:
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.withQuery(queryString("A")) .withSort(SortBuilders.fieldSort("fruitId").order(SortOrder.DESC))
.withIndices("fruit").withTypes("fruit");
I want to be able to specify some sort of "start" and "size" on the search results so that I can:
1) Return the number of results based on the "size".
2) Return documents with based on the "Start" integer. For example, if my documents where A,B,C,D,E...Z If my "start" was set to 5 and "size" was set to 5, it would return "F,G,H,I,J".
Since you're using Spring Data Elasticsearch, you need to call the withPageable() method:
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withPageable(new PageRequest(1, 5)) <--- add this
.withQuery(matchAllQuery())
.withQuery(queryString("A"))
.withSort(SortBuilders.fieldSort("fruitId").order(SortOrder.DESC))
.withIndices("fruit").withTypes("fruit");
The PageRequest constructor takes two arguments: page (0-based) and size. Here we're asking for the second page (i.e. page 1) of size 5. This is going to return you exactly what you need.
Official documentation of the PageRequest class.
Note, however, that your query still has a problem, i.e. you're calling withQuery() twice, so the second call will override the first one. In this case, it might be ok since the first call is a matchAll (basically, return everything) and the second call is more specific. But if you want to combine two different queries, you'll need to proceed differently and use a boolQuery instead.
You need From/Size:
{
"from" : 5, "size" : 5,
"query": {
....
}
}
I'm trying to query something from the Rally database. Right now I'm just trying to make sure I can get through initially. This code:
//create rallyrest object
RallyRestApi restApi = new RallyRestApi(new URI(hostname), username, password);
restApi.setApplicationName("QueryTest");
restApi.setWsapiVersion("v2.0");
restApi.setApplicationVersion("1.1");
System.out.println("1: So far, so good -- RallyRestApi object created");
try {
//create query request
String type = "HierarchicalRequirement";
QueryRequest qreq = new QueryRequest(type);
System.out.println("2: Still going -- Query Request Created");
//set fetch, filter, and project
qreq.setFetch(new Fetch("Name","FormattedID"));
qreq.setQueryFilter(new QueryFilter("Name", "contains", "freight"));
qreq.setProject(projectNumber);
System.out.println("3: Going strong -- Fetch, Filter, and Project set");
//create response from query********Blows up
QueryResponse resp = restApi.query(qreq);
System.out.println("4: We made it!");
} catch (Exception e) {
System.out.println("Error: " + e);
}
finally {
restApi.close();
}
}
gives me this error:
Exception in thread "main" com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22
at com.google.gson.JsonParser.parse(JsonParser.java:65)
at com.google.gson.JsonParser.parse(JsonParser.java:45)
at com.rallydev.rest.response.Response.<init>(Response.java:25)
at com.rallydev.rest.response.QueryResponse.<init>(QueryResponse.java:18)
at com.rallydev.rest.RallyRestApi.query(RallyRestApi.java:227)
at RQuery.main(RQuery.java:65)
Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 22
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1505)
at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1386)
at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:531)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:414)
at com.google.gson.JsonParser.parse(JsonParser.java:60)
... 5 more
Java Result: 1
Could someone please explain why this is happening? Is my code wrong? If I need to do as the error suggests and set Json.lenient(true), please give me instructions on how to do that with my code.
Thank you!
Your code works for me. I don't see anything wrong with the code.
Try different query, maybe there are some extra characters.
See this post - it mentioned a case with trailing NUL (\0) characters. Perhaps you need to set lenient to true, but I don't know how to do it: there is no direct access to it when working with Rally QueryResponse.
The reason for trying a different query is that there are two local factors: your java environment and your data. MalformedJsonException indicates bad json which points to data. You are fetching only "Name" and "FormattedID", so chances are the culprit is somewhere in the name. Try a different query, e.g. (FormattedID = US123) but choose the story that does not contain "freight" in the name. Establish that at least one particular query works - it will indicate further that the issue is indeed related to data, and not the environment.
Next, try the same query (Name contains "freight") directly in WS API, which is an interactive document where queries can be tested. An equivalent of the query in your code can also be pasted in the browser:
https://rally1.rallydev.com/slm/webservice/v2.0/hierarchicalrequirement?workspace=https://rally1.rallydev.com/slm/webservice/v2.0/workspace/123&query=(Name%20contains%20%22fraight%22)&start=1&pagesize=200&fetch=Name,FormattedID
Make sure to replace 123 in /workspace/123 with the valid OID of your workspace.
Does the query return or you see the same error in the browser?
If the query returns, what is the TotalResultCount?
The total result count will help to troubleshoot further. You may run your code one page at a time, and knowing the TotalResultCount it is possible to manipulate pagesize, start and limit to narrow down your code to the page where the culprit story exists (assuming that there is a culprit story). Here is an example:
qreq.setPageSize(200);
qreq.setStart(2);
qreq.setLimit(200);
Maximum pagesize is 200. Default is 20. The actual number to use depends on TotalResultCount.
The start index for queries begins at 1. The default is 1. In this example we start with second page
My environment is Java SE 1.6 and these jars:
httpcore-4.2.4.jar
httpclient-4.2.5.jar
commons-logging-1.1.1.jar
commons-codec-1.6.jar
gson-2.2.4.jar
rally-rest-api-2.0.4.jar
Maybe I'm really missing something.
I have indexed a bunch of key/value pairs in Lucene (v4.1 if it matters). Say I have
key1=value1 and key2=value2, e.g. as read from a properties file.
They get indexed both as specific fields and into a catchall "ALL" field, e.g.
new Field("key1", "value1", aFieldTypeMimickingKeywords);
new Field("key2", "value2", aFieldTypeMimickingKeywords);
new Field("ALL", "key1=value1", aFieldTypeMimickingKeywords);
new Field("ALL", "key2=value2", aFieldTypeMimickingKeywords);
// then get added to the Document of course...
I can then do a wildcard search, using
new WildcardQuery(new Term("ALL", "*alue1"));
and it will find the hit.
But, it would be nice to get more info, like "what was complete value (e.g. "key1=value1") that goes with that hit?".
The best I can figure out it to get the Document, then get the list of IndexableFields, then loop over all of them and see if the field.stringValue().contains("alue1"). (I can look at the data structures in the debugger and all the info is there)
This seems completely insane cause isn't that what Lucene just did? Shouldn't the Hit information return some of the Fields?
Is Lucene missing what seems like "obvious" functionality? Google and starting at the APIs hasn't revealed anything straightforward, but I feel like I must be searching on the wrong stuff.
You might want to try with IndexSearcher.explain() method. Once you get the ID of the matching document, prepare a query for each field (using the same search keywords) and invoke Explanation.isMatch() for each query: the ones that yield true will give you the matched field. Example:
for (String field: fields){
Query query = new WildcardQuery(new Term(field, "*alue1"));
Explanation ex = searcher.explain(query, docID);
if (ex.isMatch()){
//Your query matched field
}
}
I've gone through the related questions on this site but haven't found a relevant solution.
When querying my Solr4 index using an HTTP request of the form
&facet=true&facet.field=country
The response contains all the different countries along with counts per country.
How can I get this information using SolrJ?
I have tried the following but it only returns total counts across all countries, not per country:
solrQuery.setFacet(true);
solrQuery.addFacetField("country");
The following does seem to work, but I do not want to have to explicitly set all the groupings beforehand:
solrQuery.addFacetQuery("country:usa");
solrQuery.addFacetQuery("country:canada");
Secondly, I'm not sure how to extract the facet data from the QueryResponse object.
So two questions:
1) Using SolrJ how can I facet on a field and return the groupings without explicitly specifying the groups?
2) Using SolrJ how can I extract the facet data from the QueryResponse object?
Thanks.
Update:
I also tried something similar to Sergey's response (below).
List<FacetField> ffList = resp.getFacetFields();
log.info("size of ffList:" + ffList.size());
for(FacetField ff : ffList){
String ffname = ff.getName();
int ffcount = ff.getValueCount();
log.info("ffname:" + ffname + "|ffcount:" + ffcount);
}
The above code shows ffList with size=1 and the loop goes through 1 iteration. In the output ffname="country" and ffcount is the total number of rows that match the original query.
There is no per-country breakdown here.
I should mention that on the same solrQuery object I am also calling addField and addFilterQuery. Not sure if this impacts faceting:
solrQuery.addField("user-name");
solrQuery.addField("user-bio");
solrQuery.addField("country");
solrQuery.addFilterQuery("user-bio:" + "(Apple OR Google OR Facebook)");
Update 2:
I think I got it, again based on what Sergey said below. I extracted the List object using FacetField.getValues().
List<FacetField> fflist = resp.getFacetFields();
for(FacetField ff : fflist){
String ffname = ff.getName();
int ffcount = ff.getValueCount();
List<Count> counts = ff.getValues();
for(Count c : counts){
String facetLabel = c.getName();
long facetCount = c.getCount();
}
}
In the above code the label variable matches each facet group and count is the corresponding count for that grouping.
Actually you need only to set facet field and facet will be activated (check SolrJ source code):
solrQuery.addFacetField("country");
Where did you look for facet information? It must be in QueryResponse.getFacetFields (getValues.getCount)
In the solr Response you should use QueryResponse.getFacetFields() to get List of FacetFields among which figure "country". so "country" is idenditfied by QueryResponse.getFacetFields().get(0)
you iterate then over it to get List of Count objects using
QueryResponse.getFacetFields().get(0).getValues().get(i)
and get value name of facet using QueryResponse.getFacetFields().get(0).getValues().get(i).getName()
and the corresponding weight using
QueryResponse.getFacetFields().get(0).getValues().get(i).getCount()