Short question:
I'm looking for a way (java) to intercept a query to Solr and inject a few extra filtering parameters provided by my business logic. What structures should I use?
Context:
First of all, a little confession: I'm such a rookie regarding Solr. For me, setting up a server, defining a schema, coding a functional indexmanager and afterwards actually seeing the server returning the right results - exactly as intended! - was already much of an achievement for itself. Yay me!
However I'm currently working in an enterprise project that requires a little more than that. Roughly speaking, the solr instance is to be queried by several thousands of users through the very same requestHandler, being that the documents returned are automatically filtered according to a user's permission level. For example, if both the user A and the super-user B tried the very same search parameters (even the very same url), the user B would get all of user A's files and then some more. In order to accomplish this the documents are already indexed with the necessary permission level information.
Well, with this in mind and making use of Solr's extensive documentation for newb developers I tried to come up with a simple custom requestHandler that overrides the handleRequest function in order to inject the necessary extra parameters in the SolrQueryRequest. All is fine and dandy - except that I never see any difference at all in the QueryResponse, the server rudely ignoring my little manipulation. After a couple of days searching the web without so much of a hint weather if this the best approach, finally decided to come up and bother the fine folks here at StackOverflow.
So, in short, my questions are:
Is this a correct approach? Are there other alternatives? I can already grasp some of Solr's concepts, but admittedly there is much lacking and its entirely possible that am missing something.
If so, after modifying the query parameters is there anything I should do to force the QueryResponse to be updated? As far as I can tell these are merely encapsulating http requests, and I fail to sniff anything querying the server after the modifications are made.
Thanks in advance and so very sorry for the long post!
UPDATE
After a lot of reading APIs and specially much trial and error I've managed to get a functional solution. However I still fail to understand much of Solr's internals, therefore would still appreciate some enlightening. Feel free to bash at will, am still very aware of my rookiness.
The relevant part of the solution is this function which is called from by overriden handleRequestBody:
private void SearchDocumentsTypeII(SolrDocumentList results,
SolrIndexSearcher searcher, String q,
UserPermissions up, int ndocs, SolrQueryRequest req,
Map<String, SchemaField> fields, Set<Integer> alreadyFound)
throws IOException, ParseException {
BooleanQuery bq = new BooleanQuery();
String permLvl = "PermissionLevel:" + up.getPermissionLevel();
QParser parser = QParser.getParser(permLvl, null, req);
bq.add(parser.getQuery(), Occur.MUST);
Filter filter = CachingWrapperFilter(new QueryWrapperFilter(bq));
QueryParser qp = new QueryParser(q, new StandardAnalyzer());
Query query = qp.parse(q);
append (results, searcher.search(
query, filter, 50).scoreDocs,
alreadyFound, fields, new HashMap<String,Object>(), 0,
searcher.getReader(), true);
}
Basically the search query is not modified in any way, and instead a filter is applied containing the PermissionLevel of the user. Even so, why doesn't the following alternative work? The search query works perfectly when applied in the standard requestHandler, while in this case it simply doesn't hit any document.
private void SearchDocumentsTypeII(SolrDocumentList results,
SolrIndexSearcher searcher, String q,
UserPermissions up, int ndocs, SolrQueryRequest req,
Map<String, SchemaField> fields, Set<Integer> alreadyFound)
throws IOException, ParseException {
String qFiltered = q + " AND " + "PermissionLevel:" + up.getPermissionLevel();
QueryParser qp = new QueryParser(qFiltered, new StandardAnalyzer());
Query query = qp.parse(qFiltered);
append (results, searcher.search(
query, null, 50).scoreDocs,
alreadyFound, fields, new HashMap<String,Object>(), 0,
searcher.getReader(), true);
}
Good news: you don't need to write any code to do that, you just have to configure Solr properly. The superuser would hit the standard request handler while the regular user would hit another request handler (also a solr.StandardRequestHandler) configured with an invariant with the filter query you want to force upon them.
See also http://wiki.apache.org/solr/SolrRequestHandler
Oh well. As previously stated, the answer that worked for me. Feel free to comment or bash!
private void SearchDocumentsTypeII(SolrDocumentList results,
SolrIndexSearcher searcher, String q,
UserPermissions up, int ndocs, SolrQueryRequest req,
Map<String, SchemaField> fields, Set<Integer> alreadyFound)
throws IOException, ParseException {
BooleanQuery bq = new BooleanQuery();
String permLvl = "PermissionLevel:" + up.getPermissionLevel();
QParser parser = QParser.getParser(permLvl, null, req);
bq.add(parser.getQuery(), Occur.MUST);
Filter filter = CachingWrapperFilter(new QueryWrapperFilter(bq));
QueryParser qp = new QueryParser(q, new StandardAnalyzer());
Query query = qp.parse(q);
append (results, searcher.search(
query, filter, 50).scoreDocs,
alreadyFound, fields, new HashMap<String,Object>(), 0,
searcher.getReader(), true);
}
Take a look this solr wiki page it says we should first consider using Apache Manifold Framework, and if it doesn't suite your need, then write your own requestHandler
I had exact same requirement. In case anyone seeing this, here is my solution.
Request handler
public class SearchRequestHanlder extends SearchHandler {
#Override
public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throws Exception {
var map = req.getParams();
if (map instanceof MultiMapSolrParams) {
MultiMapSolrParams m = (MultiMapSolrParams) map;
MultiMapSolrParams.addParam("bq", "category:film^220", m.getMap());
}
super.handleRequestBody(req, rsp);
}
#Override
public String getDescription() {
return "Custom SearchRequestHanlder";
}
}
solrconf.xml
<lib dir="/opt/solr/data/cores/movies/lib" regex=".*\.jar" />
<!-- Make sure following line is after existing default <requestHandler name="/select" -->
<requestHandler name="/select" class="com.solrplugin.SearchRequestHanlder" />
Related
A while ago, I asked how to get the body from a Result in Play 2.5.0 Java.
The answer was basically to use play.core.j.JavaResultExtractor. I am now upgrading to 2.6, and JavaResultExtractor no longer exists (or at least is not public).
How does one do this in Play 2.6?
I did find Result.body().consumeData which seems like it might work, but also comes with the worrisome warning:
This method should be used carefully, since if the source represents an ephemeral stream, then the entity may not be usable after this method is invoked.
I suppose that, since I am doing this in an action, I could call consumeData to get all of the data into a local variable, process that and then return a new result with the stored data. That only fails in the case where the data is too big to fit into memory, which is not something I am currently expecting.
In in Play 2.6 it is still possible to re-implement 2.5 functionality. Please have a look at the example that get Json body from Result:
public static JsonNode resultToJsonNode(final Result result, final long timeout, final Materializer mat)
throws Exception {
FiniteDuration finiteDuration = Duration.create(timeout, TimeUnit.MILLISECONDS);
byte[] body = Await.result(
FutureConverters.toScala(result.body().consumeData(mat)), finiteDuration).toArray();
ObjectMapper om = new ObjectMapper();
final ObjectReader reader = om.reader();
return reader.readTree(new ByteArrayInputStream(body));
}
I'm using logback and I need to log all data queried by clients, to a log file. All the data queried by clients is needed to be logged to the same file. The logging process simply looks like below:
private static final OUTPUTFILELOGGER = Logger.getLogger(...);
String outputString = null;
try {
Map<String, Object> outputMap = doService(); // queries data requested by clients.
.... // do something after business logic..
outputLog = outputMap.toString(); // critical!!
} catch (Throwable e) {
handling exception
} finally {
OUTPUTFILELOGGER.info(outputString);
}
It usually works fine, but sometimes it arises OutOfMemoryError with the call of toString to the outputMap variable when the requested data is too big to make a string.
So I want to make it work in a way of streaming without any problem to performance. And I don't know how to make it effectively and gracefully.
Any idea?
Loop through the map so that you're only working with a small part at a time:
LOGGER.info("Map contains:")
map.forEach( (key, value) -> LOGGER.info("{}: {}", key, value));
(Assumes Java 8 and SLF4J)
However if the map is big enough for the code you've given to generate OOMs, you should probably consider whether it's appropriate to log it in such detail -- or whether your service ought to be capping the response size.
I am trying to create a JIRA plugin that does the following:
For each issue, takes all linked issues which are linked by "duplicates" or "is duplicated by" (or other predefined link types).
For each such issue, get a list (not necessarily a List object) of the voters on that issue.
My problem is that the javadoc has little to no information. Following a tutorial, I currently have:
public class VotersCount extends AbstractJiraContextProvider {
#Override
public Map<String, Integer> getContextMap(User user, JiraHelper jiraHelper) {
Map<String, Integer> contextMap = new HashMap<>();
Issue currentIssue = (Issue) jiraHelper.getContextParams().get("issue");
// Issue[] linkedIssues = currentIssue.getLinkedIssuesBy(...); //Step 1 mock code
// Voter[] voters = linkedissues[3].getVoters(); //Step 2 mock code
int count = voters.length; //Pretend there is some calculation here
contextMap.put("votersCount", count);
return contextMap;
}
}
(and I use votersCount in the .vm file.)
However, I see no explanation in the javadocs for AbstractJiraContextProvider and getContextMap so I'm not even sure if it's the right approach.
In my own research I found the class ViewVoters which has the method Collection<UserBean> getVoters(), which is something I can work with, but I don't know how to obtain or construct such an object in a way which will interact with a given issue.
I am looking for a working code to replace my 2 lines of mock code.
1) Use one of the methods from IssueLinkService. Maybe getIssueLinks
2) issueVoterAccessor.getVoterUserkeys
Instances of IssueLinkService and IssueVoterAccessor should be injected as parameters to constructor of your VotersCount.
I solved it by using the following:
To get issues linked to Issue issue by specified link types:
LinkCollection linkCollection = ComponentAccessor.getIssueLinkManager().getLinkCollectionOverrideSecurity(issue);
Set<IssueLinkType> linkTypes = linkCollection.getLinkTypes();
// Perform operations on the set to get the issues you want.
for (IssueLinkType linkType : linkTypes) {
List<Issue> l1 = linkCollection.getOutwardIssues(linkType.getName());
List<Issue> l2 = linkCollection.getInwardIssues(linkType.getName());
}
To get all the voters on Issue issue:
ComponentAccessor.getVoteManager().getVoterUserkeys(issue);
I was later shown that one can extend CalculatedCFType and override getValueFromIssue which hands you the current issue as a parameter instead of the using
Issue currentIssue = (Issue) jiraHelper.getContextParams().get("issue");
I am implementing REST through RESTlet. This is an amazing framework to build such a restful web service; it is easy to learn, its syntax is compact. However, usually, I found that when somebody/someprogram want to access some resource, it takes time to print/output the XML, I use JaxbRepresentation. Let's see my code:
#Override
#Get
public Representation toXml() throws IOException {
if (this.requireAuthentication) {
if (!this.app.authenticate(getRequest(), getResponse()))
{
return new EmptyRepresentation();
}
}
//check if the representation already tried to be requested before
//and therefore the data has been in cache
Object dataInCache = this.app.getCachedData().get(getURI);
if (dataInCache != null) {
System.out.println("Representing from Cache");
//this is warning. unless we can check that dataInCache is of type T, we can
//get rid of this warning
this.dataToBeRepresented = (T)dataInCache;
} else {
System.out.println("NOT IN CACHE");
this.dataToBeRepresented = whenDataIsNotInCache();
//automatically add data to cache
this.app.getCachedData().put(getURI, this.dataToBeRepresented, cached_duration);
}
//now represent it (if not previously execute the EmptyRepresentation)
JaxbRepresentation<T> jaxb = new JaxbRepresentation<T>(dataToBeRepresented);
jaxb.setFormattedOutput(true);
return jaxb;
}
AS you can see, and you might asked me; yes I am implementing Cache through Kitty-Cache. So, if some XML that is expensive to produce, and really looks like will never change for 7 decades, then I will use cache... I also use it for likely static data. Maximum time limit for a cache is an hour to remain in memory.
Even when I cache the output, sometimes, output are irresponsive, like hang, printed partially, and takes time before it prints the remaining document. The XML document is accessible through browser and also program, it used GET.
What are actually the problem? I humbly would like to know also the answer from RESTlet developer, if possible. Thanks
Its been a while since i've worked with Java especially exceptions. I'm in the process of adding ektorp couchdb intergration into something i'm working on. However i'm encountering content consumed exceptions.
The program in question uses twitter4j and i'm getting my statuses and writing them to a couchdb instance.
public void putTweet(Status status)
{
Map<String, Object> newTweetDoc = new HashMap<String, Object>();
String docname = status.getUser().getName() + " "
+ status.getCreatedAt().toString();
newTweetDoc.put("_id", docname);
newTweetDoc.put("User", status.getUser().getName());
newTweetDoc.put("Contents", status.getText());
newTweetDoc.put("Created", status.getCreatedAt().toString());
newTweetDoc.put("RetweetCount", status.getRetweetCount());
UserMentionEntity[] mentions = status.getUserMentionEntities();
Map<String, HashMap<String, String>> formattedMentions = formatMentions(mentions);
newTweetDoc.put("Mentions", formattedMentions);
db.addToBulkBuffer(newTweetDoc);
}
At first i tried db.create(newTweetDoc) as well. Does the couchdbConnector need to be recreated every time i try this?
db is a global CouchDbConnector:
public CouchDbConnector db = null;
public CouchTwitter()
{
//create the db connection etc
}
It's the db.create(doc) or flushBulkBuffer that results in the error. Here is the stacktrace:
Exception in thread "main" java.lang.IllegalStateException: Content has been consumed
at org.apache.http.entity.BasicHttpEntity.getContent(BasicHttpEntity.java:84)
at org.apache.http.conn.BasicManagedEntity.getContent(BasicManagedEntity.java:88)
at org.ektorp.http.StdHttpResponse.releaseConnection(StdHttpResponse.java:82)
at org.ektorp.http.RestTemplate.handleResponse(RestTemplate.java:111)
at org.ektorp.http.RestTemplate.post(RestTemplate.java:66)
at org.ektorp.impl.StdCouchDbConnector.executeBulk(StdCouchDbConnector.java:638)
at org.ektorp.impl.StdCouchDbConnector.executeBulk(StdCouchDbConnector.java:596)
at org.ektorp.impl.StdCouchDbConnector.flushBulkBuffer(StdCouchDbConnector.java:617)
I see in the above that two seperate Entity classes both call .getContent(), i've been playing around with my referenced libraries recently is it possible that its calling an old apache Http lib as well as the current?
CouchDbConnector is thread safe so you don't need recreate it for each operation.
I have never encountered your problem, your use case is pretty simple and there should not be any problem saving a basic doc.
Verify that httpclient-4.1.1 or above is in the classpath.