We Keep Coding

Slow queries for MongoDB regular expression text in Java

When searching for regular expression text in MongoDB, the speed is slow at first, so I would like to know the cause.
Only on the phenomenon JAVA Application Server will the corresponding slow query be found.
When the corresponding query is run in the MongoDB shell, it works very fast (index works well).
The number of data result values in the above query is five.
The total number of data in the collection is 450,000
Below is a process-specific query.
=====JAVA Process=====
(Very Slow 5,518ms)
public List<Contents> findContentList(int rowCnt, long rowNo, String searchContent){
Query query = new Query();
query.addCriteria((Criteria.where(DictionaryKey.content).regex("^" + searchContent)));
if (rowNo > 0) query.addCriteria(Criteria.where(DictionaryKey.contentSeq).gt(rowNo));
query.with(new Sort(Sort.Direction.ASC, DictionaryKey.contentSeq));
query.limit(rowCnt);
return this.mongoTemplate.find(query, Contents.class, Constant.CollectionName.Contents);
}
java Monitoring tool
Query : Query: { "content" : { "$regex" : "^abcd"}}, Sort: { "contentSeq" : 1}
Collection Name : contents
MongTemplate#find() [5,518ms] -- org.springframework.data.mongodb.core.mongTemplate.find()Ljava/util/List;
=====Mongodb Shell======
Mongodb query (Very Fast, index works well)
db.contents.found ({content:{"$regex" : "^abcd"}}).sort ({"contentSeq" : 1});
'contents' collection index is content_1_contentSeq_1
Please help me.

I found the reason
The cause is that some queries are using the wrong index.
The solution was to force the use of the index by giving a hint.

MongoDB + Morphia - full text search using AND instead of OR

I've setup full text search and MongoDB and it's working quite well (Mongo 2.6.5).
However it does an OR instead of and AND.
1) Is it possible to make the query an AND query, while still getting all the benefits of full text search (stemming etc.)
2) And if so, is it possible to add this option via the Morphia wrapper library
EDIT
I see that the full text search includes a 'score' for each document returned. Is it possible to only return docs with a certain score or above. Is there some score that would represent a 'fuzzy' and query. That is usually all tokens are in the document but not absolutely always. If so this would solve the problem as well.
Naturally if possible to do this via Morphia that would be super helpful. But I can use the native java driver as well.
Any pointers in the correct direction, much appreciated.
EDIT
Code looks like this, I'm using Morphia 1.0.1:
Datastore ds = Dao.instance().getDatabase();
Query<Product> q = ds.createQuery(Product.class).search("grey vests");
List<Product> prods = q.asList();
Printing the query gives:
{ "$text" : { "$search" : "grey vests"}}
Note: I am able to do take an intersection of multiple result sets to create an AND query. However this is very slow since something like "grey" will return a massive result set and be slow at feeding the results back.
EDIT
I've tried to chain the search() calls and add a single 'token' to each call. But I am getting a run time error. Code becomes:
q.search("grey").search("vests");
The query I get is (which seems like it's doing the right thing) ...
{ "$and" : [ { "$text" : { "$search" : "grey"}} , { "$text" : { "$search" : "vests"}}]}
The error is:
com.mongodb.MongoQueryException: Query failed with error code 17287 and error message 'Can't canonicalize query: BadValue Too many text expressions' on server ...
at com.mongodb.connection.ProtocolHelper.getQueryFailureException(ProtocolHelper.java:93)

Hibernate returns list of nulls although executed SQL returns values

I'm using hibernate as an ORMapper. I want to execute an actually rather simple hql query:
SELECT a
FROM Foo a
WHERE a.status = :A0status
ORDER BY a.bookingTypeCode ASC,
a.priority ASC
This hql query is then converted into a sql query which looks something like this:
select a.*
from Foo a
where a.status='A'
order by a.bookingtypecode ASC,
a.priority ASC
When I execute the sql on the oracle database using the Oracle SQL Developer I get 17 rows returned. However, when I execute the hql query (using the list method of a Query I get a list of 17 elements that are all null. Although the number of elements is correct, not a single one of the elements is actually loaded.
This is the way I create and execute my query:
// the hql query is stored in the hqlQuery variable;
// the parameter are stored in a Map<String, Object> called params
Query hQuery = hibSession.createQuery(hqlQuery);
for (Entry<String, Object> param : params.entrySet()) {
String key = param.getKey();
Object value = param.getValue();
hQuery.setParameter(key, value);
}
List<?> result = hQuery.list();
Does anyone know what might be the problem here?
Update 1
I've recently upgrade from hibernate 3.2 to 4.3.5. Before the upgrade everything worked fine. After the upgrade I get this error.

I've set the Log level of hibernate to TRACE and found the problem. It was actually a mapping/logic/database error. The primary key consisted of two columns (according to the entity class) and one of these columns was nullable. However a primary key can never be nullable. Therefore hibernate always returned null.

If you have not set a custom (and buggy) ResultTransformer, my second best guess is that your debugger is lying to you. Does you code actually receives a list of null?
Also make sure to test with the code you are showing is. Too many times, people simplify things and the devil is in the details.

This error is happening to me. MySQL query browser works, but in hibernate of 7 columns and only one column always came with all null fields. I checked all the ids and they were not null. The error was in the construction of SQL Native. I had to change the way of writing it. Ai worked.
SELECT c.idContratoEmprestimo as idContratoEmprestimo,
c.dtOperacao as dataOperacao,
p.cpf as cpf,
p.nome as nome,
(Select count(p2.idParcelaEmprestimo) from EMP_PARCELA p2 where p2.valorPago > 0 and p2.dtPagamento is not null
and p2.idContratoEmprestimo = c.idContratoEmprestimo and p2.mesCompetencia <= '2014-08-01') as parcelasPagas, c.numeroParcelas as numeroParcelas,
pe.valorPago as valorParcela
FROM EMP_CONTRATO c inner join TB_PARTICIPANTE_DADOS_PLANO AS pp on pp.idParticipantePlano = c.idParticipantePlano
inner join TB_PARTICIPANTE as p on p.id = pp.idParticipante
inner join TB_PARTICIPANTE_INSTITUIDOR as pi on pi.PARTICIPANTE_ID = p.id
inner join EMP_PARCELA as pe on pe.idContratoEmprestimo = c.idContratoEmprestimo
where c.dtInicioContrato <= '2014-08-01' and pi.INSTITUIDOR_ID = 1
and c.avaliado is true
and pe.mesCompetencia = '2014-08-01'
and c.deferido is true
and c.dtQuitacao is null
and c.dtExclusao is null
and pe.valorPago is not null
group by c.idContratoEmprestimo
order by p.nome

How to print the retrieved statements in SPARQL after executing a query

I am quite new to Java Sesame and SPARQL. I have added statements into the Sesame repository and am now trying to retrieve some statements and simply print them. From what I understood that I need to parse the query in order to print the retrieved statements. The code below shows where I have reached so far:
String queryString = "SELECT ?subject ?object WHERE { ?subject <http://example.org/is> ?object . } LIMIT 1";
...
SPARQLParser parser = new SPARQLParser();
ParsedQuery query = parser.parseQuery(queryString, null);
StatementPatternCollector collector = new StatementPatternCollector();
query.getTupleExpr().visit(collector);
List<StatementPattern> patterns = collector.getStatementPatterns();
// To print the first statement only for example.
System.out.println(patterns.get(0));
Here is the output:
StatementPattern
Var (name=name)
Var (name=-const-1, value=http://example.org/is, anonymous)
Var (name=object)
According to the output, it does not show me the subject and object. My questions are: How can I print the results of the query as shown above. Is this code the right way to parse the query for printing the statements later. Your help would be very much appreciated.

You get the results of a query by evaluating the query, not by parsing it. What your code does is inspect the query's algebra model and retrieve patterns from that algebra. This approach will not give you query results.
Evaluating a query in Sesame is far simpler than this:
// open a connection to the Sesame repository containing your statements
RepositoryConnection conn = repository.getConnection();
try {
// create a prepared query object from your query string
TupleQuery query = conn.prepareTupleQuery(QueryLanguage.SPARQL, queryString);
// evaluate the query on the repository
TupleQueryResult result = query.evaluate();
// iterate over the results and do something with each result
while (result.hasNext()) {
BindingSet s = result.next();
Value subject = s.getValue("subject");
Value object = s.getValue("object");
System.out.println("value of ?subject: " + subject);
System.out.println("value of ?object: " + object);
}
} finally {
conn.close();
}
As Joshua pointed out, your query retrieves variable bindings (for the variables mentioned in your SELECT clause), not RDF statements. You can of course re-create the RDF statement from those variable bindings in Java, but if you really want full statements rather than just the subject and object, it's probably easier to switch to using a SPARQL CONSTRUCT query instead of a SELECT query.
For more information on how to evaluate different kinds of queries and work with the results in Sesame, see the Sesame user documentation (section 6.5 in particular) and the API Javadoc.

Unexpected MongoDB "OR" query behaviour

I'm testing out spring-data and it's mongodb support.
I have a question about the query creation when using or-queries. Consider the following:
Query query = new Query().or(new Query(where("receiverId").is(userId)), new Query(where("requesterId").is(userId)));
query.and(where("status").is(status));
This will result in the following mongodb query:
"$or" : [ { "receiverId" : { "$oid" : "4d78696025d0d46b42d9c579"}} , { "requesterId" : { "$oid" : "4d78696025d0d46b42d9c579"}}] , "status" : "REQUESTED"}
This returns zero results while one is expected. Running this query in mongodb command results in following error:
error: { "$err" : "invalid operator: $oid", "code" : 10068 }
Modifying the query and running it in mongodb command works fine:
{ "$or" : [ { "receiverId" : ObjectId("4d78696025d0d46b42d9c579")} , { "requesterId" : ObjectId("4d78696025d0d46b42d9c579")}] , "status" : "REQUESTED"}
Notice the use of ObjectId("...") instead of $oid.
Am I going about something the wrong way? Maybe setting up the query wrong?

Are you inspecting that query variable at runtime or is that what you are seeing in MongoDB's logs?
Int he C# driver, if you inspect the query variable, you see $oid as well, but that is not the actual query that is sent to the server. At some point, it changes that to a valid MongoDB query.
If you are running on linux, you may want to start up mongosniff which will show you realtime queries, updates and inserts as they happen. If you are on Windows, you should start up mongod.exe with -vvvv flag which will enable it to log every query, update, insert, or command to the log file.
Then you can actually see the exact query that is being submitted.