We Keep Coding

Slow queries for MongoDB regular expression text in Java

When searching for regular expression text in MongoDB, the speed is slow at first, so I would like to know the cause.
Only on the phenomenon JAVA Application Server will the corresponding slow query be found.
When the corresponding query is run in the MongoDB shell, it works very fast (index works well).
The number of data result values in the above query is five.
The total number of data in the collection is 450,000
Below is a process-specific query.
=====JAVA Process=====
(Very Slow 5,518ms)
public List<Contents> findContentList(int rowCnt, long rowNo, String searchContent){
Query query = new Query();
query.addCriteria((Criteria.where(DictionaryKey.content).regex("^" + searchContent)));
if (rowNo > 0) query.addCriteria(Criteria.where(DictionaryKey.contentSeq).gt(rowNo));
query.with(new Sort(Sort.Direction.ASC, DictionaryKey.contentSeq));
query.limit(rowCnt);
return this.mongoTemplate.find(query, Contents.class, Constant.CollectionName.Contents);
}
java Monitoring tool
Query : Query: { "content" : { "$regex" : "^abcd"}}, Sort: { "contentSeq" : 1}
Collection Name : contents
MongTemplate#find() [5,518ms] -- org.springframework.data.mongodb.core.mongTemplate.find()Ljava/util/List;
=====Mongodb Shell======
Mongodb query (Very Fast, index works well)
db.contents.found ({content:{"$regex" : "^abcd"}}).sort ({"contentSeq" : 1});
'contents' collection index is content_1_contentSeq_1
Please help me.

I found the reason
The cause is that some queries are using the wrong index.
The solution was to force the use of the index by giving a hint.

MongoDB + Morphia - full text search using AND instead of OR

I've setup full text search and MongoDB and it's working quite well (Mongo 2.6.5).
However it does an OR instead of and AND.
1) Is it possible to make the query an AND query, while still getting all the benefits of full text search (stemming etc.)
2) And if so, is it possible to add this option via the Morphia wrapper library
EDIT
I see that the full text search includes a 'score' for each document returned. Is it possible to only return docs with a certain score or above. Is there some score that would represent a 'fuzzy' and query. That is usually all tokens are in the document but not absolutely always. If so this would solve the problem as well.
Naturally if possible to do this via Morphia that would be super helpful. But I can use the native java driver as well.
Any pointers in the correct direction, much appreciated.
EDIT
Code looks like this, I'm using Morphia 1.0.1:
Datastore ds = Dao.instance().getDatabase();
Query<Product> q = ds.createQuery(Product.class).search("grey vests");
List<Product> prods = q.asList();
Printing the query gives:
{ "$text" : { "$search" : "grey vests"}}
Note: I am able to do take an intersection of multiple result sets to create an AND query. However this is very slow since something like "grey" will return a massive result set and be slow at feeding the results back.
EDIT
I've tried to chain the search() calls and add a single 'token' to each call. But I am getting a run time error. Code becomes:
q.search("grey").search("vests");
The query I get is (which seems like it's doing the right thing) ...
{ "$and" : [ { "$text" : { "$search" : "grey"}} , { "$text" : { "$search" : "vests"}}]}
The error is:
com.mongodb.MongoQueryException: Query failed with error code 17287 and error message 'Can't canonicalize query: BadValue Too many text expressions' on server ...
at com.mongodb.connection.ProtocolHelper.getQueryFailureException(ProtocolHelper.java:93)

Why are upserts so slow for MongoDB Java API?

Using Mongo Java Driver 2.13 and Mongo 3.0.
I am trying to move from Spring Data save() to MongoDB API's Bulk Writing since I am saving/updating about 100K objects. I am trying to write the Service/Repository layer code where I can pass in a Collection of my specific Objects and be able to either create new records or update existing records, or in other words upsert. When I do an insert the performance is very acceptable.
If I update the code to do upserts the performance is just way too slow. Am I doing something wrong in the following code sample (note it is scaled down to just the necessary logic, i.e. no error handling):
public void save(Collection<MyDomainObject> objects) {
BulkWriteOperation bulkWriter = dbCollection.initializeUnorderedBulkOperation();
for(MyDomainObject mdo : objects) {
DBObject dbObject = convert(mdo);
bulkWriter.find(new BasicDBObject("id",mdo.getId()))
.upsert().updateOne(new BasicDBObject("$set",dbObject));
}
bulkWriter.execute(writeConcern);
}
Note that I also tried replaceOne() instead of updateOne() with the same results.
I also noticed in the Mongo log that "nscannedObjects" keeps increasing while "nMatched", "nModified" and "upsert" are never larger than 1. Does this mean that it is table scanning for each record?
Am I using upsert the correct way? Any other suggestions?

Thanks to ry_donahue I figured out the issue.
It was not using the correct ID field, which is the index. In the conversion of the domain object to a DBObject there ended up being an "id" and an "_id" field.
I also changed updateOne() to replaceOne(). So now the code looks like this:
public void save(Collection<MyDomainObject> objects) {
BulkWriteOperation bulkWriter = dbCollection.initializeUnorderedBulkOperation();
for(MyDomainObject mdo : objects) {
DBObject dbObject = convert(mdo);
bulkWriter.find(new BasicDBObject("_id",new ObjectId(mdo.getId()))).upsert().replaceOne(dbObject);
}
bulkWriter.execute(writeConcern);
}
This now gives very good performance.

How to display a database columns using CRUD in play framework?

I want to display the data of a postgresql database using the CRUD in the play framework; I looked for any examples to get idea which I didn't find after a long time of searching in google. Someone help me with this if you can or post a valid link regarding this. Thanks in advance!
I use, Play 1.2.5, java and postgresql.

I assume you want to do this in your application code in runtime.
You can execute query using DB plugin to search DB metadata using native PostgreSQL queries.
Here's an example how to get column names of my system DOCUMENT table:
List<String> columns = new ArrayList<>();
ResultSet result = DB.executeQuery("select column_name from INFORMATION_SCHEMA.COLUMNS where table_name = 'DOCUMENT';");
while(result.next()) {
String column = result.getString(1);
columns.add(column);
}
This note that code is somewhat simplified and you should use prepared statements if anything in this query will be inserted from data that user or any other system entered.
Use DB.getConnection().prepareStatement() to get PreparedStatement instance.

alias in mongo db

I've started to fiddle with mongo db and came up with a question.
Say, I have an object (POJO) with an id field (say, named 'ID') that I would like to represent in JSON and store/load in/from Mongo DB.
As far as I understood any object always has _id field (with underscore, lowercased).
What I would like to do is: during the query I would like the mongo db to return me my JSON with field ID instead of _id.
In SQL I would use something like
SELECT _id as ID ...
My question is whether its possible to do this in mongo db, and if it is, the Java based Example will be really appreciated :)
I understand that its possible to iterate over the records and substitute the _id with ID manually but I don't want this O(n) loop.
I also don't really want to duplicate the lines and store both "id" and "_id"
So I'm looking for solution at the level of query or maybe Java Driver.
Thanks in advance and have a nice day

Mongodb doesnt use SQL , its more like Object Query Language and Collections.
what you can try is , some thing similar to below code using Mongo Java Driver
Pojo obj = new PojoInstance();
obj.setId(id);
db.yourdb.find(obj);

I've end up using the following approach in the Java Driver:
DBCursor cursor = runSomeQuery();
try {
while(cursor.hasNext()) {
DBObject dbObject = cursor.next();
ObjectId id = (ObjectId) dbObject.get("_id");
dbObject.removeField("_id");
dbObject.put("ID", id.toString());
System.out.println(dbObject);
}
} finally {
cursor.close();
}
I was wondering whether this is the best solution or I have other better options
Mark

Here's an example of what I am doing in Javascript. It may be helpful to you. In my case I am removing the _id field and aliasing the two very nested fields to display simpler names.
db.players.aggregate([
{ $match: { accountId: '12345'}},
{ $project: {
"_id": 0,
"id": "$id",
"masterVersion": "$branches.master.configuration.player.template.version",
"previewVersion": "$branches.preview.configuration.player.template.version"
}
}
])
I hope you find this helpful.