I've started to fiddle with mongo db and came up with a question.
Say, I have an object (POJO) with an id field (say, named 'ID') that I would like to represent in JSON and store/load in/from Mongo DB.
As far as I understood any object always has _id field (with underscore, lowercased).
What I would like to do is: during the query I would like the mongo db to return me my JSON with field ID instead of _id.
In SQL I would use something like
SELECT _id as ID ...
My question is whether its possible to do this in mongo db, and if it is, the Java based Example will be really appreciated :)
I understand that its possible to iterate over the records and substitute the _id with ID manually but I don't want this O(n) loop.
I also don't really want to duplicate the lines and store both "id" and "_id"
So I'm looking for solution at the level of query or maybe Java Driver.
Thanks in advance and have a nice day
Mongodb doesnt use SQL , its more like Object Query Language and Collections.
what you can try is , some thing similar to below code using Mongo Java Driver
Pojo obj = new PojoInstance();
obj.setId(id);
db.yourdb.find(obj);
I've end up using the following approach in the Java Driver:
DBCursor cursor = runSomeQuery();
try {
while(cursor.hasNext()) {
DBObject dbObject = cursor.next();
ObjectId id = (ObjectId) dbObject.get("_id");
dbObject.removeField("_id");
dbObject.put("ID", id.toString());
System.out.println(dbObject);
}
} finally {
cursor.close();
}
I was wondering whether this is the best solution or I have other better options
Mark
Here's an example of what I am doing in Javascript. It may be helpful to you. In my case I am removing the _id field and aliasing the two very nested fields to display simpler names.
db.players.aggregate([
{ $match: { accountId: '12345'}},
{ $project: {
"_id": 0,
"id": "$id",
"masterVersion": "$branches.master.configuration.player.template.version",
"previewVersion": "$branches.preview.configuration.player.template.version"
}
}
])
I hope you find this helpful.
Related
I am using Spring data mongo to insert a record to Mongo,
here is my code
mongoTemplate.save(person,"personCollection");
Here is my person object
public class Person implements Serializable {
int age;
String address;
String name;
//Getters and setters including hashcode and equals
}
my address is null here , after inserting the record in the collection, the data is populated with only age and name
i know that mongodb treats null value and noKey as the same thing, but my requirement is to even populate the address:null to have the schema consistent how do i do this with Spring Data mongo
current o/p: {"age":21,"name":"john Doe"}
expected o/p: {"age":21,"name":"john Doe","address":null}
NoSQL DB works in a different way compared to RDBMS.
the document {"age":21,"name":"john Doe"} is same as {"age":21,"name":"john Doe";"address":null}
instead of storing the key's as null better to not store the key at all this improves the performance of your reads/updates against the DB.
However, if your usecase still demands to sore null due to whatever the reasons it might be convert your POJO to BSONObject and then persist the BSONObject in the MongoDB.
Here is the example ( but however it will be only a work around to get the things going)
BSONObject personBsonObj = BasicDBObjectBuilder.start()
.add("name","John Doe")
.add("age",21)
.add("address",null).get();
if you are using spring data mongo use
mongoTemplate.insert(personBsonObj,"personCollection");
document in the db:
db.personCollection.findOne().pretty();
{"age":21,"name":"John Doe";"address":null}*
I've solved this problem using the below code
final Document document = new Document();
document.put("test1", "test1");
document.put("test2", null);
document.put("test3", "test3");
mongoTemplate.getCollection("your-collection-name").insert(document);
Here instead of using BSONObject, I used Document object and it worked fine.
Document inserted in DB
{
"_id" : ObjectId("some-id"),
"test1" : "test1",
"test2" : null,
"test3" : "test3"
}
I have been using open source data set provider Casper to achieve in-memory representation of a collection of Database objects in Java.
Github Repository : https://github.com/casperds/casperdatasets
Below is the code that I have been using to pull data in Casper datasets
String[] primaryKeys = { "QUESTION_ID" };
if (resultSet != null)
{
container = CDataCacheDBAdapter.loadData(resultSet, null, primaryKeys,new HashMap<Object, Object>());
lCDataRowset = container.getAll();
preparedStatement.close();
resultSet.close();
}
The problem with using this is, when I don't mention primary keys then DBAdapter does not load data. And If I mention some column as primary keys then "Order By" does not have effect in the dataset. It just orders by primary keys.
I want to be able to pull data in dataset in order the way I have mentioned in the query.
Did anybody face this issue? Any kind of help is appreciated!! Thanks
Well it turned out to be very stupid issue. If you pass null for primaryKeys parameter then it returns data in the order the way it returns in MySQL query browser.
I thought this could help someone someday. That's why keeping this post other wise I would have deleted it.
I cannot for the life of me find out how to get a count for a find query using the java driver in mongo db. Can someone please put me out of my misery?
I have the following:
MongoCursor<Document> findRes = collection.find().iterator();
But there is no count method that I can find anywhere.
public Long getTotalCount(String collectionName, Document filterDocument) {
MongoCollection collection = database.getCollection(collectionName);
return filterDocument != null ? collection.count(filterDocument) : collection.count();
}
Where filterDocument is org.bson.Document with filter criterias or null if you want to get total count
You may also use more powerful Filters class. Example: collection.count(Filters.and(Filters.eq("field","value"),second condition and so on));
So, in order to be able to take both Document and Filters as param you may change signature to public Long getTotalCount(String collectionName, Bson filterDocument) {
long rows = db.getCollection(myCollection).count(new Document("_id", 10)) ;
this is in Java, myCollection is collection name.
MongoDB has inbuilt method count() that can be called on cursor to find the number of documents returned.
I tried following piece of code in mongodb, that worked well, can be easily applied in java or any other language too:
var findres = db.c.find()
findres.count() gave output 29353
cursor.count() is what you're looking for I believe. Your find query returns a Cursor so you can just call count() on that.
Using Mongo Java Driver 2.13 and Mongo 3.0.
I am trying to move from Spring Data save() to MongoDB API's Bulk Writing since I am saving/updating about 100K objects. I am trying to write the Service/Repository layer code where I can pass in a Collection of my specific Objects and be able to either create new records or update existing records, or in other words upsert. When I do an insert the performance is very acceptable.
If I update the code to do upserts the performance is just way too slow. Am I doing something wrong in the following code sample (note it is scaled down to just the necessary logic, i.e. no error handling):
public void save(Collection<MyDomainObject> objects) {
BulkWriteOperation bulkWriter = dbCollection.initializeUnorderedBulkOperation();
for(MyDomainObject mdo : objects) {
DBObject dbObject = convert(mdo);
bulkWriter.find(new BasicDBObject("id",mdo.getId()))
.upsert().updateOne(new BasicDBObject("$set",dbObject));
}
bulkWriter.execute(writeConcern);
}
Note that I also tried replaceOne() instead of updateOne() with the same results.
I also noticed in the Mongo log that "nscannedObjects" keeps increasing while "nMatched", "nModified" and "upsert" are never larger than 1. Does this mean that it is table scanning for each record?
Am I using upsert the correct way? Any other suggestions?
Thanks to ry_donahue I figured out the issue.
It was not using the correct ID field, which is the index. In the conversion of the domain object to a DBObject there ended up being an "id" and an "_id" field.
I also changed updateOne() to replaceOne(). So now the code looks like this:
public void save(Collection<MyDomainObject> objects) {
BulkWriteOperation bulkWriter = dbCollection.initializeUnorderedBulkOperation();
for(MyDomainObject mdo : objects) {
DBObject dbObject = convert(mdo);
bulkWriter.find(new BasicDBObject("_id",new ObjectId(mdo.getId()))).upsert().replaceOne(dbObject);
}
bulkWriter.execute(writeConcern);
}
This now gives very good performance.
I am newbie to mongodb. May I know how to avoid duplicate entries. In relational tables, we use primary key to avoid it. May I know how to specify it in Mongodb using java?
Use an index with the {unique:true} option.
// everyone's username must be unique:
db.users.createIndex({email:1},{unique:true});
You can also do this across multiple fields. See this section in the docs for more details and examples.
A unique index ensures that the indexed fields do not store duplicate values; i.e. enforces uniqueness for the indexed fields. By default, MongoDB creates a unique index on the _id field during the creation of a collection.
If you wish for null values to be ignored from the unique key, then you have to also make the index sparse (see here), by also adding the sparse option:
// everyone's username must be unique,
//but there can be multiple users with no email field or a null email:
db.users.createIndex({email:1},{unique:true, sparse:true});
If you want to create the index using the MongoDB Java Driver. Try:
Document keys = new Document("email", 1);
collection.createIndex(keys, new IndexOptions().unique(true));
This can be done using "_id" field although this use is discouraged.
suppose you want the names to be unique, then you can put the names in "_id" column and as you might know "_id" column is unique for each entry.
BasicDBObject bdbo = new BasicDBObject("_id","amit");
Now , no other entry can have name as "amit" in the collection.This can be one of the way you are asking for.
As of Mongo's v3.0 Java driver, the code to create the index looks like:
public void createUniqueIndex() {
Document index = new Document("fieldName", 1);
MongoCollection<Document> collection = client.getDatabase("dbName").getCollection("CollectionName");
collection.createIndex(index, new IndexOptions().unique(true));
}
// And test to verify it works as expected
#Test
public void testIndex() {
MongoCollection<Document> collection = client.getDatabase("dbName").getCollection("CollectionName");
Document newDoc = new Document("fieldName", "duplicateValue");
collection.insertOne(newDoc);
// this will throw a MongoWriteException
try {
collection.insertOne(newDoc);
fail("Should have thrown a mongo write exception due to duplicate key");
} catch (MongoWriteException e) {
assertTrue(e.getMessage().contains("duplicate key"));
}
}
Theon solution didn't work for me, but this one did:
BasicDBObject query = new BasicDBObject(<fieldname>, 1);
collection.ensureIndex(query, <index_name>, true);
I am not a Java programmer however you can probably convert this over.
MongoDB by default does have a primary key known as the _id you can use upsert() or save() on this key to prevent the document from being written twice like so:
var doc = {'name': 'sam'};
db.users.insert(doc); // doc will get an _id assigned to it
db.users.insert(doc); // Will fail since it already exists
This will stop immediately duplicates. As to multithread safe inserts under certain conditions: well, we would need to know more about your condition in that case.
I should add however that the _id index is unqiue by default.
using pymongo it looks like:
mycol.create_index("id", unique=True)
where myCol is the collection in the DB
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pymongo
myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["mydatabase"]
mycol = mydb["customers"]
mycol.create_index("id", unique=True)
mydict = {"name": "xoce", "address": "Highway to hell 666", "id": 1}
x = mycol.insert_one(mydict)
Prevent mongoDB to save duplicate email
UserSchema.path('email').validate(async(email)=>{
const emailcount = await mongoose.models.User.countDocuments({email})
return !emailcount
}, 'Email already exits')
May this help ur question...
worked for me..
use in user model.
refer for explaination
THANKS...