We Keep Coding

MongoDB how to find all documents who's array contains a specific string?

Below is an example of my "Event" document in MongoDB. I want to be able to query all of the Event documents where the attendees array contains "623d03730e82c57fefa52fb2" (a user ID).
Here is one of my event documents:
_id: ObjectId(623ce74372a28f08dea6c959)
description: "Fun BBQ to celebrate my 21st!"
host: "623d03730e82c57fefa52fb2"
invitees: Array
location: "My address..."
name: "Fun Birthday BBQ"
private: true
date: "03/28/22"
end: "11:15 PM"
start: "06:35 PM"
attendees:Array
0: "623d03730e82c57fefa52fb2"
Here is my broken query code:
String id = "623d03730e82c57fefa52fb2";
// I have also tried Document queryFilter = new Document("attendees", id);
Document queryFilter = new Document("attendees", new Document("$in", Arrays.asList(id)));
The above code always returns an empty result. To clarify I am using Java and MongoDB Realms but that shouldn't matter.

You don't need $in, use only $eq is ok.
db.collection.find({
attendees: "623d03730e82c57fefa52fb2"
})
mongoplayground

For easier and more efficient queries, it's important that the types of field values remain consistent.
For example, if "_id" is stored at an ObjectId, then query parameters should also be of type ObjectId. Likewise, if they are strings, then consistently use strings.
If different value types are stored for individual field values, successful queries can still be possible, but not as efficiently since the field types must be considered in the queries. For example, if trying to find a doc by a field that may have a string or an ObjectId type, the query must either search for both types, or the query writer must know the type beforehand. It's easier and more efficient to just pick one type for a field and stick to it.

Spring Batch Paging with sortKeys and parameter values

I have a Spring Batch project running in Spring Boot that is working perfectly fine. For my reader I'm using JdbcPagingItemReader with a MySqlPagingQueryProvider.
#Bean
public ItemReader<Person> reader(DataSource dataSource) {
MySqlPagingQueryProvider provider = new MySqlPagingQueryProvider()
provider.setSelectClause(ScoringConstants.SCORING_SELECT_STATEMENT)
provider.setFromClause(ScoringConstants.SCORING_FROM_CLAUSE)
provider.setSortKeys("p.id": Order.ASCENDING)
JdbcPagingItemReader<Person> reader = new JdbcPagingItemReader<Person>()
reader.setRowMapper(new PersonRowMapper())
reader.setDataSource(dataSource)
reader.setQueryProvider(provider)
//Setting these caused the exception
reader.setParameterValues(
startDate: new Date() - 31,
endDate: new Date()
)
reader.afterPropertiesSet()
return reader
}
However, when I modified my query with some named parameters to replace previously hard coded date values and set these parameter values on the reader as shown above, I get the following exception on the second page read (the first page works fine because the _id parameter hasn't been made use of by the paging query provider):
org.springframework.dao.InvalidDataAccessApiUsageException: No value supplied for the SQL parameter '_id': No value registered for key '_id'
at org.springframework.jdbc.core.namedparam.NamedParameterUtils.buildValueArray(NamedParameterUtils.java:336)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.getPreparedStatementCreator(NamedParameterJdbcTemplate.java:374)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:192)
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:199)
at org.springframework.batch.item.database.JdbcPagingItemReader.doReadPage(JdbcPagingItemReader.java:218)
at org.springframework.batch.item.database.AbstractPagingItemReader.doRead(AbstractPagingItemReader.java:108)
Here is an example of the SQL, which has no WHERE clause by default. One does get created automatically when the second page is read:
select *, (select id from family f where date_created between :startDate and :endDate and f.creator_id = p.id) from person p
On the second page, the sql is modified to the following, however it seems that the named parameter for _id didn't get supplied:
select *, (select id from family f where date_created between :startDate and :endDate and f.creator_id = p.id) from person p WHERE id > :_id
I'm wondering if I simply can't use the MySqlPagingQueryProvider sort keys together with additional named parameters set in JdbcPagingItemReader. If not, what is the best alternative to solving this problem? I need to be able to supply parameters to the query and also page it (vs. using the cursor). Thank you!

I solved this problem with some intense debugging. It turns out that MySqlPagingQueryProvider utilizes a method getSortKeysWithoutAliases() when it builds up the SQL query to run for the first page and for subsequent pages. It therefore appends and (p.id > :_id) instead of and (p.id > :_p.id). Later on, when the second page sort values are created and stored in JdbcPagingItemReader's startAfterValues field it will use the original "p.id" String specified and eventually put into the named parameter map the pair ("_p.id",10). However, when the reader tries to fill in _id in the query, it doesn't exist because the reader used the non-alias removed key.
Long story short, I had to remove the alias reference when defining my sort keys.
provider.setSortKeys("p.id": Order.ASCENDING)
had to change to in order for everything to work nicely together
provider.setSortKeys("id": Order.ASCENDING)

I had the same issue and got another possible solution.
My table T has a primary key field INTERNAL_ID.
The query in JdbcPagingItemReader was like this:
SELECT INTERNAL_ID, ... FROM T WHERE ... ORDER BY INTERNAL_ID ASC
So, the key is: in some conditions, the query didn't return results, and then, raised the error above No value supplied for...
The solution is:
Check in a Spring Batch decider element if there are rows.
If it is, continue with chunk: reader-processor-writer.
It it's not, go to another step.
Please, note that they are two different scenarios:
At the beginning, there are rows. You get them by paging and finally, there are no more rows. This has no problem and decider trick is not required.
At the beginning, there are no rows. Then, this error raised, and the decider solved it.
Hope this helps.

MongoDB: insert documents with specific id instead of auto generated ObjectID

I need to insert documents on MongoDB (with specific id instead of auto generated ObjectID) using java..
To insert one document or update if exist, I tried use findOne to search for the id, if it doesn't exist then insert the id and then findAndModify. It works but I don't feel that it's efficient way, it's time consuming. Is there a better way to achieve that?
To insert multiple documents at once,I'm following this solution. but I don't know how I can insert my custom Id instead of objectID?
Any help will be appreciated

For your first problem MongoDB has upsert so
db.collection.update(
{query for id},
{document},
{upsert: true}
)
or in the Java driver
yourCollection.update(searchObject, modifiedObject, true, false);
If you want to set a custom ID you just set the _id key manually i.e.
yourBasicDBObject.put("_id",yourCustomId)
you just have to ensure it is unique for each document.
You will also need to set the _id in your modifiedObject otherwise a new one will be generated.
As for the bulk operations, just setting a custom ID for each document by giving the _id key should also work.

Try this #ebt_dev:
db.collection("collectionname").insertOne(data, { forceServerObjectId: false })

Getting "err" : "E11000 duplicate key error when inserting into mongo using the Java driver

Exception in thread "main" com.mongodb.MongoException$DuplicateKey: {
"serverUsed" : "localhost/127.0.0.1:27017" , "err" : "E11000 duplicate
key error index: twitterdb03.LevelAFollowers.$id dup key: { :
ObjectId('52d5636de408652b4853a8fe') }" , "code" : 11000 , "n" : 0 ,
"connectionId" : 12 , "ok" : 1.0}
I'm using mongo 2.11.1
Never had problems with simple write operations in java
myMap.put(inid, followersList);
myObj.putAll(myMap);
myIdMapCollection.insert(myObj);

I found an answer on this page. I’m guessing your code looks something like this (greatly simplified)?:
doc = {}
for i in xrange(2):
doc['i'] = i
collection.insert(doc)
The problem is that PyMongo injects an _id field into the document, if the _id field does not exist, before inserting it (_id is always generated client side with 10gen drivers). That means that the first time through the loop _id is added by the insert method. Since doc is defined outside the loop, each subsequent pass through the loop uses the same value for _id.
Solution:
Delete the key _id
for i in xrange(2):
doc['i'] = i
if '_id' in doc:
del doc['_id']
collection.insert(doc)
Or create manually a new one:
from bson.objectid import ObjectId
for i in xrange(2):
doc['i'] = i
doc['_id'] = ObjectId()
collection.insert(doc)

Try calling myIdMapCollection.save(myObj); instead of myIdMapCollection.insert(myObj);
The save method, unlike insert does upsert, meaning if a document contains _id, it replaces that document.
My guess is that you had fetched the DBObject using a cursor | query, had manipulated it, and you want to persist the changes. In that case, save is the right way to do it.
So, when calling insert the DBObject is already associated with _id, calling insert thus fails, because you already have a document with that _id in the collection, which should be unique (duplicate index error).

Mongodb avoid duplicate entries

I am newbie to mongodb. May I know how to avoid duplicate entries. In relational tables, we use primary key to avoid it. May I know how to specify it in Mongodb using java?

Use an index with the {unique:true} option.
// everyone's username must be unique:
db.users.createIndex({email:1},{unique:true});
You can also do this across multiple fields. See this section in the docs for more details and examples.
A unique index ensures that the indexed fields do not store duplicate values; i.e. enforces uniqueness for the indexed fields. By default, MongoDB creates a unique index on the _id field during the creation of a collection.
If you wish for null values to be ignored from the unique key, then you have to also make the index sparse (see here), by also adding the sparse option:
// everyone's username must be unique,
//but there can be multiple users with no email field or a null email:
db.users.createIndex({email:1},{unique:true, sparse:true});
If you want to create the index using the MongoDB Java Driver. Try:
Document keys = new Document("email", 1);
collection.createIndex(keys, new IndexOptions().unique(true));

This can be done using "_id" field although this use is discouraged.
suppose you want the names to be unique, then you can put the names in "_id" column and as you might know "_id" column is unique for each entry.
BasicDBObject bdbo = new BasicDBObject("_id","amit");
Now , no other entry can have name as "amit" in the collection.This can be one of the way you are asking for.

As of Mongo's v3.0 Java driver, the code to create the index looks like:
public void createUniqueIndex() {
Document index = new Document("fieldName", 1);
MongoCollection<Document> collection = client.getDatabase("dbName").getCollection("CollectionName");
collection.createIndex(index, new IndexOptions().unique(true));
}
// And test to verify it works as expected
#Test
public void testIndex() {
MongoCollection<Document> collection = client.getDatabase("dbName").getCollection("CollectionName");
Document newDoc = new Document("fieldName", "duplicateValue");
collection.insertOne(newDoc);
// this will throw a MongoWriteException
try {
collection.insertOne(newDoc);
fail("Should have thrown a mongo write exception due to duplicate key");
} catch (MongoWriteException e) {
assertTrue(e.getMessage().contains("duplicate key"));
}
}

Theon solution didn't work for me, but this one did:
BasicDBObject query = new BasicDBObject(<fieldname>, 1);
collection.ensureIndex(query, <index_name>, true);

I am not a Java programmer however you can probably convert this over.
MongoDB by default does have a primary key known as the _id you can use upsert() or save() on this key to prevent the document from being written twice like so:
var doc = {'name': 'sam'};
db.users.insert(doc); // doc will get an _id assigned to it
db.users.insert(doc); // Will fail since it already exists
This will stop immediately duplicates. As to multithread safe inserts under certain conditions: well, we would need to know more about your condition in that case.
I should add however that the _id index is unqiue by default.

using pymongo it looks like:
mycol.create_index("id", unique=True)
where myCol is the collection in the DB
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pymongo
myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["mydatabase"]
mycol = mydb["customers"]
mycol.create_index("id", unique=True)
mydict = {"name": "xoce", "address": "Highway to hell 666", "id": 1}
x = mycol.insert_one(mydict)

Prevent mongoDB to save duplicate email
UserSchema.path('email').validate(async(email)=>{
const emailcount = await mongoose.models.User.countDocuments({email})
return !emailcount
}, 'Email already exits')
May this help ur question...
worked for me..
use in user model.
refer for explaination
THANKS...