I am using spring-data-mongodb and I want to use a cursor for an aggregate operation.
MongoTemplate.stream() gets a Query, so I tried creating the Aggregation instance, convert it to a DbObject using Aggregation.toDbObject(), created a BasicQuery using the DbObject and then invoke the stream() method.
This returns an empty cursor.
Debugging the spring-data-mongodb code shows that MongoTemplate.stream() uses the FindOperation, which makes me thinkspring-data-mongodb does not support streaming an aggregation operation.
Has anyone been able to stream the results of an aggregate query using spring-data-mongodb?
For the record, I can do it using the Java mongodb driver, but I prefer using spring-data.
EDIT Nov 10th - adding sample code:
MatchOperation match = Aggregation.match(Criteria.where("type").ne("AType"));
GroupOperation group = Aggregation.group("name", "type");
group = group.push("color").as("colors");
group = group.push("size").as("sizes");
TypedAggregation<MyClass> agg = Aggregation.newAggregation(MyClass.class, Arrays.asList(match, group));
MongoConverter converter = mongoTemplate.getConverter();
MappingContext<? extends MongoPersistentEntity<?>, MongoPersistentProperty> mappingContext = converter.getMappingContext();
QueryMapper queryMapper = new QueryMapper(converter);
AggregationOperationContext context = new TypeBasedAggregationOperationContext(MyClass.class, mappingContext, queryMapper);
// create a BasicQuery to be used in the stream() method by converting the Aggregation to a DbObject
BasicQuery query = new BasicQuery(agg.toDbObject("myClass", context));
// spring-mongo attributes the stream() method to find() operationsm not to aggregate() operations so the stream returns an empty cursor
CloseableIterator<MyClass> iter = mongoTemplate.stream(query, MyClass.class);
// this is an empty cursor
while(iter.hasNext()) {
System.out.println(iter.next().getName());
}
The following code, not using the stream() method, returns the expected non-empty result of the aggregation:
AggregationResults<HashMap> result = mongoTemplate.aggregate(agg, "myClass", HashMap.class);
For those who are still trying to find the answer to this:
From spring-data-mongo version 2.0.0.M4 onwards (AFAIK) MongoTemplate got an aggregateStream method.
So you can do the following:
AggregationOptions aggregationOptions = Aggregation.newAggregationOptions()
// this is very important: if you do not set the batch size, you'll get all the objects at once and you might run out of memory if the returning data set is too large
.cursorBatchSize(mongoCursorBatchSize)
.build();
data = mongoTemplate.aggregateStream(Aggregation.newAggregation(
Aggregation.group("person_id").count().as("count")).withOptions(aggregationOptions), collectionName, YourClazz.class);
Related
https://github.com/spring-projects/spring-data-mongodb/issues/2821
https://jira.spring.io/browse/DATAMONGO-1922?redirect=false
I have been looking for ReactiveBulk operations to update documents as a batch in Spring WebFlux.
Like in the Mongo Template
var bulkOps = mongoTemplate.bulkOps()
for(dto : List<DTO> DTOs) {
Query query = new Query();
query.addCriteria(Criteria.where(ID).is(dto.getId()));
Update update = new Update()
.set(STATUS, dto.getStatus())
bulkOps.updateOne(query, update)
}
bulkOps.execute();
is there a workaround to implement that operation in reactive way since reactivemongotemplate look like does not support that operation currently?
similiar topic in so: Bulk Update with ReactiveMongoTemplate
Quickly remind that Bulk is different than UpdateMulti.
Bulk is meant to write multiple queries, therefore, update various objects. On the other side, UpdateMulti is intended to update all lines where expression matches
As for reactive bulk, you should be able to use ReactiveMongoTemplate and implement something like that:
reactiveMongoTemplate.getCollection("collection_name")
.flatMap(mongoCollection -> {
List<UpdateOneModel<Document>> operations = DTOs.stream()
.map(dto -> {
Document doc = new Document("status", dto.getStatus());
reactiveMongoTemplate.getConverter().write(dto, doc);
Document filter = new Document("id", dto.getId());
return new UpdateOneModel<Document>(filter, new Document("$set", doc));
}
}).toList();
return Mono.from(mongoCollection.bulkWrite(operations));
});
You can also add custom options to bulkWrite() if you desire.
If more filter is needed, you can append them to the document
Document filter = new Document("id", dto.getId())
.append("country", dto.getCountry);
I'm trying to insert a single ArrayList containing JSONS into a mongodb collection with this,
MongoClient mongo = new MongoClient("localhost", 27017);
DB db = mongo.getDB("structure");
DBCollection collection = db.getCollection("chapter");
List<Document> data = new ArrayList<>();
collection.insertMany(data);
String str = "[{\"id\":1,\"data\":\"data1\"},{\"id\":2,\"data\":\"data2\"}]";
DBObject dbObject = (DBObject) JSON.parse(str);
collection.insert(dbObject);
But I get the exception,
Exception in thread "main" java.lang.IllegalArgumentException: BasicBSONList can only work with numeric keys, not: [_id]
Can anyone show me the correct way to do this?
Insert ArrayList mongodb
The question above is about bulk insert of JSONS, not as a single one.
My question is unique
The exception gives a hint of what the problem is: a list cannot be used as a record (or a map-like data structure).
To quote the MongoDB documentation on documents that compose a collection:
Document Structure
MongoDB documents are composed of field-and-value
pairs and have the following structure:
{
field1: value1,
field2: value2,
field3: value3,
...
fieldN: valueN
}
So what you need to do, in your case, because you just want to insert many documents in one call, is to use collection.insertMany:
List<Document> documents = ...; //convert your list to a List<Document>
collection.insertMany(documents);
Have a look at this
https://docs.mongodb.com/manual/reference/method/db.collection.insertMany/
List<DBobject> data = new ArrayList<>();
Colletions.insertMany(data);
I am trying to get the Data from Mongodb whose Model/Domain is unknown.
Can i get that using Mongo Template.
e.g.
mongoTemplate.find(query,<Dynamic Class?>)
You can use DBObject. If you take a look at its implementations (BasicDBObject...) it's an HashMap (key/values) containing all fields:
#Autowired
private MongoTemplate mongoTemplate;
DBObject query = new BasicDBObject("field", "value");
DBCursor dbCursor = mongoTemplate.getCollection("collectionName").find(query);
Iterator<DBObject> iterator = dbCursor.iterator();
while(iterator.hasNext()){
Object value = iterator.next().get("otherfield");
}
I want to perform a query on a field that is greater than or equal to, AND less than or equal to(I'm using java btw). In other words. >= and <=. As I understand, mongoDB has $gte and $lte operators, but I can't find the proper syntax to use it. The field i'm accessing is a top-level field.
I have managed to get this to work:
FindIterable<Document> iterable = db.getCollection("1dag").find(new Document("timestamp", new Document("$gt", 1412204098)));
as well ass...
FindIterable<Document> iterable = db.getCollection("1dag").find(new Document("timestamp", new Document("$lt", 1412204098)));
But how do you combine these with each other?
Currently I'm playing around with a statement like this, but it does not work:
FindIterable<Document> iterable5 = db.getCollection("1dag").find(new Document( "timestamp", new Document("$gte", 1412204098).append("timestamp", new Document("$lte",1412204099))));
Any help?
Basically you require a range query like this:
db.getCollection("1dag").find({
"timestamp": {
"$gte": 1412204098,
"$lte": 1412204099
}
})
Since you need multiple query conditions for this range query, you can can specify a logical conjunction (AND) by appending conditions to the query document using the append() method:
FindIterable<Document> iterable = db.getCollection("1dag").find(
new Document("timestamp", new Document("$gte", 1412204098).append("$lte", 1412204099)));
The constructor new Document(key, value) only gets you a document with one key-value pair. But in this case you need to create a document with more than one. To do this, create an empty document, and then add pairs to it with .append(key, value).
Document timespan = new Document();
timespan.append("$gt", 1412204098);
timespan.append("$lt", 1412204998);
// timespan in JSON:
// { $gt: 1412204098, $lt: 1412204998}
Document condition = new Document("timestamp", timespan);
// condition in JSON:
// { timestamp: { $gt: 1412204098, $lt: 1412204998} }
FindIterable<Document> iterable = db.getCollection("1dag").find(condition);
Or if you really want to do it with a one-liner without temporary variables:
FindIterable<Document> iterable = db.getCollection("1dag").find(
new Document()
.append("timestamp", new Document()
.append("$gt",1412204098)
.append("$lt",1412204998)
)
);
I have the following code:
#RequestMapping(value = "/envinfo", method = RequestMethod.GET)
#ResponseBody
public Map getEnvInfo()
{
BasicQuery basicQuery = new BasicQuery("{_id:'51a29f6413dc992c24e0283e'}", "{'envinfo':1, '_id': false }");
Map envinfo= mongoTemplate.findOne(basicQuery, Map.class, "jvmInfo");
return envinfo;
}
As you can notice, the code:
Retrieves JSON from MongoDB
Converts it to a Map object
The Map object is then converted to JSON by Spring MongoData before it is returned to the browser.
Is it possible to directly return the raw json from MongoDb without going through the intermediate conversion steps?
There's two way's you can do this right now:
1. Using the CollectionCallback on MongoTemplate
You can use a CollectionCallback to deal with the returned DBObject directly and simply toString() it:
template.execute("jvmInfo", new CollectionCallback<String>() {
String doInCollection(DBCollection collection) {
DBCursor cursor = collection.find(query)
return cursor.next().toString()
}
}
Yo'll still get the exception translation into Spring's DataAccessExceptions. Note, that this is slightly brittle as we expect only a single result to be returned for the query but that's probably something you have to take care of when trying to produce a String anyway.
2. Register a Converter from DBObject to String
You can implement a Spring Converter to do the toString() for you.
class DBObjectToStringConverter implements Converter<DBObject, String> {
public String convert(DBObject source) {
return source == null ? null : source.toString();
}
}
You can then either use the XML configuration or override customConversions() to return a new CustomConversions(Arrays.asList(new DBObjectToStringConverter())) to get it registered with your MongoConverter. You can then simply do the following:
String result = mongoTemplate.findOne(basicQuery, String.class, "jvmInfo");
I will add the just showed converter to Spring Data MongoDB and register it by default for the upcoming 1.3 GA release and port the fix back to 1.2.x as part of the fix for DATAMONGO-743.
As Oliver points out, you can use Spring Data for that, but an alternative which you may or may not prefer would be to use MongoDB's more low-level Java Driver. Take a look at the MongoDB Java Driver 3.x or MongoDB Java Driver 2.x documentation for instructions on using that driver.
Basically, what you need to do is this:
MongoDB Java Driver 3.x
MongoClient mongoClient = new MongoClient();
MongoDatabase db = mongoClient.getDatabase("test");
MongoCollection coll = db.getCollection("testCollection");
BasicDBObject query = new BasicDBObject("_id", "51a29f6413dc992c24e0283e");
try (MongoCursor<Document> cursor = collection.find(query).iterator()) {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
}
MongoDB Java Driver 2.x
MongoClient mongoClient = new MongoClient();
DB db = mongoClient.getDB("test");
DBCollection coll = db.getCollection("testCollection");
BasicDBObject query = new BasicDBObject("_id", "51a29f6413dc992c24e0283e");
try (DBCursor cursor = coll.find(query)) {
while(cursor.hasNext()) {
System.out.println(cursor.next());
}
}
That will print out all the documents in the collection that have a field _id with a value 51a29f6413dc992c24e0283e.
For me the following worked perfectly:
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.query.Query;
import org.bson.Document;
List<Document> documents = mongoTemplate.find(query, Document.class, "collection_name");