I have values that coming from MongoDB stored in a DBObject. and I needed to store that value in a set one by one. As being new to MongoDB I am not actually getting idea coherently how to proceed that.
String date = sdf.format(cal2.getTime());
List<String> dateList = new ArrayList<String>();
for (int i = 0; i < 6; i++) {
Date dateParsed = sdf.parse(date);
dateParsed.setDate(dateParsed.getDate() - i);
dateList.add(sdf.format(dateParsed));
}
Set<String> values2= new HashSet<String>();
for (String str : dateList) {
BasicDBObject find1 = new BasicDBObject("_ky", str);
DBObject values1= someDB.findOne(find1);
Iterator iter = values1.iterator(); /*giving error the method not found (becasue values1 is a dbObject)*/
while (iter.hasNext()) {
values2.add(//???//);
}
}
Any help on how can I iterate the DBObject- values1 and add those values in a set- values2 would be great.
You can call values1.keySet() and iterate over that and get() any values or use values1.toMap() and iterate that Map like you would any other.
The primary abstraction in the Mongo Java Driver is the DBObject which acts like a wrapper around Java's Map<String,Object>.
import java.util.Arrays;
import com.mongodb.BasicDBList;
import com.mongodb.BasicDBObject;
import com.mongodb.DBObject;
public class DBObjectKeySetDemo {
public static void main( String[] args ) {
DBObject dbo = new BasicDBObject( "firstName", "Robert" ).append( "lastName", "Kuhar" );
dbo.put( "age", 49 );
dbo.put( "hobbies", Arrays.asList( "Fly Fishing", "Board Games", "Roller Derby" ) );
DBObject browser = new BasicDBObject( "implementation", "Chrome" );
browser.put( "vendor", "Google" );
BasicDBList bookmarks = new BasicDBList();
bookmarks.add( new BasicDBObject( "name", "StackOverflow" ).append( "URL", "http://stackoverflow.com" ) );
bookmarks.add( new BasicDBObject( "name", "MMS" ).append( "URL", "https://mms.mongodb.com" ) );
browser.put( "bookmarks", bookmarks );
dbo.put( "browser", browser );
for ( String key : dbo.keySet() ) {
System.out.println( "key: " + key + " value: " + dbo.get( key ) );
}
System.out.println( "dbo: " + dbo );
}
}
The "gotcha" is that you can only work directly with "top level" element. For example, in the above example, through the Java API, you have no way to directly reference "browser.vendor". Through the Java API you would have to first get the "browser" sub-document, and then get the "vendor" field.
Clear? As mud? It helped me to just think of the abstraction as a Map<String,Object> where Object, in the case of a sub-document, may itself be a Map<String,Object>.
Related
I'm trying to implement bulk update for a list of objects. But I'm getting the following error
java.lang.IllegalArgumentException: Invalid BSON field name bookingId
at org.bson.AbstractBsonWriter.writeName(AbstractBsonWriter.java:532)
at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:114)
at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:398)
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:377)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
The implementation for the updates are as follows:
public void _deleteAll( ArrayList<T> pojos ) {
Assert.notNull( pojos, "Entity must not be null!" );
BulkOperations ops = mongoTemplate.bulkOps( BulkOperations.BulkMode.ORDERED, className, collectionName );
for ( T pojo : pojos ) {
System.out.println( ( ( BaseEntity ) pojo ).get_id() );
( ( BaseEntity ) pojo ).setIsDeleted( true );
Query query = new Query( Criteria.where( "_id" ).is( ( ( BaseEntity ) pojo ).get_id() ) );
Document document = Document.parse( pojo.toString() ); //toString() returns JSON string
Update update = Update.fromDocument( document );
ops.updateOne( query, update );
}
ops.execute();
}
Second approach with WriteModel
public void _deleteAll( ArrayList<T> pojos ) {
MongoCollection<Document> collection = mongoTemplate.getCollection( collectionName );
List<WriteModel<Document>> writes = new ArrayList<>();
for ( T pojo : pojos ) {
( ( BaseEntity ) pojo ).setIsDeleted( true );
writes.add(
new UpdateOneModel<Document>(
new Document().append( "_id", ( ( BaseEntity ) pojo ).get_id() ), Document.parse( pojo.toString() )
)
);
}
collection.bulkWrite( writes );
}
The JSON data is similar to this:
{
"_id" : ObjectId("5d81d87ac290c518433e66d9"),
"bookingId" : 100344391,
"body" : "{\"title\”:\"Some title\”,\”booking_room_id\":null,\”message\”:\"some message\”}",
“nestedObject" : {
“someKey" : “some value"
},
"created_at" : ISODate("2019-09-18T07:10:50.067Z"),
"updated_at" : ISODate("2019-09-18T07:28:16.062Z"),
"isDeleted" : false,
"_class" : "com.test.mongo_test.entity.mongo.SomePojo"
}
Found the answer here. Basically what I had to do was add the query command $set in the document.
writes.add(
new UpdateOneModel<>(
new Document( "_id", new ObjectId( id ) ),
new Document( "$set", Document.parse( pojo.toString() ) )
)
);
Lets see what we have. First file [Interface Class]:
list arrayList
list linkedList
Second file[Class countOfInstanse]:
arrayList 120
linkedList 4
I would like to join this two files by key[Class] and get count per each Interface:
list 124
and code:
public class Main
{
public static void main( String[] args )
{
String docPath = args[ 0 ];
String wcPath = args[ 1 ];
String stopPath = args[ 2 ];
Properties properties = new Properties();
AppProps.setApplicationJarClass( properties, Main.class );
AppProps.setApplicationName( properties, "Part 1" );
AppProps.addApplicationTag( properties, "lets:do:it" );
AppProps.addApplicationTag( properties, "technology:Cascading" );
FlowConnector flowConnector = new Hadoop2MR1FlowConnector( properties );
// create source and sink taps
Tap docTap = new Hfs( new TextDelimited( true, "\t" ), docPath );
Tap wcTap = new Hfs( new TextDelimited( true, "\t" ), wcPath );
Fields stop = new Fields( "class" );
Tap classTap = new Hfs( new TextDelimited( true, "\t" ), stopPath );
// specify a regex operation to split the "document" text lines into a token stream
Fields token = new Fields( "token" );
Fields text = new Fields( "interface" );
RegexSplitGenerator splitter = new RegexSplitGenerator( token, "[ \\[\\]\\(\\),.]" );
Fields fieldSelector = new Fields( "interface", "class" );
Pipe docPipe = new Each( "token", text, splitter, fieldSelector );
// define "ScrubFunction" to clean up the token stream
Fields scrubArguments = new Fields( "interface", "class" );
docPipe = new Each( docPipe, scrubArguments, new ScrubFunction( scrubArguments ), Fields.RESULTS );
Fields text1 = new Fields( "amount" );
// RegexSplitGenerator splitter = new RegexSplitGenerator( token, "[ \\[\\]\\(\\),.]" );
Fields fieldSelector1 = new Fields( "class", "amount" );
Pipe stopPipe = new Each( "token1", text1, splitter, fieldSelector1 );
Pipe tokenPipe = new CoGroup( docPipe, token, stopPipe, text, new InnerJoin() );
tokenPipe = new Each( tokenPipe, text, new RegexFilter( "^$" ) );
// determine the word counts
Pipe wcPipe = new Pipe( "wc", tokenPipe );
wcPipe = new Retain( wcPipe, token );
wcPipe = new GroupBy( wcPipe, token );
wcPipe = new Every( wcPipe, Fields.ALL, new Count(), Fields.ALL );
// connect the taps, pipes, etc., into a flow
FlowDef flowDef = FlowDef.flowDef().setName( "wc" ).addSource( docPipe, docTap ).addSource( stopPipe, classTap ).addTailSink( wcPipe, wcTap );
// write a DOT file and run the flow
Flow wcFlow = flowConnector.connect( flowDef );
wcFlow.writeDOT( "dot/wc.dot" );
wcFlow.complete();
}
}
[I decided to resolve this issue step-by-step and left final result here for others. So first step - Couldn`t join two files with one key via Cascading (Not Completed yet) ]
I would convert the two files to two Map objects, iterate through the keys and sum up the numbers. Then you can write them back to a file.
Map<String,String> nameToType = new HashMap<String,String>();
Map<String,Integer> nameToCount = new HashMap<String,Integer>();
//fill Maps from file here
Map<String,Integer> result = new HashMap<String,Integer>();
for (String name: nameToType.keyset())
{
String type = nameToType.get(name);
int count = nameToCount.get(type);
if (!result.containsKey(type))
result.put(type,0);
result.put(type, result.get(type) + count);
}
For this example from the MongoDB documentation, how do I write the query using MongoTemplate?
db.sales.aggregate(
[
{
$group : {
_id : { month: { $month: "$date" }, day: { $dayOfMonth: "$date" }, year: { $year: "$date" } },
totalPrice: { $sum: { $multiply: [ "$price", "$quantity" ] } },
averageQuantity: { $avg: "$quantity" },
count: { $sum: 1 }
}
}
]
)
Or in general, how do I group by a calculated field?
You can actually do something like this with "project" first, but to me it's a little counter-intuitive to require a $project stage before hand:
Aggregation agg = newAggregation(
project("quantity")
.andExpression("dayOfMonth(date)").as("day")
.andExpression("month(date)").as("month")
.andExpression("year(date)").as("year")
.andExpression("price * quantity").as("totalAmount"),
group(fields().and("day").and("month").and("year"))
.avg("quantity").as("averavgeQuantity")
.sum("totalAmount").as("totalAmount")
.count().as("count")
);
Like I said, counter-intuitive as you should just be able to declare all of this under $group stage, but the helpers don't seem to work this way. The serialization comes out a bit funny ( wraps the date operator arguments with arrays ) but it does seem to work. But still, this is two pipeline stages rather than one.
What is the problem with this? Well by separating the stages the stages the "project" portion forces the processing of all of the documents in the pipeline in order to get the calculated fields, that means it passes through everything before moving on to the group stage.
The difference in processing time can be clearly seen by running the queries in both forms. With a separate project stage, on my hardware takes three times longer to execute than the query where all fields are calculated during the "group" operation.
So it seems the only present way to construct this properly is by building the pipeline object yourself:
ApplicationContext ctx =
new AnnotationConfigApplicationContext(SpringMongoConfig.class);
MongoOperations mongoOperation = (MongoOperations) ctx.getBean("mongoTemplate");
BasicDBList pipeline = new BasicDBList();
String[] multiplier = { "$price", "$quantity" };
pipeline.add(
new BasicDBObject("$group",
new BasicDBObject("_id",
new BasicDBObject("month", new BasicDBObject("$month", "$date"))
.append("day", new BasicDBObject("$dayOfMonth", "$date"))
.append("year", new BasicDBObject("$year", "$date"))
)
.append("totalPrice", new BasicDBObject(
"$sum", new BasicDBObject(
"$multiply", multiplier
)
))
.append("averageQuantity", new BasicDBObject("$avg", "$quantity"))
.append("count",new BasicDBObject("$sum",1))
)
);
BasicDBObject aggregation = new BasicDBObject("aggregate","collection")
.append("pipeline",pipeline);
System.out.println(aggregation);
CommandResult commandResult = mongoOperation.executeCommand(aggregation);
Or if all of that seems to terse to you, then you can always work with the JSON source and parse that. But of course, it has to be valid JSON:
String json = "[" +
"{ \"$group\": { "+
"\"_id\": { " +
"\"month\": { \"$month\": \"$date\" }, " +
"\"day\": { \"$dayOfMonth\":\"$date\" }, " +
"\"year\": { \"$year\": \"$date\" } " +
"}, " +
"\"totalPrice\": { \"$sum\": { \"$multiply\": [ \"$price\", \"$quantity\" ] } }, " +
"\"averageQuantity\": { \"$avg\": \"$quantity\" }, " +
"\"count\": { \"$sum\": 1 } " +
"}}" +
"]";
BasicDBList pipeline = (BasicDBList)com.mongodb.util.JSON.parse(json);
Another alternative is to use a custom aggregation operation class defined like so:
public class CustomAggregationOperation implements AggregationOperation {
private DBObject operation;
public CustomAggregationOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
Then implement it in the pipeline like so:
Aggregation aggregation = newAggregation(
new CustomAggregationOperation(
new BasicDBObject(
"$group",
new BasicDBObject("_id",
new BasicDBObject("day", new BasicDBObject("$dayOfMonth", "$date" ))
.append("month", new BasicDBObject("$month", "$date"))
.append("year", new BasicDBObject("$year", "$date"))
)
.append("totalPrice", new BasicDBObject(
"$sum", new BasicDBObject(
"$multiply", Arrays.asList("$price","$quantity")
)
))
.append("averageQuantity", new BasicDBObject("$avg", "$quantity"))
.append("count",new BasicDBObject("$sum",1))
)
)
)
So it's basically just an interface that is consistent with that used by the existing pipeline helpers, but instead of using other helper methods it directly takes a DBObject definition to return in pipeline construction.
I'm testing against a database with one document in it. With the following code I get the following result from the database
BasicDBObject keys = new BasicDBObject();
keys.put("symbol", 1 );
keys.put("price", 1 );
keys.put("Exchange", "SH");
keys.put("NumShares", new BasicDBObject("$gte", 0) );
BasicDBObject empty = new BasicDBObject();
DBCursor cursor_02 = coll_tmp.find( empty, keys );
System.out.println( "<2> " + cursor_02.count());
while(cursor_02.hasNext()) {
System.out.println( "<2> " + cursor_02.next());
}
This will print out the following data
<2> { <id & oid> , "symbol" : "ESLR" , "Exchange" : "SH"}
The document also has a "NumShares" field set to a value of 2.34508872. Wanting to try out the $gt, $gte, $lt, $lte, and $ne operators, I add in a line of code after the last 'keys.put()' line.
keys.put("Price", new BasicDBObject("$gte", 0) );
This code should continue to select the lone document, as it's NumShares field is indeed greater than or equal to zero. Instead I get the following error.
Can't canonicalize query: BadValue Unsupported projection option: NumShares: { $gte: 0 }
The goal ultimately is to have a MongoDB equivalent of the SQL query "SELECT price, symbol, exhange FROM stock WHERE NumShares >= 0, etc..."
I realize that mkyong doesn't cover this particular query structure in his tutorial, so I'm asking you all. Thanks
You want to put your criteria in the query, not in the list of keys you want to return:
BasicDBObject keys = new BasicDBObject(); // will specify the returned fields (SELECT)
keys.put("symbol", 1 );
keys.put("price", 1 );
keys.put("Exchange", 1);
BasicDBObject query = new BasicDBObject(); // will select the documents you want (WHERE)
query.put("numShares", new BasicDBObject("$gte", 0) );
DBCursor cursor_02 = coll_tmp.find( query, keys );
System.out.println( "<2> " + cursor_02.count());
while(cursor_02.hasNext()) {
System.out.println( "<2> " + cursor_02.next());
}
You need to put your result-restricting query arguments to your first Object of the query. The second Object is a Projection of the result. It can only specify the tags to be displayed (as true/1 or false/0) there are no other restrictions like "$lte" allowed. Putting it to the query object like in the following should work:
BasicDBObject keys = new BasicDBObject();
keys.put("symbol", 1 );
keys.put("price", 1 );
keys.put("Exchange", 1);
BasicDBObject search = new BasicDBObject();
search.put("Exchange", "SH");
search.put("numShares", new BasicDBObject("$gte", 0) );
DBCursor cursor_02 = coll_tmp.find(search, keys);
Also MongoDBs keys of documents are case-sensitive. So if your key is "NumShares" you cannot find it by looking for "numShares".
I need to implement "Popular Keyword" in my web app so that most searched keywords will show there.. For that i have created a db in mongo. I need to retrieve data from mongo so that mostly occuring keywords should sort and should return to my page. In sql it is just like as,
select names,count(names) from keyword_db group by names order by names;
need both java code and mongo shell query..
You might want to try for the mongo shell query (I'm sorting in descending order:
db.keyword_db.aggregate ( [
{ $group: { _id: "$names", count: { $sum: 1 } } },
{ $sort: { count: -1 } } ] )
For the Java version:
DBCollection myColl = db.getCollection("keyword_db");
// for the $group operator
DBObject groupFields = new BasicDBObject( "_id", "$names");
groupFields.put("count", new BasicDBObject( "$sum", 1));
DBObject group = new BasicDBObject("$group", groupFields );
// for the $sort operator
DBObject sortFields = new BasicDBObject("count", -1);
DBObject sort = new BasicDBObject("$sort", sortFields );
// run the aggregation
AggregationOutput output = myColl.aggregate(group, sort);
System.out.println( output.getCommandResult() );
The following page may help you also:
http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
For sorting you can use following..1 for ascending and -1 for desc.
db.keyword_db.find( { names: "A" } ).sort( { user_id: 1 } )
and for groupBy You can use Aggregation framework..
http://docs.mongodb.org/manual/reference/method/db.collection.group/