I installed 4.4 version MongoDB and reproduced replica set, sharding for my Java application.
Now, I am looking for the approach of run MongoDB shell command in my Java application for check and monitoring reproduced MongoDB Cluster's replica set and sharding status. I'm curious how to achieve this.
I tried to find answers. but, I couldn't found clear answer from other stackoverflow question regarding MongoDB. because most questions discuss base on 3.x version.
So, I researched MongoDB document and driver for 4.4 version and Tested that like below.
run MongoDB command
I tried to run "sh.status()" and "rs.status()" command using MongoDB 4.5 driver through MongoClient object and runCommand() method. but, Java return error message.
I realized that runCommand() method only allow "Mongo" commands mentioned on below link and not Mongo Shell Command.
link : https://www.mongodb.com/docs/v4.4/reference/command/
I couldn't found contrasting command with sh.status() from above link.
String uriString = "mongodb://"+userName+":"+passWord+"#"+targetHost+":"+portNo;
uriString = uriString +"/?authSource="+tgDBName+"&authMechanism="+authMechanism;
MongoClient mongoClient = MongoClients.create(uriString);
String commandString = "sh.status()";
Bson command = new BsonDocument(commandString, new BsonInt64(1));
Document commandResult = database.runCommand(command);
com.mongodb.MongoCommandException: Command failed with error 59 (CommandNotFound):
'no such cmd: sh.status()' on server ....
run javascript stored in MongoDB
I found other way to run sh.status() and get metrics.
That is run javascript stored in MongoDB's system.js collection using $eval.
So, I researched above method and found below links.
link 1 : How do I execute a MongoDB js script using the Java MongoDriver
link 2 : https://www.mongodb.com/docs/v4.4/tutorial/store-javascript-function-on-server/
And I reached answer that db.eval function was deprecated from 3.x and removed from 4.2 version.
Finally, I considering access to MongoDB using JSch and run prompt.
But, I think that is not formal approach. so, I want know other way for get metrics
like "run MongoDB Shell Command in java using 4.4 version driver"
Note : when run sh.status() command in mongos, I received metrics successfully like below.
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("625513e0838da178377f6900")
}
shards:
{ "_id" : "sh01", "host" : "sh01/WIN-BKEV4AO0KED:27011,WIN-BKEV4AO0KED:27012,WIN-BKEV4AO0KED:27013", "state" : 1 }
{ "_id" : "sh02", "host" : "sh02/WIN-BKEV4AO0KED:27021,WIN-BKEV4AO0KED:27022,WIN-BKEV4AO0KED:27023", "state" : 1 }
active mongoses:
"4.4.13" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
rs.status() is a mongo shell helper for db.adminCommand({ replSetGetStatus: 1 })
I don't work with Java, but I guess you must use MongoDatabase.runCommand. Could be similar to this:
MongoDatabase database = mongoClient.getDatabase("admin");
String commandString = "replSetGetStatus";
Bson command = new BsonDocument(commandString, new BsonInt64(1));
Document commandResult = database.runCommand(command);
For sh.status() or db.printShardingStatus() it is a bit more difficult. Documentation says:
The db.printShardingStatus() method run in mongosh does not return JSON. Use db.printShardingStatus() for manual inspection, and Config Database in scripts.
printShardingStatus
function printShardingStatus(configDB, verbose) {
// configDB is a DB object that contains the sharding metadata of interest.
// Defaults to the db named "config" on the current connection.
if (configDB === undefined)
configDB = db.getSiblingDB('config');
var version = configDB.getCollection("version").findOne();
if (version == null) {
print(
"printShardingStatus: this db does not have sharding enabled. be sure you are connecting to a mongos from the shell and not to a mongod.");
return;
}
var raw = "";
var output = function(indent, s) {
raw += sh._shardingStatusStr(indent, s);
};
output(0, "--- Sharding Status --- ");
output(1, "sharding version: " + tojson(configDB.getCollection("version").findOne()));
... many more lines
If you really like to get identical output to db.printShardingStatus(), then you would need to convert the content from that function into Java.
Related
i'm using
Trans.setParameterValue("whereStr", "");
Trans.sendToSlaveServer(transMeta, transExecutionConfiguration, rep, null)
to set a Parameter for a kettle Transformation, it will run on a remote server.
However, the remote server can't find the ParameterValue : whereStr.
Any Suggestions？
here is my Transformation setting :
setting1
setting2
problem solved.
1. remove ktr parameters setting
2. using
transMeta.addParameterDefinition("whereString", " and a.id > 1000021 ", " whereString");
Using the code:
all_reviews = db_handle.find().sort('reviewDate', pymongo.ASCENDING)
print all_reviews.count()
print all_reviews[0]
print all_reviews[2000000]
The count prints 2043484, and it prints all_reviews[0].
However when printing all_reviews[2000000], I get the error:
pymongo.errors.OperationFailure: database error: Runner error: Overflow sort stage buffered data usage of 33554495 bytes exceeds internal limit of 33554432 bytes
How do I handle this?
You're running into the 32MB limit on an in-memory sort:
https://docs.mongodb.com/manual/reference/limits/#Sort-Operations
Add an index to the sort field. That allows MongoDB to stream documents to you in sorted order, rather than attempting to load them all into memory on the server and sort them in memory before sending them to the client.
As said by kumar_harsh in the comments section, i would like to add another point.
You can view the current buffer usage using the below command over the admin database:
> use admin
switched to db admin
> db.runCommand( { getParameter : 1, "internalQueryExecMaxBlockingSortBytes" : 1 } )
{ "internalQueryExecMaxBlockingSortBytes" : 33554432, "ok" : 1 }
It has a default value of 32 MB(33554432 bytes).In this case you're running short of buffer data so you can increase buffer limit with your own defined optimal value, example 50 MB as below:
> db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes:50151432})
{ "was" : 33554432, "ok" : 1 }
We can also set this limit permanently by the below parameter in the mongodb config file:
setParameter=internalQueryExecMaxBlockingSortBytes=309715200
Hope this helps !!!
Note:This commands supports only after version 3.0 +
solved with indexing
db_handle.ensure_index([("reviewDate", pymongo.ASCENDING)])
If you want to avoid creating an index (e.g. you just want a quick-and-dirty check to explore the data), you can use aggregation with disk usage:
all_reviews = db_handle.aggregate([{$sort: {'reviewDate': 1}}], {allowDiskUse: true})
(Not sure how to do this in pymongo, though).
JavaScript API syntax for the index:
db_handle.ensureIndex({executedDate: 1})
In my case, it was necessary to fix nessary indexes in code and recreate them:
rake db:mongoid:create_indexes RAILS_ENV=production
As the memory overflow does not occur when there is a needed index of field.
PS Before this I had to disable the errors when creating long indexes:
# mongo
MongoDB shell version: 2.6.12
connecting to: test
> db.getSiblingDB('admin').runCommand( { setParameter: 1, failIndexKeyTooLong: false } )
Also may be needed reIndex:
# mongo
MongoDB shell version: 2.6.12
connecting to: test
> use your_db
switched to db your_db
> db.getCollectionNames().forEach( function(collection){ db[collection].reIndex() } )
I've created (in code) a default collection in MongoDB and am querying it, and have discovered that while the code will return all the data when I run it locally, it won't when I query it on a deployment server. It returns a maximum of 256 records.
Notes:
This is not a capped collection.
Locally, I'm running 3.2.5, the remote MongoDB version is 2.4.12
I am not using the limit parameter. When I use it, I can limit both the local and deployment server, but the deployment server will still never return more than 256 records.
The amount of data being fetched from the server is <500K. Nothing huge.
The code is in Clojure, using Monger, which itself just calls the Java com.mongodb stuff.
I can pull in more than 256 records from the remote server using Robomongo though I'm not sure how it does this, as I cannot connect to the remote from the command line (auth failed using the same credentials, so I'm guessing version incompatibility there).
Any help is appreciated.
UPDATE: Found the thing that triggers the problem: When I sort the output, it reduces the output to 256—but only when I pull from Mongo 2.4! I don't know if this is a MongoDB itself, the MongoDB java class, or Monger, but here is the code that illustrates the issue, as simple as I could make it:
(ns mdbtest.core
(:require [monger.core :as mg]
[monger.query :as mq]))
(defn get-list []
(let [coll (mq/with-collection
(mg/get-db
(mg/connect {:host "old-mongo"}) "mydb") "saves"
(mq/sort (array-map :createdDate -1)))] ;;<<==remove sort
coll))
You need to specify a bigger batch-size, the default is 256 records.
Here's an example from my own code:
=> (count (with-db (q/find {:keywords "lisa"})
(q/sort {:datetime 1}) ))
256
=> (count (with-db (q/find {:keywords "lisa"})
(q/sort {:datetime 1})
(q/batch-size 1000) ))
688
See more info here: http://clojuremongodb.info/articles/querying.html#setting_batch_size
Consider the follow the sample aggregate query to find if a field exists or not:
db.coll.aggregate({"$project":{"exists":{ "$cond" : [ {"$eq": ["$fieldName", undefined] } , "true" , "false" ]}}})
This aggregate works as expected in mongo shell. But using null in the place of undefined didn't work in mongo shell(why?), as below:
db.coll.aggregate({"$project":{"exists":{ "$cond" : [ {"$eq": ["$fieldName", null] } , "true" , "false" ]}}})
I want to use this operation through java driver.
But I dont know how to convert this to java driver as it doesn't have undefined. Using null in the place of undefined didn't work in java too..
Why is it designed this way?
Looks like I miss something important.
I have some records inserted into mongoDB that contains fields with national characters. There are no problem to insert it to DB or find them and all values looks pretty good.
But if I try to find particular one with "find()" or "regex()" they return nothing. For example:
DBObject query = new BasicDBObject();
query.put("position", Pattern.compile(".*forsøg.*"));
--or--
query.put("position","forsøg");
System.out.println(collection.find(query).count()); // prints 0
in log
query={ "position" : { "$regex" : ".*������.*"}}
--or---
query={ "position" : "������"}
Field value for "position" is equal "forsøg" ofc. Pattern.matches(".*forsøg.*", "forsøg") returns true.
If I replace pattern with one containing only ASCII characters (".abc." for example ) all methods work as expected. Collection.findAll() return all saved instances with readable and correct values.
Versions: MongoDB 2.0.6 64bit, mongo-java-driver 2.8.0, Java 7. I tried the same with spring-data-mongodb 1.0.2.RELEASE but removed it.
Looks like I meet a strange bug related with a maven + testng. The same code executed from .war and from testsuit provides totally different result in database.
The difference may be easily found if point your browser to
http://127.0.0.1:28017/baseName/collectionName/
and look at the values after each execution.