I have problem to close a connection that is always created after accessing the MongoDB cursor.
I am gathering the information about the connections from the console of the running mongod instance. I am not using any mongodb replication nor sharding yet.
For example a sample code causing an opened connection (assume 'myQuery' is just some query, coll is mapped using setInternalClass to the class MyObject):
DBCursor find = coll.find(myQuery);
List<MyObject> myObjects = new ArrayList<MyObject>();
while(find.hasNext()) { // this line opens the connection
MyObject next = (MyObject) find.next();
myObjects.add(next);
}
find.close(); // this line will not close the connection
You are calling close on your find object, which is a DBCursor. As mentioned in the API Documentation this will close only the cursor, not the underlying connection. To close the connections you should call close on the Mongo/MongoClient object, which either inherits or has a close function that will as per the API close all connections to the MongoDB server instance.
Related
I'm currently creating an API server that reads and writes. Using MongoDB
The library uses Mongoose.
I wonder if db.close() must be used when reading and writing.
datamodel.js:
var db = mongoose.connect('mongodb://localhost/testdb', {useNewUrlParser: true,useUnifiedTopology:true});
mongoose.Promise = global.Promise;
.....
Boards = mongoose.model("boards", BoardSchema);
exports.Boards = Boards;
routes/getList.js:
let result = await Boards.find().sort({"date": -1});
Should I close the DB declared above with db.close() when reading or writing?
(Very generic answer, but should help you get started with what to research)
Closing MongoDB connection depends on how is the connection established in the first place.
Are you initialising the connection on server startup: If yes, you should not close the connection. (But initialising the connection on server startup is bad idea because, if connection is lost to server (like database server restart), then you would also have to restart the application or set reconnectTries)
Are you using connection pool: If you are using connection pool, then closing and opening of connections is taken care by Mongoose itself. All you have to do is, release the connection after use, so that, it's available for other requests.
Are you creating connection per request: If yes, then you should close the connection before returning the response or you would quickly run out of available connections at database server.
you can call mongoose.disconnect() to close the connection
We are currently dealing with the function, that has to work partially with the database, and partially with a service, whice operations are time-consuming. So, generally speaking, here is a transactional method, that has a code like this:
Connection conn = null;
try {
conn = getConnection(); // This I get from the connection pool
Employee emp = queryDatabase(id);
// Point A - ??
Response resp = makeLongTimeServiceCall(emp);
// Point B - ??
anotherQueryDatabase(resp);
} catch (Exception e) {
throw e; // And this also rolls back the transaction
} finally {
// If conn is not closed, close it, so there would be no leaks
}
So the big question is - should I close the connection at point A and then get it again from the connection pool at point B, so that other servlets could use that connection while I interact with the service? My guess is that I should, but will this hurt my transaction?
In many circumstances: yes, closing and reopening the connection sounds good. However you need to understand the implication of doing them in two separate transactions (by closing and re-opening the connection you are inherently doing them in separate transaction).
If another user invokes an operation that changes the state of your data at Point B will the end result still be correct?
If you don't have a transaction you can close the connection and ask for a new connection when needed.
Remember that if you are using a connection pool closing a connection will not really close the connection, but only flag it as reusable by other threads.
Database connections shouldn't be left open. Open the connections just when you need to execute a query and close it as early as possible. So my solution is, YES.
if your application doesn't close connection properly may lead to some issues like the
connection pool maxing out.
Applications Not Properly Closing Connections:
When writing an application to use the WebSphere Application Server(WAS) datasource, the best way is fallowing get/use/close pattern.
1.get
-This is when the application makes a request to get a connection from the pool.
The application will first lookup the datasource name and then do a getConnection() call.
2.use
-This is when the application executes a query and waits for a response
3.close
-This is the final stage when the application has received the response from the database and is now done using that connection.
-The application should call close() in a finally block to close out the connection and return it to the free pool.
If your application does not follow this pattern, you may see connections staying open for long periods of time. This is especially seen when the application does not close the connection properly. After the response is received from the database, the application should be calling close in a finally block. If the application does not explicitly close the connection, that connection will stay open forever in WAS until that server is restarted. This is known as a connection leak. To resolve this type of issue, you will have to modify your application to close every connection
for further information: https://www.ibm.com/support/pages/common-reasons-why-connections-stay-open-long-period-time-websphere-application-server
I'm playing around a bit with the MongoDB driver for Java. So I just created a simple application to connect to a MongoDB server and select a database.
So I created an instance of MongoClient and selected a 'DB':
try
{
MongoClient client = new MongoClient("localhost", 27017);
DB database = client.getDB("example");
}catch(Exception e){
e.printStackTrace();
}
Because of the fact that there is no running instance of mongod on my machine, I expected that client would throw an Exception. Unfortunately that isn't the case.
Even when selecting the database nothing happens. It just behaves like if there was a running mongod instance.
I looked into the documentation about the Java driver but couldn't find anything about it. Same with Google.
Is there anything I missed?
I'm using the latest MongoDB driver (version 2.12.2) from the official website.
It is expected behaviour. The driver does not attempt to connect to the database until it is needed. If you try the mongo shell, you do not get the error if the database does not exist.
When you try to insert a document into a non-existent collection it is created for you automatically and that is when the connection is lazily established. It is first when you actually perform some db operation (find(), insert() etc.) that the connection is checked for.
Try doing an insert to a collection. Connections are lazily initialized and validated.
I have a main controller servlet in which i instantiate a datasource. The servlet opens and closes the connections. Mainly, the servlet instantiates a command from the application using the "factory pattern". here is some code to explain:
public void init() throws ServletException {
super.init();
try {
datasource =(DataSource) getServletContext().getAttribute("DBCPool");
}
catch (Exception e) {
}
}
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
//some code...
Connection connection = null;
if(cmd.mightNeedLazyLoadingAConnection)
{
connection = null;
}
else
connection = getConnection();//where getConnection is a method: datasource.getconnection();
//now a command (a java class) is instantied, to which the "null" CONNECTION obj is passed as parameter
cmdFactory.getInstance().getCommand(Cmd).execute(tsk,connection);
//some code
//Then wherever there is catch exception i close() the connection
// and it is always closed in finally
finally {
if(connection!=null)
connection.close()
}
}
Now , this has a problem for the first case, ie connection=null, as it does never close the connection in the "finally" part (explained why in Update below).
"connection=null" is for cases where commands might not need to open a db connection because the data it is seeking for are cached in an identity map.
I tried to pass the "Connection" obj as a "null" parameter in the .execute(tsk,connection); and then in the corresponding java class to open a connection if needed
--> it did open the connection inside the command, however when process goes back to servlet : "Connection" is null as thus not closed.
What can i do to make the "Connection" obj's value get updated so that when back in servlet it is not "Null" anymore and i'd be able to close it?
I generally prefer to use a controller servlet that opens/close db connections, so what would be the best way to deal this kind of scenario where you have to do some sort of "lazy loading" a db connection from the pool and at the same time keep the opens/close of db connection assigned to the servlet?
Update (to explain further):
say i have a command : X.java
this command might/might not need a db connection (depends if the data searched for are in the identity map or not)
The system that i would like to have is:
(1)"client request"
(2)---> "Servlet": command.execute(connection)//where connection = null for now
(3) ---> "Command X": Do i need to go to database or record is in identity map?
(3.a) Case where it is needed to go to the database:
(3.a.1)connection = datasource.getconnection
(3.a.2) go get the data
(4)--->back to servlet: close "connection" in "Servlet"
Right now it is working until (3.a.2), but once back in (4) it appears that connection is still "null" and thus the code:
finally {
if(connection!=null)
connection.close()
}
Does not work (doesn't close the connection) and thus the db pool get drained like that.
How could connection - which starts as "null" and changes inside command "X"- get "globaly" updated to the its new value, and not only "updated" inside the scope of command "X"?
SOLUTION(S)
In case you are encountering the same scenario, you can chose of these 2 solutions:
You can Either use LazyConnectionDataSourceProxy, as mentionned by #Ryan Stewart for a "clean abstraction" and more professional solution
Or if you'd like use my solution described below (Basically i implemented a class similar to "LazyConnectionDataSourceProxy" but it is not as clean, it has less abstraction of details than "LazyConnectionDataSourceProxy")
My personal solution, Details:
I created a "Helper" class, which constructor takes the "datasource" as parameter
This helper class has methods to: "Lazy get" connection from pool,"close" connection
This class is instantiated in the servlet, and it gets a connection from the pool Only if needed throughout the application.
This is the code i added/modified in the servlet:
Connection connection = null;
if(cmd.mightNeedLazyLoadingAConnection)
{
helper hp = new helper(datasource);
cmdFactory.getInstance().getCommand(Cmd).execute(tsk,hp);
}
else
{
connection = getConnection();
cmdFactory.getInstance().getCommand(Cmd).execute(tsk,connection);
}
Then say in a command "X" , a db connection is needed i do:
Connection connection = hp.LazyGet();//Now got a connection from the pool
And this way, when proccess flow is back to the servlet level, i can :
Close
rollback
commit
etc..
All on this hp object of the helper class.
What benefits do i get from this:
I limit all database open / close / commit / rollback in one place, ie the Servlet, which is responsible of executing commands.
Having 3 cases: never needs db / always needs db / might need db thus now i decreased calls to the database by 1/3 , which is quite a lot knowing that database call grows exponentially with new features and new users registrations.
It might not be the Cleanest workaround, but between this way and having an additional "unnecessary" 1/3 database calls, it surely is better. Or just use LazyConnectionDataSourceProxy if you want a Tested, abstract and clean method.
Use a LazyConnectionDataSourceProxy. Then just get a "connection" every time, but a real connection is opened only when you actually do something that requires one. Thus you obey the "create/destroy" wisdom which Hiro2k pointed out because the connection's lifecycle is completely managed by your servlet.
In your specific case the only way to do it would be to return the connection. Java doesn't have any pass by reference semantics that could help you, unlike C where you could pass in the reference to the connection and then set it within the method.
I don't recommend your method return the connection, instead remember this simple rule and everything will work as you expect:
The object that creates it, is responsible for destroying it.
If what you want to do is not instantiate a connection for commands that don't require one, then add a method to your command interface that simply returns if needs one.
Command command = cmdFactory.getInstance().getCommand(Cmd);
if(command.requiresConnections){
connection = getConnection();
}
command.execute(tsk,connection);
I have create a getDBConnection method in my Java application. This returns a connection object, and hence I haven't closed this connection in this method itself.
Now, I am invoking this method from various methods in my application at regular intervals, and closing them inside a try - finally block. I thought this should free up the connection after use. However, I am seeing a large number of connections opened (about 50) in the MySQL Administrator's Server Connections tab.
//Defining a method to retrieve a database connection
// PropDemo is a properties class that retrieves Database related values from a file
public Connection getDBConnection() {
//Instantiating the Properties object
PropDemo prop = new PropDemo();
Connection con = null;
// Retrieving values from the parameters.properties file
String JdbcDriver = prop.getMessage("JdbcDriver");
String JdbcUrlPrefix = prop.getMessage("JdbcUrlPrefix");
String DBIP = prop.getMessage("DBIP");
String DBName = prop.getMessage("DBName");
String DBUser = prop.getMessage("DBUser");
String DBPassword = prop.getMessage("DBPassword");
try {
// Loading and instantiating the JDBC MySQL connector driver class
Class.forName(JdbcDriver).newInstance();
con = DriverManager.getConnection(JdbcUrlPrefix + DBIP + "/" + DBName, DBUser, DBPassword);
if (con.isClosed())
Logger.log("Connection cannot be established", "vm");
} catch (Exception e) {
Logger.log("Exception: " + e, "vm");
Logger.log(Logger.stack2string(e), "vm");
}
return con;
}
I am also closing the associated ResultSet and Statement Objects. What could be missing here?
I am planning to replace all the Statements with PreparedStatements for efficiency and security reasons. Will that help significantly? What else can be done?
EDIT:
This is just a core java application that is repeatedly quering for changes in some fields in a MySQL database through MySQL-JDBC connector. I am not using any framework like Spring or Hibernate.
Your code looks sane.
That's how you're creating a new connection.
Probably the error is where you close it.
You should close it in a finally block.
Some additional questions.
1) Are you sure those 50 conections come from this program ? Maybe there are some others comming from your same office. To confirm this you would need to stop the program, and look again in your connection monitor.
2) Does your application uses many connection simultaneously? Probably its a peak when you're using 50 at the same time.
If you can post the code where you close the connection. Chances are the problem is there.
Additionally I would suggest you to use a connection pool. You can build one your self or you can see the results from this page:
How many JDBC connections in Java?
Are you closing the connection object when you application closes as well?
Are you using your JDBC connection within a J2EE application server or with Hibernate?
Both of these tend to start out with a fairly high connection pool to begin with, so you would see a large number.
Check out the details on connection pooling.
You could take a Singleton approach to the problem and only create a new Connection object if the current one is null:
If (connectionObject != null){
return connectionObject;
}else {
//create new connection object
}
This will make sure that you only have one non-null connection at any time.