I have the following scenario:
MethodA() calls MethodB()
MethodB() calls MethodC()
All methods have to execute some query to DB. So, to do this I create a connection object and pass it along the chain of methods to reuse the connection object.
Assumption here is that connection pooling is not being employed.
Now my question is, should only a single connection be opened and reused and be closed at the starting point (in the above example, the connection will be opened and closed in MethodA) ? or should I create a separate connection for each method?
Reusing the connection seems better, but then I will have to keep the connection open till the control comes back to MethodA().
I have read that reusing the connection is better as they are expensive to create. But then I have also read that its better to close the connection as soon as possible, i.e., once you are done with the query call.
Which approach is better and why?
It sounds like you are only querying the DB and not updating or inserting. If that is the case then you avoid many of the transactional semantics in such a nested procedure call.
If that is true, then simply connect once, do all your querying and close the connection. While usage of a connection pool is somewhat orthogonal to your question - use one if you can. They greatly simplify your code because you can have the pool automatically test the connection before it gives one to you. It will auto-create a new connection if the connection was lost (let's say because the DB was bounced).
Finally, you want to minimize the number of times you create a DB Connection BECAUSE it is expensive. However, this is often non-trivial. Databases themselves only support a maximum number of connections. If there are many clients, then you would need to take this into consideration. If you have the trivial case - one database and your program is the only one making connections, then open the connection and leave it open for the duration of the program. This would require you to validate it, so using a DB Pool, size of 1, avoids that.
Related
I am using ORMLite (Java) with HikariCP and I have a lot of code that's called asynchronously i.e. CompletableFuture.supplyAsync(Supplier). Inside a Supplier I have various calls to different ORMLite Dao objects inside these tasks. I ran into a deadlock scenario and found that increasing the size of the connection pool seemed to have fixed the issue.
After some research, I found that this is a well-documented problem. Having looked at the information provided here:
"Each thread is waiting for a second database connection, and none would become available since all threads are blocked."
This caught my eye. When looking at the ORMLite source code, I found that for each Dao method called, a connection is acquired. In my case, I am calling multiple Dao methods in each asynchronous task I create via CompletableFuture which brings me to my question.
My understanding is that I should only acquire a single connection in each thread so is there a way to do this with ORMLite? Bearing in mind I have calls to different Dao objects and having done some further research, Dao#callBatchTasks(Callable) caught my eye but I am unsure whether I can have calls to different Dao objects inside the Callable. Or is there a flaw in my logic or something fundamental I am missing in my write-up?
I found that for each Dao method called, a connection is acquired
If ORMLite needs to go to the database, it needs to get a connection, right? Is there something that it is not doing right?
My understanding is that I should only acquire a single connection in each thread so is there a way to do this with ORMLite?
I don't think that there is anything that says that a thread can't open multiple database connections. As long as they are all closed when you are done with te operation, then I think this is fine. If you are querying different DAOs that all live on the same database then you can actually reuse the connection each time as long as you are done with it. I.e. if you are paging across the results using an open connection then in the middle query another table, then this might screw up the cursor. However, if you do a dao1.queryAll() and then a dao2.queryAll(), this can use the same connection since the first query will have finished.
But if you are using a connection pool, the pool should do the connection management for you with help from ORMLite's ConnectionSource. The pool is supposed to create a new one for you if needed and otherwise reuse old connections. It's a bit strange to have a limit on your database connections.
Dao#callBatchTasks(Callable) caught my eye but I am unsure whether I can have calls to different Dao objects inside the Callable.
You can call different dao objects as long as they are all on the same database. You actually could call a dao with connections to a different database, but you'd have no guarantee around transactions between the databases.
For example if you did something like the following.
You call dao1.callBatchTasks(callable)
In your Callable, you call dao2.delete(entity1) which works.
Still in the Callable, you call dao2.delete(entity2) which throws.
The dao1 transaction will be rolled back, but the entity1 would still be deleted.
The scenario you quoted in the statement
Each thread is waiting for a second database connection, and none
would become available since all threads are blocked.
would result in a deadlock only when a thread is holding a lock to connection A AND trying to acquire Connection B, which might be held by another thread which might be trying to get hold of Connection A. As long as you are not "holding" one resource while trying to "grab" the other one, you should be good.
Problem:
Program uses com.mchange.v2.c3p0.ComboPooledDataSource to connect to Sybase server
Program executes 2 methods, runSQL1() and runSQL2(), in sequence
runSQL1() executes SQL which creates a #temptable
SELECT * INTO #myTemp FROM TABLE1 WHERE X=2
runSQL2() executes SQL which reads from this #temptable
SELECT * FROM #myTemp WHERE Y=3
PROBLEM: runSQL2() gets handed a different DB connection from the pool than the one handed to runSQL1().
However, Sybase #temptables are connection-specific, therefore runSQL2() fails when it can't find the table.
The most obvious solution I can think of (aside from degenerate one of making pool size 1, at which point we don't even need a pool), is to somehow remember which specific connection from the pool was used by runSQL1(), and have runSQL2() request the same connection.
Is there a way to do this in com.mchange.v2.c3p0.ComboPooledDataSource?
If possible, I'd like an answer which is concurrency-safe (in other words, if connection used in runSQL1() is being used by another thread, runSQL2()'s call to get connection will wait until that connection is released by another thread).
However, if that's impossible, I'm OK with the answer which assumes that DB connections (the ones I care about) are all happening in one single thread, and therefore any connection requested by runSQL2() will be 100% available if it was available to runSQL1().
I'm also welcoming of any solutions that address the problem some other way, as long as they don't involve "stop using #temptables" as part of the solution.
Easiest and most obvious way to do that is just to request connection from the pool and then run runSQL1() and runSQL2() with that connection. Usage pattern being suggested in the question goes against general design principles of connection pool managers, as it will effectively promote them to some kind of transaction manager.
There are Java frameworks that might aid in the above. For example in Spring #Transaction or TransactionTemplate can be used to demarcate transaction boundaries and it will guarantee that single connection is used by single thread (or more precisely, according to transaction propagation annotations). Spring can use many transaction managers, but probably simplest would be to use DataSourceTransactionManager and it can also be configured to use c3p0 as DataSource.
I have a static java connection to a databases.
1st Thread
with my static connection I am creating a statement and call to executeQuery() method. at this point I'm waiting because the query results is very big and I have to wait more than 15 minutes.
2nd Thread
with the same static connection I am trying to make a call to createStatement() method. And here the 2nd thread is blocked.
There are many ways to solve this issue, but I don't find anywhere why this aproach should not working. Any ideas?
In a common way, the single Connection to the database is implemented through Sockets mechanism. Better to say, that your single connection is some kind of pipe, which is using for concrete query which is executing one per time.
It is hard to imagine such optimal and high-perfomance code implementation, where though one socket multiple queries and their results are sent in parallel. Such implementation could cause the confusion with data-packets and at least their lost.
As karmanaut mentioned above in comments, you need to implement DB pool, i.e. pool of connections, which will be used for each statement concurrently.
I am developing a simple database project. It has some functions that will be performed on database. I need advice about opening and closing database. Which way is the better below?
When program starts, open database and close when program finishes.
When program starts, when a function is called, open database, and at the end of this function close database.
I am confused for which one is better. In the first case database will be open during the program. In the second case database opening and closing operation will be performed at each function call and also for each function. This will reduce efficiency and also makes code longer.
So which one is the better?
Better is subjective. If you have only a few clients connecting to the database, for short periods of time then perhaps leaving a connection open is the better way to go. However, if you have hundreds of clients needing connections, there are limited number of simultaneous connections that can be supported, keeping the connection open for long periods would be detrimental.
There is an overhead involved in opening and closing connections to the database but you have to balance that against whether the language actually pools the connections for you so it only appears that the connection is closed but if needed will be reused and if not really closed after some period of time, and many other possible criteria.
Better is what meets your needs in the simplest to maintain manner.
Make a separate function like getConnection() which will create and return a Connection object. Put all your code that will create the connection there.
now if in any function you need a database connection call getConnection()
public void myfunction()
{
javax.sql.Connection con = MyConnectionClass.getConnection();
con.open();
// do your business logic here
con.close();
}
There is very little cost to leaving a connection open longer than necessary, so that would favor your idea to open when the program starts and close upon exiting the program.
What is a Connection Object in JDBC ? How is this Connection maintained(I mean is it a Network connection) ? Are they TCP/IP Connections ? Why is it a costly operation to create a Connection every time ? Why do these connections become stale after sometime and I need to refresh the Pool ? Why can't I use one connection to execute multiple queries ?
These connections are TCP/IP connections. To not have to overhead of creating every time a new connection there are connection pools that expand and shrink dynamically. You can use one connection for multiple queries. I think you mean that you release it to the pool. If you do that you might get back the same connection from the pool. In this case it just doesn't matter if you do one or multiple queries
The cost of a connection is to connect which takes some time. ANd the database prepares some stuff like sessions, etc for every connection. That would have to be done every time. Connections become stale through multiple reasons. The most prominent is a firewall in between. Connection problems could lead to connection resetting or there could be simple timeouts
To add to the other answers:
Yes, you can reuse the same connection for multiple queries. This is even advisable, as creating a new connection is quite expensive.
You can even execute multiple queries concurrently. You just have to use a new java.sql.Statement/PreparedStatement instance for every query. Statements are what JDBC uses to keep track of ongoing queries, so each parallel query needs its own Statement. You can and should reuse Statements for consecutive queries, though.
The answers to your questions is that they are implementation defined. A JDBC connection is an interface that exposes methods. What happens behind the scenes can be anything that delivers the interface. For example, consider the Oracle internal JDBC driver, used for supporting java stored procedures. Simultaneous queries are not only possible on that, they are more or less inevitable, since each request for a new connection returns the one and only connection object. I don't know for sure whether it uses TCP/IP internally but I doubt it.
So you should not assume implementation details, without being clear about precisely which JDBC implementation you are using.
since I cannot comment yet, wil post answer just to comment on Vinegar's answer, situation with setAutoCommit() returning to default state upon returning connection to pool is not mandatory behaviour and should not be taken for granted, also as closing of statements and resultsets; you can read that it should be closed, but if you do not close them, they will be automatically closed with closing of connection. Don't take it for granted, since it will take up on your resources on some versions of jdbc drivers.
We had serious problem on DB2 database on AS400, guys needing transactional isolation were calling connection.setAutoCommit(false) and after finishing job they returned such connection to pool (JNDI) without connection.setAutoCommit(old_state), so when another thread got this connection from pool, inserts and updates have not commited, and nobody could figure out why for a long time...