When using JDBC in Java, the generally accepted method of querying a database is to acquire a connection, create a statement from that connection, and then execute a query from that statement.
// load driver
Connection con = DriverManager.getConnection(..);
Statement stmt = con.createStatement();
ResultSet result = stmt.executeQuery("SELECT..");
// ...
However, I am unsure of how to treat a second query to the same database.
Can another query be executed safely on the same Statement object, or must another statement be created from the Connection object in order to execute another query?
If the same Statement object can be used for multiple queries, what is the purpose of the Statement class (since it would then make more sense for a Connection.executeQuery() method to exist)?
Yes you can reuse the Statement object, but the ResultSet objects returned by the executeQuery closes already opened resultsets.
See the javadoc for the explanation
By default, only one ResultSet object per Statement object can be open
at the same time. Therefore, if the reading of one ResultSet object is
interleaved with the reading of another, each must have been generated
by different Statement objects. All execution methods in the Statement
interface implicitly close a statment's current ResultSet object if an
open one exists.
So the following occurs:
// load driver
Connection con = DriverManager.getConnection(..);
Statement stmt = con.createStatement();
ResultSet result = stmt.executeQuery("select ..");
// do something with result ... or not
ResultSet result2 = stmt.executeQuery("select ...");
// result is now closed, you cannot read from it anymore
// do something with result2
stmt.close(); // will close the resultset bound to it
For example you can find an open source implementation of Statement in the jTDS project.
In the Statement.executeQuery() method you can see a call to initialize() that closes all the resultsets already opened
protected void initialize() throws SQLException {
updateCount = -1;
resultQueue.clear();
genKeyResultSet = null;
tds.clearResponseQueue();
// FIXME Should old exceptions found now be thrown instead of lost?
messages.exceptions = null;
messages.clearWarnings();
closeAllResultSets();
}
Programmatically, you can reuse the same connection and the same statement for more than one query and close the statement and the connection at the end.
However, this is not a good practice. Application performance is very sensitive to the way database is accessed. Ideally, each connection should be open for the least amount of time possible. Then, the connections must be pooled. Going by that, you would enclose each query in a block of {open connection, create a prepared statement, run query, close statement, close connection}. This is also the way most SQL Templates are implemented. If concurrency permits, you can fire several such queries at the same time using a thread pool.
I have one thing to add should you use Connection and Statement in a threaded environment.
My experience shows that stmt.executeQuery(..) is save to use in a parallel environment but with the consequence that each query is serialized and thus processed sequencially, not yielding any speed-ups.
So it es better to use a new Connection (not Statement) for every thread.
For a standard sequential environment my experience has shown that reusing Statements is no problem at all and ResultSets need not be closed manually.
I wouldn't worry about creating new statements. However opening up a database connection may be resource intensive and opening and closing connections does impact performance.
Leaving up connections in some self management way usually is pretty bad.
You should consider using connection pooling. You usually issue a close commando however you are only giving that connection back to the pool. When you request a new connection then it will reuse the connection you gave back earlier.
You may want to have different statements for one connection. Statement is an implementation and an interface. Depending on what you need you sometimes want a use a CallableStatment. Some some logic may be reused when required.
Usually, it's one statement for one query. It might not be necessary to do that but when writing real application, you don't want to repeat those same steps again and again. That's against the DRY principal, plus it also will get more complicated as the application grows.
It's good to write objects that will handle that kind of low level (repetitive) stuffs, and provide different methods to access db by providing the queries.
Well that's why we have the concept of classes in object oriented programming . A class defines constituent members which enable its instances to have state and behavior. Here statement deals with everything related to an sql statement. There are so many more function that one might perform like batch queries etc.
Related
I am using streaming in my query to MySQL DB. Works fine, until I issue another query during the streaming. That is fully OK and explained in the java.sql.SQLException:
Streaming result set com.mysql.jdbc.RowDataDynamic#16559dec is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
Since I really would like to make my second query in the middle of streaming, apparently I just need to use another DB connection for that.
So how can I force using another connection in the same thread?
I am using Spring Data with Hibernate
Please do not suggest fetching all at once or paging, that's not the point of the question.
Edits:
streaming in my case is having a select from long table (millions of records), where next row is only transferred when it is requested. It is not streaming the file content. This is the article about that: http://knes1.github.io/blog/2015/2015-10-19-streaming-mysql-results-using-java8-streams-and-spring-data.html
using another thread is a solution, but the question is about having in the same thread 2 connections at the same time
Program flow:
run the query using stream (using connection 1)
for every row from the stream
do something on db (using connection 2)
streaming finishes, connection 1 closes
So 1 thread with { 1. open conn, 2 - 8. asyncroneously stream, 9. close conn }, { 3. open or use conn, 4. query, 9. close or skip }??
Try using a connection pool instead if the streaming is short enough.
Otherwise is this also a case where large streams are better stored as file name in the database (using a UUID for instance to generate file names). Then the streaming can be done outside of the database, and you could throttle the streaming to hamper a self-made DenialOfService.
After question has been reedited.
So the scenario is:
void f() {
open conn
do a java 8 stream
g()
close conn
}
void g() {
open conn
...
close conn
}
This is possible, with several ways of handling it: with a single global connection, with a connection pool, transactional or not cq. autocommit.
For queries I guess the most important is to close things. Try-with resources is ideal to not leak resources.
try (Connection conn = ...) {
...
try (PreparedStatement stm = conn.prepareStatement(sql)) {
...
try (ResultSet rs = stm.executeQuery()) {
... the stream
}
}
}
The above closes also the ResultSet, which might have gotten you in troubles.
It is also quite thinkable you pass the stream on, it has access the result set.
As Stream is AutoCloseable too, you might need to tweak the code there. Or use a CachedRowSet instead of a ResultSet.
Sorry for this indeterminate answer.
I'm not sure I understand fully what you want to do. I take it that you want to stream your data, transform somehow each item and then persist the result. If that's the case, I believe that having a separate 'persist' method with annotation #Transactional(propagation = Propagation.REQUIRES_NEW) should do the trick.
If you want more than one processing your stream (say, because you do some REST calls as part of it or, for some other reason you think that it might take long), you might want to consider pushing the streamed elements to a blocking queue and have multiple threads reading from there and doing whatever you need doing with each item.
I am trying to execute multiple queries in a single function using single connection object. I would like to know what is the best practice to close the database connection in a scenario like this. Currently , I close the connection once all the db calls are completed. I am wondering whether I need to close the connection and open a new connection for every db call. Which is better?
You should keep the Connection open as long as possible. Creating a database connection is a (relatively) expensive operation, so you don't want to do it more often that you need to.
To manage the lifetime, you should use the try-with-resources statement assuming you are on at least Java 7:
try (Connection connection = myDataSource.getConnection()) {
// Do your queries here
}
Is it safe or recommended to execute independent statements while you have a result open? Does it matter if they are attached to a different connection or the same one as the result set? I'm particularly concerned with how the result holds locks if any, which could cause deadlock.
Ex.
while(resultSet.next()) {
Execute separate statements in here ( same or different connection )
}
Also is a result set backed by an underlying cursor or something else?
Ty
There are several questions here.
First, generally "yes" it is possible and common to run other SQL statements while iterating over a ResultSet. And yes, ResultSets are backed by a cursor.
It is also possible to create a deadlock doing this, so you just need to be aware of that. If the SQL being executed inside of your loop is not modifying rows in the same table as the ResultSet, then you should ensure that the ResultSet is created with a concurrency mode of CONCUR_READ_ONLY, and in general try to use TYPE_FORWARD_ONLY.
For example:
Statement stmt = con.createStatement(ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
If you use CONCUR_READ_ONLY and TYPE_FORWARD_ONLY in general locks that block writes should not be generated. Using the same Connection object is also recommended because then both the cursor and the SQL that is modifying other objects are within the same transaction and are therefore less likely to cause a deadlock.
Its safe according to my point of view..
The ResultSet is usually linked to the Statement which is usually linked to the Connection.
A ResultSet object is automatically closed when the Statement object
that generated it is closed, re-executed, or used to retrieve the next
result from a sequence of multiple results.
Close ResultSet when finished
Close ResultSet object as soon as you finish
working with ResultSet object even
though Statement object closes the
ResultSet object implicitly when it
closes, closing ResultSet explicitly
gives chance to garbage collector to
recollect memory as early as possible
because ResultSet object may occupy
lot of memory depending on query.
ResultSet.close();
It is perfectly safe. You have two basic choices:
Use a different Connection object.
Use the same Connection object.
The primary difference is in how transactions are handled.
Different Connections are always in different transactions
Different statements in the same Connection can be made to be in the same transaction if you set auto commit mode to false.
Two statements that are not in the same transaction are guaranteed by the SQL server not to interfere with each other (usually this means that the server will hold onto copies of any modified data if an older transaction might still need it). It is also down to the server how it performs locking (if at all) but it must guarantee that no deadlock can occur. Usually it does this by a process called serialization, which involves storing transactions in a log until they can be guaranteed to execute without deadlock.
We have a java application which we use more then one statement variable. The problem why need more then one statement is that at time while runnning a loop for one result inside the loop we need some other query operation to be done. Most of the places the single stmt is used many times and finally we close. What we would like to confirm now is that we are not closing the resultset variables and we notice the usage of memory fluctuates.So what is the best mechanism to close the resultset immediately after we got the results or towards the end just before stmt is being closed?
According to the JDBC Specification and the Statement.close() API doc it should be sufficient:
Note:When a Statement object is closed, its current ResultSet object, if one exists, is also closed.
Based on that you should be able to assume that you only need to close a statement. However as a Statement can have a longer lifetime than the use of a single ResultSet obtained from it, it is a good idea in general to close the ResultSet as soon as possible.
A Statement can have a longer lifetime because for example you use it again to execute another query, or in case of a PreparedStatement to execute the query again with different parameters. In the case you execute another query, the previously obtained ResultSet will be closed.
Close it when you are finished with it, same as any other resource. Mo point in holding on to it for longer, especially if you are down to measuring memory, which implies a memory concern.
What happens with the statement when result set is closed?
Statement stmt = null;
ResultSet rs = null;
try {
stmt = con.createStatement();
rs = stmt.executeQuery(query.toString());
...
}
// lots of code
rs.close()
Note: A ResultSet object is automatically closed by the Statement
object that generated it when that Statement object is closed,
re-executed, or is used to retrieve the next result from a sequence of
multiple results.
But what happens when ResultSet is closed first?
For what matter, what should happen first?
You should close the objects in the reverse order of creating them.
So first, the ResultSet, then the Statement and in the end, Connection.
If you don't close the Statement it stays open and you can execute another query with it (if it's a PreparedStatement or CallableStatement you are able to change query parameters).
Also note, that what you quoted is a JDBC specification, and the implementation of it is up to a JDBC driver provider. Usually, you should not trust them, and manually close those objects.
The other thing - I think more important - if you are using connection pools (like on JBoss), closing connection just releases it back to the pool, and the underlying objects are not released. And because of this, its recommended to always manually release all the objects you created.
The Statement actually can be reused. Though if you're repeating the same query, you should use a PreparedStatement instead as the query will be compiled once.
They of course should be closed when you're done with them, but closing the ResultSet doesn't automatically close the Statement for good reason.
Nothing happens to the Statement -- it stays open. You can continue to use the Statement however you'd like, but be sure to close it when you're finished.