I am getting a No operations allowed after statement closed. - very obvious and also self explanatory
as to what is going on with my code. In any case I am wondering how I can do this in a cleaner way
public class BaseClass {
Connection con;
Statement st;
protected void establishDBConnection() throws ClassNotFoundException,
SQLException {
Class.forName("com.mysql.jdbc.Driver");
String cString = "....";
con = DriverManager.getConnection(cString, user, password);
st = con.createStatement();
}
public BaseClass() {
try {
createDBConnection();
} catch (Exception e) {
e.printStackTrace();
}
}
}
public ClassB extends BaseClass {
public ClassB() {
super();
}
public void doSomething() {
try {
String q = "select * from my_table";
String moreQuery = "update my_table ...."
String anotherQuery = "do something fancy..."
rs = st.executeQuery(q);
while (rs.next()) {
st.executeUpdate(moreQuery);
st.executeUpdate(anotherQuery);
}
} catch (Exception e) {
e.printStackTrace();
System.out.println("Error in getAllAssociatesOfMra: " + e);
}
}
}
Currently my code is throwing a com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after statement closed.
The exception is obvious as to what is going on but I was wondering how I can go about handling the close in the BaseClass.
Update
I am aware that there a couple of related questions like mine. The only problem with those questions is that everything is done in the main class. Consider this to be kind of design/abstraction question
Your design is not good. You should be getting the connection, preferably from a connection pool, creating the statements in the beginning of your doSomething() method (for example calling the superclass method), and then closing the Statements and ResultSets when you've done "something".
Before you can make a good design you have to understand what you're trying to accomplish. So I want to establish some goals for this, then look at how the design meets those goals. In order to get the goals for this, let's go over how database connections work.
A database is a separate process, it can be on the same machine or on a different machine where it's accessed over the network. Network connections can go stale due to transient conditions, database downtime, etc. Even if the database is on the same machine and the network is not an issue, it's still bad form to have your application dependent on the state of a separate process with no way to recover, it means that if the database goes down the application can't recover by itself, you have to restart it.
Some properties of connections:
They take a while to get initialized, long enough you wouldn't want to create a new one for every user request. (This will not be nearly as big an issue if the database is on the same machine.)
There is a limited number of them, you don't want one user request to take more than necessary because that will limit the number of other users who can connect concurrently.
There's a commit method on your database connection object that lets you group your operations into transactions, so that a) your queries have a consistent view of the data, and b) the operations get applied in an all-or-nothing manner (if an operation fails you don't have half-done junk cluttering up the database that you have to undo). Currently your connections are in autocommit mode, which means each operation is committed separately (and you can't group operations together).
They're synchronized so only one thread at a time can use them. That way multiple users can't corrupt each others' work, but if you have only one connection your application won't scale since every user is waiting in line for the connection.
From all this we can derive some goals for the design. One is that we want to be able to reinitialize database connections that can go bad, we want to have multiple ones available so everybody's not waiting on the same one, and we want to associate a connection with a user request so that we can group the operations for a given user into transactions.
It's hard to tell what your posted code does exactly because it depends on the scope of the objects, which the question leaves unspecified. If these objects are created per-user-request then you will get a new connection every time, which solves the staleness issue, but may be slow. Having a different connection for every table can make the slowness worse, limits the application's concurrency unnecessarily, and also doesn't allow for transactions that group operations on different tables, because you don't have a common connection object to call commit on. How connections get closed is not apparent; it would be wasteful for them to get abandoned to timeout.
A commonly used alternative approach would be to pool the database connections (where the pool can test database connections and replace stale ones) and hand them out to user requests, returning them to the pool when the request is done with them (the pool can wrap a connection in an object where calling close on it returns it to the pool). In the meantime the user thread can execute operations on the connection, create its own statements and close them on the way out, and commit the results.
Spring has a well-thought out way of handling this situation which follows the approach described above (except with a lot more functionality). Some people dislike frameworks for overcomplicating things, but I recommend at least looking at Spring examples. That way you are aware of an alternative viable approach to organizing your code and you can improve your own design.
If I understand your question and objective, you will need to create multiple Statement objects in doSomething(), and you need to clean up your Statements and ResultSet in a finally block with something like -
Statement st = con.createStatement();
String q = "select * from my_table";
String moreQuery = "update my_table ....";
String anotherQuery = "do something fancy...";
ResultSet rs = st.executeQuery(q);
try {
while (rs.next()) {
Statement stmt = null;
try {
stmt = con.createStatement();
stmt.executeUpdate(moreQuery);
} finally {
if (stmt != null) {
stmt.close();
}
}
try {
stmt = con.createStatement();
stmt.executeUpdate(anotherQuery);
} finally {
if (stmt != null) {
stmt.close();
}
}
}
} finally {
if (rs != null) {
rs.close();
}
if (st != null) {
st.close();
}
}
I suggest few things:
Use connection pool design
To prevent statement close, you can use finally block to close them
Since you have a query after another query, use transaction (commit/rollback) to prevent things "half done"
Related
Using Spring Boot with Spanner in the Google Cloud Env. we are now struggling with performance issues.
To demonstrate that I set up a small demo case baselining our different approaches how to retrieve data from spanner.
The first approach
uses "native" drivers from Google to instantiate a dbClient and retrieves data like so.
#Repository
public class SpannerNativeDAO implements CustomerDAO {
private final DatabaseClient dbClient;
private final String SQL = "select * from customer where customer_id = ";
public SpannerNativeDAO(
#Value("${spring.cloud.gcp.spanner.instanceId}") String instanceId,
#Value("${spring.cloud.gcp.spanner.database}") String dbId,
#Value("${spring.cloud.gcp.spanner.project-id}") String projectId,
#Value("${google.application.credentials}") String pathToCredentials)
throws IOException {
try (FileInputStream google_application_credentials = new FileInputStream(pathToCredentials)) {
final SpannerOptions spannerOptions =
SpannerOptions.newBuilder().setProjectId(projectId)
.setCredentials(ServiceAccountCredentials.fromStream(google_application_credentials)).build();
final Spanner spanner = spannerOptions.getService();
final DatabaseId databaseId1 = DatabaseId.of(projectId, instanceId, dbId);
dbClient = spanner.getDatabaseClient(databaseId1);
// give it a first shot to speed up consecutive calls
dbClient.singleUse().executeQuery(Statement.of("select 1 from customer"));
}
}
private Customer readCustomerFromSpanner(Long customerId) {
try {
Statement statement = Statement.of(SQL + customerId);
ResultSet resultSet = dbClient.singleUse().executeQuery(statement);
while (resultSet.next()) {
return Customer.builder()
.customerId(resultSet.getLong("customer_id"))
.customerStatus(CustomerStatus.valueOf(resultSet.getString("status")))
.updateTimestamp(Timestamp.from(Instant.now())).build();
}
} catch (Exception ex) {
//log
}
return null;
}
....
}
The second approach
uses the Spring Boot Data Starter (https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-starters/spring-cloud-gcp-starter-data-spanner)
and simply goes like this
#Repository
public interface SpannerCustomerRepository extends SpannerRepository<Customer, Long> {
#Query("SELECT customer.customer_id, customer.status, customer.status_info, customer.update_timestamp "
+ "FROM customer customer WHERE customer.customer_id = #arg1")
List<Customer> findByCustomerId(#Param("arg1") Long customerId);
}
Now if i take the first approach, establishing a initial gRPC connection to Spanner takes > 5 seconds and all consecutive calls are around 1 sec. The second approach takes only approx. 400ms for each call after the initial call.
To test differences I wired up both solutions in one Spring Boot Project and compared it to a in memory solution (~100ms).
All given timings refer to local tests on dev machines but go back to investigating performance problems within the cloud environment.
I testet several different SpannerOptions (SessionOptions) with no results and ran a profiler on the project.
I seems like 96% of response time comes from establishing a gRPC channel to spanner, whereas the database itself processes and responds within 5ms.
We really don't understand the behaviour. We only work with very little test-data and a couple of small tables.
The DatabaseClient is supposed to manage the ConnectionPool and is itself wired into a Singleton-Scoped Repository-Bean. So Sessions should be reused, rigt?
Why does the first approach take much longer than the second one. The Spring FW itself simply uses the DatabaseClient as member within the SpannerOperations / SpannerTemplate.
How can we generally reduce latency. More than 200ms for plain response on each db call seems four times more than we would have expected. (I am aware that local timing benchmarks need to be treated with care)
Tracing give us good visibility into the client, hopefully it can help you with diagnosing the latencies.
Running TracingSample, I get from stackdriver. There are different backends you can use, or print it out as logs.
The sample above also exports http://localhost:8080/rpcz and http://localhost:8080/tracez you can poke around to check latencies and traces.
A tutorial on setting it up: Cloud Spanner, instrumented by OpenCensus and exported to Stackdriver
The problem here is not related to Spring or DAO's, but that you are not closing the ResultSet that is returned by the query. This causes the Spanner library to think that the session that is used to execute your query is still in use, and causes the library to create a new session every time you execute a query. This session creation, handling and pooling is all taken care of for you by the client library, but it does require you to close resources when they are no longer being used.
I tested this with very simple example, and I can reproduce the exact same behavior as what you are seeing by not closing the ResultSet.
Consider the following example:
/**
* This method will execute the query quickly, as the ResultSet
* is closed automatically by the try-with-resources block.
*/
private Long executeQueryFast() {
Statement statement = Statement.of("SELECT * FROM T WHERE ID=1");
try (ResultSet resultSet = dbClient.singleUse().executeQuery(statement)) {
while (resultSet.next()) {
return resultSet.getLong("ID");
}
} catch (Exception ex) {
// log
}
return null;
}
/**
* This method will execute the query slowly, as the ResultSet is
* not closed and the Spanner library thinks that the session is
* still in use. Executing this method repeatedly will cause
* the library to create a new session for each method call.
* Closing the ResultSet will cause the session that was used
* to be returned to the session pool, and the sessions will be
* re-used.
*/
private Long executeQuerySlow() {
Statement statement = Statement.of("SELECT * FROM T WHERE ID=1");
try {
ResultSet resultSet = dbClient.singleUse().executeQuery(statement);
while (resultSet.next()) {
return resultSet.getLong("ID");
}
} catch (Exception ex) {
// log
}
return null;
}
You should always place ResultSets (and all other AutoCloseables) in a try-with-resources block whenever possible.
Note that if you consume a ResultSet that is returned by Spanner completely, i.e. you call ResultSet#next() until it returns false, the ResultSet is also implicitly closed and the session is returned to the pool. I would however recommend not to rely solely on that, but to always wrap a ResultSet in a try-with-resources.
Can you confirm that the performance doesn't change if the SQL strings are made the same between the two methods? (* vs spelling them out individually).
Also, since you're expecting a single customer in the first method, I'm inferring that the customer ID is a key column? If so, you can use the read-by-key methods from SpannerRepository, and that might be faster than a SQL query.
We have to read a lot of data from a HDD (~50GB) into our database, but our multithreading procedure is pretty slow (~2h for ~10GB), because of a Thread lock inside of org.sqlite.core.NativeDB.reset[native] (see thread sampler).
We read our data relatively fast and use our insert method to execute a prepared statement. But only if we collected like 500.000 datasets we commit all these statements to our database. Currently we use JDBC as Interface for our sqlite database.
Everything works fine so far, if you use one thread total. But if you want to use multiple threads you do not see much of a performance/speed increase, because only one thread can run at time, and not in parallel.
We already reuse our preparedStatement and all threads use one instance of our Database class to prevent file locks (there is one connection to the database).
Unfortunately we have no clue how to improve our insert method any further. Is anyone able to give us some tips/solutions or a way how to not use this NativeDB.reset method?
We do not have to use SQLite, but we would like to use Java.
(Threads are named 1,2,...,15)
private String INSERT = "INSERT INTO urls (url) VALUES (?);";
public void insert(String urlFromFile) {
try {
preparedStatement.setString(1, urlFromFile);
preparedStatement.executeUpdate();
} catch (SQLException e) {
e.printStackTrace();
}
}
Updated insert method as suggested by #Andreas , but it is still throwing some Exceptions
public void insert(String urlFromFile) {
try {
preparedStatement.setString(1, urlFromFile);
preparedStatement.addBatch();
++callCounter;
if (callCounter%500000 == 0 && callCounter>0){
preparedStatement.executeBatch();
commit();
System.out.println("Exec");
}
} catch (SQLException e) {
e.printStackTrace();
}
}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.sqlite.core.CorePreparedStatement.batch(CorePreparedStatement.java:121)
at org.sqlite.jdbc3.JDBC3PreparedStatement.setString(JDBC3PreparedStatement.java:421)
at UrlDatabase.insert(UrlDatabase.java:85)
Most databases have some sort of bulk insert functionality, though there's no standard for it, AFAIK.
Postrgresql has COPY, and MySql has LOAD DATA, for instance.
I don't think that SQLite has this facility, though - it might be worth switching to a database that does.
SQLite has no write concurrency.
The fastest way to load a large amount of data is to use a single thread (and a single transaction) to insert everything into the DB (and not to use WAL).
I am new to jdbc programming. I am creating instance of PreparedStatement multiple times and assigning it to same reference variable. Do I need to close the first instance of PreparedStatement prior to creating second instance of PreparedStatement?
oPrepStmt = oRoot.con.prepareStatement(strSql);
if (strProviderType.length() > 0) {
strSql += " and users.usertype IN (?)";
// DO I need to close prepare statement, before creating another instance of preparestatement and assigning to same reference variable.
// i.e. oPrepStmt.close();
oPrepStmt = oRoot.con.prepareStatement(strSql);
oPrepStmt.setString(2,strProviderType);
}
oPrepStmt.setInt(1,oRoot.getTrUserId());
Does the unclosed first instance of preparedstatement causes resource leaks?
JDBC statements implement AutoCloseable and therefore indicate that should explicitly be closed when no longer needed.
An object that may hold resources (such as file or socket handles)
until it is closed. The close() method of an AutoCloseable object is
called automatically when exiting a try-with-resources block for which
the object has been declared in the resource specification header.
This construction ensures prompt release, avoiding resource exhaustion
exceptions and errors that may otherwise occur.
So as suggested by the Javadoc use a try-with-resources statement:
try (PreparedStatement pstmt = oRoot.con.prepareStatement(strSql)) {
... run sql commands ...
}
In your example you create a statement and discard it for some cases. Better to avoid this and write like:
boolean checkUserType = strProviderType.length();
try (PreparedStatement pstmt = oRoot.con.prepareStatement(checkUserType ? strSql : strSql + " and users.usertype IN (?)") {
oPrepStmt.setInt(1,oRoot.getTrUserId());
if (checkUserType)
oPrepStmt.setString(2,strProviderType);
...
}
You should always close a statement when you are done with it. In some databases/JDBC drivers, a statement also has a serverside handle. Not closing the statement will leave that handle open on the server, causing unnecessary resource consumption (mostly memory, but it might lock certain metadata objects).
On top of that on the driver side not closing the statement could also have additional resource consumption (memory, listeners to connection events, etc). Closing as soon as possible is therefor advisable.
A driver will likely deallocate this eventually, either through a finalizer, or when you close the connection, but it is not a good idea to rely on that (eg connections in a connection pool do not always correctly close statements when they are returned to the pool, etc).
Now as to your specific problem, you should modify your code to something like:
if (strProviderType.length() > 0) {
strSql += " and users.usertype IN (?)";
}
try (PreparedStatement oPrepStmt = oRoot.con.prepareStatement(strSql)) {
oPrepStmt.setInt(1,oRoot.getTrUserId());
if (strProviderType.length() > 0) {
oPrepStmt.setString(2, strProviderType);
}
oPrepStmt.execute(); // or executeQuery or executeUpdate
}
I also included a try-with-resources to ensure the statement is closed as soon as possible.
BTW, likely your use of IN(?) is not going to work like this on most (all?) databases. See PreparedStatement IN clause alternatives?
I have a problem and need some enlightenment here..
I am using trigger to detect change made to my database, means that I set all my table with trigger for insert, update, and delete (MySQL)
Then I write that change into a table that I have made specifically to contain all information about the change. Let's name it xtable. (This table is not equipped with trigger)
My Java program need to continuously read that xtable to let other application know about the change.
Well the problem is, when I read the xtable in a loop, I can only read the initial value of the xtable that is when I established the connection to the database. (connection is established outside the loop)
If a change has been made to the database which will lead to new row in xtable, this new row which is produced by the trigger is not detected no matter how many times I read it with executing "select * from xtable" query..
The code look like this:
public static void main(String[] args) {
Connection conn = null;
try {
conn = database.getConnection();
Statement state = conn.createStatement();
String query = "select * from `xtable`;";
while (true) {
ResultSet rs = state.executeQuery(query);
while(rs.next){
// Some code for letting the other application know of the change
}
}
} catch (SQLException ex) {
} finally {
if (conn != null) {
conn.close();
}
}
}
So basically if I run the program while the xtable is empty, I always gain an empty ResultSet even when there is a new row after sometimes.
Actually this problem can be solved by established the connection inside the loop, but then it will lead to another problem because it will consume more and more resource as the loop go around. (I have already try this and it will eventually use all resource on my computer after sometimes even when I have already properly closed it)
So can anyone please give me some suggestion what to do?
This is my first time posting a question here, I am sorry if there is some rule that I don't follow and please give me the right direction.
Thereis such thing as transaction isolation. It could be possible that your connection does not see changes because you did not commited transaction coming from trigger, or you did not started new one on client side. Impossible to tell without seeing your database set up.
PS: Message queuing is way better alternative
I think you'd better consider trigger instead of querying to the DBMS by looping.
If you use trigger you don't have to use that 'while' loop from Java side to check the change of DB.
Instead, trigger mechanism which is embedded in the DBMS will notify the Java side when the change happens.
For Oracle, you can call Java method from PL/SQL.
For PostgreSQL, you can call Java method from PL/Java.
For CUBRID, you can call Java method from Java stored procedure.
For MySQL, you can call Java method but I don't think it is as easy as above.
I wish this link would help you out. http://code.rocksol.it/call-java-from-mysql-trigger
Or google this keyword, "mysql java user defined functions"
Connection connection=getConnection();
statement="query";
try {
stmt = connection.prepareStatement(statement);
ResultSet rs = stmt.executeQuery();
while (rs.next()) {
}
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (connection != null && stmt != null) {
stmt.close();
}
}
What is the best way to set up connection with mysql's jdbc?
And execute simple statement. How to do that?
Thank you.
The basic boilerplate for MySQL/JDBC goes something like this:
Get the connection:
Class.forName("com.mysql.jdbc.Driver").newInstance();
Connection conn = DriverManager.getConnection("jdbc:mysql://databaseName");
Execute the statement:
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * from tableName");
while (rs.next()) {
System.out.println(rs.getString(1));
}
Close the statement and connection:
rs.close();
stmt.close();
conn.close();
You just need to make sure you have the driver installed and/or in your CLASSPATH.
This is the twenty first century - use a JPA (ORM) implementation. But if you insist on going back to the metal (at the risk of down votes) -
There are many ways of getting a JDBC connection from some driver. Using reflection with a hardwired class name is the commonest and perhaps most brain damaged. If you're going to hardwire a class name, you might as well as get the benefits of normal code (compiler catches typos, no extraneous exceptions to deal with, easier to read, explicit dependencies, better tool support, etc).
Also get in to the habit of clearing up resources safely.
So:
public static void main(String[] args) throws SQLException {
Driver driver = new com.mysql.jdbc.Driver();
Connection connection = driver.connect(
"jdbc:mysql://mydatabase",
new java.util.Properties() {{
put("user", "fred");
}}
);
try {
PreparedStatement statement = connection.prepareStatement(
"SELECT insideLeg FROM user WHERE name=?"
);
try {
statement.setString(1, "jim");
ResultSet results = statement.executeQuery();
try {
if (results.next() {
System.out.println("= "+results.getLong(1));
} else {
System.out.println("Missing.");
}
} finally {
results.close();
}
} finally {
statement.close();
}
} finally {
connection.close();
}
}
What a mess! And it doesn't even use transactions yet. Yes, use an ORM. They're very respectable these days.
You wont need to do all that for every single statement. You don't want to go around creating instantiating drivers every time. In particular the execute around idiom is useful.
It depends on your case.
If you simply need to execute some queries from standalone application then you should use single connection like:
Class.forName ("yourDriverName");
Connection cn = DriverManager.getConnection ("db url");
Statement st = cn.createStatement ();
ResultSet rs = st.executeQuery ("select * from foo");
while (rs.next()) {
doSmth ();
}
rs.close ();
st.close ();
cn.close ();
But if you are developing real application (specially web-application) then use DataSource's. Read manual of your DB and Web-server how to configure datasource. DataSource allows you to use connection-pooling - it'll nessecary to increase performance.
Configuring DataSource isn't difficult process.
Here's the sun documentation for creating a JDBC connection. From there it's easy to get access to a Statement object and run some simple SQL.
For production level systems you'll probably also want to create a connection pool.
Use Spring Framework's JDBC abstraction framework - all you need to do is create a context XML file, and use the JDBC template class. Just a few lines of XML + Java code will get you going. The advantage is keeping your connection details out of compiled Java. See:
http://www.springbyexample.org/examples/simple-spring-jdbc-template.html