Trying to use sql statement batchs to do the following: every 5 minutes, add a statement to batch (current counter) then every hour send, send the statements to the database.
I'm curious though, do I need to reinitialize the statement/connection whenever I add to it or send the batch?
here's how i think i would go about doing this, just need some clarification on how to do it smarter or if this is the best way
on program startup, initialize the following
Connection connection = null;
Statement statement = null;
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/youtube", "root", "root");
statement = connection.createStatement();
then every 5 minutes....
addToBatch(connection, statement, counter, time, date);
then hour...
statement.executeBatch();
am i missing anything? do i need to remake the connection?
any information is helpful, thank you!
It would be better to keep the query information in a simple list and then create batch and add statements when required.
Your connection could get timed out at client side (connection pool settings) or server side. For e.g., in MySql wait_timeout parameter.
Even if it is not closed, it is not a good idea to hold up db resources for such long periods unnecessarily.
Hope it helps.
You can add autoReconnect=true parameter to your connection string
Related
My server app uses prepared statements in almost all cases, to prevent sql injection. Nevertheless a possibility is needed providing special users executing raw SELECT queries.
How can I more or less securely make sure the query does not modify the database? Is it possible to execute a query read only, or is there any other 'secure' way making sure noone tries any sql injection?
(Using sqlite3, so I cannot use any privileges)
Thanks a lot!
JDBC supports read-only connections by calling Connection.setReadOnly(true). However the javadoc says:
Puts this connection in read-only mode as a hint to the driver to enable database optimizations.
Some JDBC drivers will enforce the read-only request, others will use it for optimizations only, or simply ignore it. I don't know how sqlite3 implements it. You'll have to test that.
Otherwise, you could do a "simple" parse of the SQL statement, to ensure that it's a single valid SELECT statement.
I'm not aware of a general JBDC configuration which specifies readonly. But Sqlite does have special database open modes and this can be leveraged in your connection to your sqlite database. Eg.
Properties config = new Properties();
config.setProperty("open_mode", "1"); //1 == readonly
Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db", config);
Credit: https://stackoverflow.com/a/18092761/62344
FWIW All supported open modes can be seen here.
If you use some sort of factory class to create or return connections to the database, you can individually set connections to be read-only:
public Connection getReadOnlyConnection() {
// Alternatively this could come from a connection pool:
final Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db");
conn.setReadOnly(true);
return conn;
}
If you're using a connection pool, then you may also want to provide a method for getting writeable connections too:
public Connection getWriteableConnection() {
final Connection conn = getPooledConnection(); // I'm assuming this method exists!
conn.setReadOnly(false);
return conn;
}
You could also provide just a single getConnection(boolean readOnly) method and simply pass the parameter through to the setReadOnly(boolean) call. I prefer the separate methods personally, as it makes your intent much clearer.
Alternatively, some databases like Oracle provide a read only mode that can be enabled. SQLite doesn't provide one, but you can emulate it by simply setting the actual database files (including directories) to read only on the filesystem itself.
Another way of doing it is as follows (credit goes to deadlock for the below code):
public Connection getReadOnlyConnection() {
SQLiteConfig config = new SQLiteConfig();
config.setReadOnly(true);
Connection conn = DriverManager.getConnection("jdbc:sqlite:sample.db",
config.toProperties());
}
We have BoneCP library managing our connection pools. I'd like to know if the following statement will return the connection back to the pool.
statement = conn.createStatement();
....
lots of code.
....
Connection conn = statement.getConnection();
statement.close();
conn.close();
Will the above code close the connection and put the connection back into the pool?
Update:
I'm asking this question because when I run the statistics on connectionpool, i still see the conPool.getTotalLeased() showing that 2 connections are being used. But, I've closed the connections using the above mechanism.
The whole sense of a pool is to hold already established connections to your database. When you retrieve a connection from your pool you save the time to connect to your database.
What you are seeing is the pool holding your connections so it is all fine.
When you close your connection, it is only returned to the pool and marked as available for your next retrieval.
Yes it does moves the connection back to the pool. There was a mistake when getting the connection from the pool, fixed it, now i'm not seeing the 2 connections in the totalLeased() method.
Mistake that I found
conPool.getConnection(); // loitering connection which we see in the getTotalLeased();
..
Statement st = conPool.getConnection().getStatement(); //we have handle to this connection.
return st;
I just want to make sure that if I use the following, I am opening a separate DB session and not resuing the same one (for testing I need individual sessions).
Connection connection = DriverManager.getConnection(URL,USER,PASSWORD);
each time I do the above code, I run my query, then do a connection.close()
So for example:
while(some condition) {
Connection connection = DriverManager.getConnection(URL,USER,PASSWORD);
//now use the connection to generate a ResultSet of some query
connection.close();
}
So, each iteration of the loop (each query) needs its own session.
Is this properly opening separte sessions as I need (and if not, what would I need to add/change)? thanks
The javadoc says:
Attempts to establish a connection to
the given database URL
Slightly woolly language, and I suspect that this is up to the JDBC driver, but I'd be surprised if this did anything other than open a new connection.
I suppose it's possible for a JDBC driver to perform connection pooling under the hood, but I'd be surprised to see that.
In the case of the Oracle JDBC driver, this will open a new connection every time. This is a relatively slow process in Oracle, you may want to consider using a connection pool (e.g. Apache Commons DBCP, or c3p0) to improve performance.
I am facing an issue while executing queries.I use the same resultSet and statement for excecuting all the queries.Now I face an intermittent SQlException saying that connection is already closed.Now we have to either have separate resultSet for each query or have lock like structure.Can anyone tell which is better.I think introducing locks will slow down the process.Am I right?
Update:
To be more clear.The error may happen because the finally block gets called before all the queries get executed and the connection gets closed and exception will be thrown.
This is the exception I get
java.sql.SQLException: Connection has
already been closed. at
weblogic.jdbc.wrapper.PoolConnection.checkConnection(PoolConnection.java:81)
at
weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:68)
at
weblogic.jdbc.wrapper.ResultSet_com_informix_jdbc_IfxResultSet.next(Unknown
Source) at
com.test.test.execute(test.java:76)
at
org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:413)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:225)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1858)
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:459)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
at
weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
at
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
at
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
at
weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
at
weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
Sample code:
ResultSet rst=null;
Statement stmt=null;
Connection con=DBConnection.getConnection();
stmt=con.createStatement();
rst=stmt.executeQuery("select * from dual");
while(rst.next())
{ : ://Some code }
rst=stmt.executeQuery("select * from doctor where degree="BM");
while(rst.next())
{ //blah blah }
finally
{
//close con,rst and stmt
}
you are not reusing the resultset, you are leaking resultsets.
rst=stmt.executeQuery... generates a new resultset and the previous resultset is never closed :(
It appears that the code in question has issues in multi-threaded environment.
DBConnection.getConnection() is probably returning the same connection to all threads. When multiple threads are processing multiple requests, the first thread that finishes execution of the method will close the connection, leaving all other threads high and sundry.
I'm speculating here, but is appears that the connection object returned by DBConnection is an instance member of the DBConnection object, and that would qualify as a bad practice for a connection manager in a multi-threaded environment.
A code fix would avoid the usage of instance members for Connection, Statement (and the like), and the ResultSet objects.
I'm not sure what's going on without knowing more about your code. Is it threaded ? Is the underlying database going down (or are you losing connectivity to it).
One thing I would do is to implement connection pooling (via Apache DBCP, say). This framework will maintain a pool of connections to your database and validate these connections before handing them out to you. You would ask for a new connection each time you make a query (or perhaps set of queries) but because they're pooled this shouldn't be a major oeverhead.
Unless your connection to the database has really been closed I think you did something more like this:
try {
return resultSet.getBoolean("SUCCESS");
} finally {
resultSet.close();
}
This code will actually close the connection before your result set is being evaluated, resulting in the exception you show.
I have create a getDBConnection method in my Java application. This returns a connection object, and hence I haven't closed this connection in this method itself.
Now, I am invoking this method from various methods in my application at regular intervals, and closing them inside a try - finally block. I thought this should free up the connection after use. However, I am seeing a large number of connections opened (about 50) in the MySQL Administrator's Server Connections tab.
//Defining a method to retrieve a database connection
// PropDemo is a properties class that retrieves Database related values from a file
public Connection getDBConnection() {
//Instantiating the Properties object
PropDemo prop = new PropDemo();
Connection con = null;
// Retrieving values from the parameters.properties file
String JdbcDriver = prop.getMessage("JdbcDriver");
String JdbcUrlPrefix = prop.getMessage("JdbcUrlPrefix");
String DBIP = prop.getMessage("DBIP");
String DBName = prop.getMessage("DBName");
String DBUser = prop.getMessage("DBUser");
String DBPassword = prop.getMessage("DBPassword");
try {
// Loading and instantiating the JDBC MySQL connector driver class
Class.forName(JdbcDriver).newInstance();
con = DriverManager.getConnection(JdbcUrlPrefix + DBIP + "/" + DBName, DBUser, DBPassword);
if (con.isClosed())
Logger.log("Connection cannot be established", "vm");
} catch (Exception e) {
Logger.log("Exception: " + e, "vm");
Logger.log(Logger.stack2string(e), "vm");
}
return con;
}
I am also closing the associated ResultSet and Statement Objects. What could be missing here?
I am planning to replace all the Statements with PreparedStatements for efficiency and security reasons. Will that help significantly? What else can be done?
EDIT:
This is just a core java application that is repeatedly quering for changes in some fields in a MySQL database through MySQL-JDBC connector. I am not using any framework like Spring or Hibernate.
Your code looks sane.
That's how you're creating a new connection.
Probably the error is where you close it.
You should close it in a finally block.
Some additional questions.
1) Are you sure those 50 conections come from this program ? Maybe there are some others comming from your same office. To confirm this you would need to stop the program, and look again in your connection monitor.
2) Does your application uses many connection simultaneously? Probably its a peak when you're using 50 at the same time.
If you can post the code where you close the connection. Chances are the problem is there.
Additionally I would suggest you to use a connection pool. You can build one your self or you can see the results from this page:
How many JDBC connections in Java?
Are you closing the connection object when you application closes as well?
Are you using your JDBC connection within a J2EE application server or with Hibernate?
Both of these tend to start out with a fairly high connection pool to begin with, so you would see a large number.
Check out the details on connection pooling.
You could take a Singleton approach to the problem and only create a new Connection object if the current one is null:
If (connectionObject != null){
return connectionObject;
}else {
//create new connection object
}
This will make sure that you only have one non-null connection at any time.