I am using static variables pretty much heavily in my application. Now after the application status is finished I am facing a problem in garbage collection. The variables that are declares as static are never garbage collected and my memory runs out quickly.
The specific problem is on mysql connection. I am storing the connection variable in a static variable and so I don't have to open the connection every time I run a query. This leads to a problem of usage of all memory every time I use the connection variable to execute the query and the used memory is not released. Is it a good idea to store the connection variable in static variable ? when I tried to open and close the connection every time without static variable I solved the memory management problem but the responsiveness of the application is slowed down by 10 to 20 times.
Do you need more information to understand this problem ? If yes please ask me without down voting. Thanks!
EDIT
This is my connector class
import java.sql.*;
public class connect {
public Connection conn = null;
public connect() {
try {
if (conn == null) {
String userName = "root";
String password = "password";
String url = "jdbc:mysql://localhost/pos?zeroDateTimeBehavior=convertToNull";
Class.forName("com.mysql.jdbc.Driver").newInstance();
conn = DriverManager.getConnection(url, userName, password);
System.out.println("Database connection established");
}
} catch (Exception e) {
System.err.println("Cannot connect to database server");
}
}
}
This is my class where i am storing the connection
public class variables {
public static connect con = new connect();
}
And this method i use to execute the query
public class mysql_query {
public static ResultSet execute_mysql(Connection con, String sqlStatement) {
try {
//ResultSet result = null;
java.sql.Statement cs = con.createStatement();
ResultSet result = cs.executeQuery(sqlStatement);
return result;
} catch (SQLException ex) {
Logger.getLogger(mysql_query.class.getName()).log(Level.SEVERE, null, ex);
return null;
}
}
public static void main(String args[]){
String sql = "SELECT * FROM pos_user_login WHERE moderator='1' AND "
+ "company_id='1'";
ResultSet rs = execute_mysql(variables.con.conn, sql);
}
}
Just an idea: You might not be closing your ResultSet and Statement objects, correctly. If you don't do that, the MySQL JDBC driver might keep a hold on many resources that you don't need anymore. Especially ResultSet can be very painful, as some parts of the database cursor are still in memory.
An example to give you an idea is this:
PreparedStatement stmt = null;
ResultSet rs = null;
try {
stmt = connection.prepareStatement(...);
rs = stmt.executeQuery();
}
// Close your resources in a finally block! Because the finally block
// is executed even if you have exceptions in the try block.
// If you do this a lot of times, write utility methods...
finally {
try {
if (rs != null) {
rs.close();
}
} catch (SQLException ignore) {}
try {
if (stmt != null) {
stmt.close();
}
} catch (SQLException ignore) {}
}
Maybe it'd be better to look at using a connection pool rather than the static variable... Connection pools maintain a bunch of open connections and serve them out when they're needed. Should solve your performance problem and your memory problem.
a static variable will not garbage collected but if you are just storing a some connection data it should not be a problem. What are you exactly storing?
Matteo
Well, judging by what you say, you have an object (let's call it Obj) class which contains the static variable with the connection. Due to you creates a new Obj each time and you stores it at that moment, I think you are doing a lot of copies of the connection which the JVM is unable to clean because they are static.
You could consider the possibility of store this kind of information in a Model class, or remove the static mark in order to let the JVM collect this objects properly.
Related
I have connection provider class as bleow to return connection.
public class ConnectionProvider {
static {
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
public static Connection ConnectDB() throws ClassNotFoundException, SQLException {
try (Connection connection = DriverManager
.getConnection("jdbc:mysql://localhost:3306/jspservlet_test","root", "root");
) {
return connection;
}
}
}
Here is main method to call connection provider.
public void Test() {
try {
Connection con = ConnectionProvider.ConnectDB();
PreparedStatement ps = con.prepareStatement("");
} catch (SQLException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
But "com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed." error are always show at below line of code.
PreparedStatement ps = con.prepareStatement("");
Because, according to Oracle documentation, If use try with resources java 7 features, resources are auto close after try block even it's errors occurred or not. So even I returned the connection it's already closed.
Let me know, my usage logic is wrong?
How can I return this connection inside try with resource?
I tried many time googling for solution but does not get convenience answers for me.
Let me know your suggestion and feedback please.
What you can't do...
With a try-with-resources as you have it after you return the connection you return(d) is close(d). You can't return the connection from inside the try with resources.
What you can do...
Pass the connection (inside your try-with-resources) to a method that takes a connection. You can also use a ConnectionPool, and get the Connection when you need it (to create and execute a query).
Let me know, my usage logic is wrong?
The usage of 'try-with-resources' logic is wrong in this context, because the intention of ConnectDB() is to return a connection instance which could be actually used by the caller to send a SQL statement, but instead, the connection instance is getting auto-closed, before it could be used by the caller, because of using 'try-with-resources' construct of Java.
Quick how-to on try-with-resource and JDBC
Your ConnectionProvider's ConnectDB already declares it is throwing SQLException - so no need to catch it in here: (You should consider replacing this code with connection pool maybe)
public class ConnectionProvider {
static {
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
public static Connection ConnectDB() throws SQLException {
return DriverManager.getConnection("jdbc:mysql://localhost:3306/jspservlet_test","root", "root");
}
}
Instead use try-with-resource in your test-class to clean up your code and focus on errors your SQL code
might have:
public void Test() {
try (Connection con = ConnectionProvider.ConnectDB();
PreparedStatement ps = con.prepareStatement("SELECT 1")) {
//Prepare your Statement
ps.setInt(1, 1);
//And another try-with-resource for the result - note the closing brace
try(ResultSet rs = ps.executeQuery()) {
while(rs.next()) {
//Handle your Result
System.out.println(rs.getString(1));
}
} // This closes try-with-resource. Exception will be rethron to be caught in outer catch!
} catch (SQLException e) {
//SQL is Broken - but only ONE catch to catch them all
e.printStackTrace();
}
}
That way you gain
Better readability for your code (no calls to close surrounded by finally and if != null)
Centralized error handling if anything in your SQL code breaks (so you can focus on functional error of "statement didn't run")
Better code quality: No need to worry about Cursors, Statements, Connections not being propery closed.
I'm not sure if this code is correct. I get it from an example of my Java course but I see that in fact it never closes the connection and the exceptions doesn't look to be catched correctly. (I call the query methods from my business tier).
public class Persistence {
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
static final String DB_URL = "jdbc:mysql://localhost/myDB";
static final String USER = "user";
static final String PASS = "pass";
private static Connection con;
static {
openConnection();
}
private static boolean openConnection() {
try {
Class.forName(JDBC_DRIVER).newInstance();
con = DriverManager.getConnection(DB_URL, USER, PASS);
return true;
} catch (InstantiationException ex) {
ex.printStackTrace();
return false;
} catch (IllegalAccessException ex) {
ex.printStackTrace();
return false;
} catch (ClassNotFoundException e) {
e.printStackTrace();
return false;
} catch (SQLException e) {
System.err.println("SQL problem: " + e.getMessage());
return false;
}
}
//----------EXAMPLE QUERY-----------
public static String someQuery() throws SQLException {
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("SELECT column FROM myDB");
String data;
while (rs.next()) {
data = rs.getString("column");
}
rs.close();
st.close();
return data;
}
}
Should I open and close the connection inside every query method and delete the "static{}" expression?
Like this? (still not sure about the exceptions):
public static String someQuery() throws SQLException {
openConnection();
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("SELECT column FROM myDB");
String data;
while (rs.next()) {
data = rs.getString("column");
}
rs.close();
st.close();
con.close();
return data;
}
Thanks for your answers.
The static block is only executed once, so you open a single connection and then keep it open for the duration of the program.
That does work but is flawed for a number of reasons. For example as soon as you start multi-threading it is completely useless.
Your second example is better but still flawed, the best approach would be:
Use a connection pool to keep the connections open, request one from the pool when you need it.
Use a try-finally block to ensure you always close the connection and/or return it to the pool when done.
You do not need to create a new instance of JDBC Driver class
1) Change this to Class.forName(JDBC_DRIVER).newInstance() to Class.forName(JDBC_DRIVER)
You just need to register the class with the JVM(which involves intialization of static variables and blocks)
2)Try creating connection thread pool as suggested above
3)Use a try with resources block for Connection,Statement and ResultSet as all the three interfaces extend java.lang.AutoCloseable.As such,your resources are always closed without you having to write boiler plate code.Also,the exception in your business logic does not get masked by any exception occuring while closing a resource in finally block.
But of course you know JDK7 for that.These were implemnted as part of Project Coin.Just google it and you will get more information
I am using singleton database connection inside my java application, here is code of my connection manager class:
public abstract class DatabaseManager {
//Static instance of connection, only one will ever exist
private static Connection connection = null;
private static String dbName="SNfinal";
//Returns single instance of connection
public static Connection getConnection(){
//If instance has not been created yet, create it
if(DatabaseManager.connection == null){
initConnection();
}
return DatabaseManager.connection;
}
//Gets JDBC connection instance
private static void initConnection(){
try{
Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
String connectionUrl = "jdbc:sqlserver://localhost:1433;" +
"databaseName="+dbName+";integratedSecurity=true";
DatabaseManager.connection =
DriverManager.getConnection(connectionUrl);
}
catch (ClassNotFoundException e){
System.out.println(e.getMessage());
System.exit(0);
}
catch (SQLException e){
System.out.println(e.getMessage());
System.exit(0);
}
catch (Exception e){
}
}
public static ResultSet executeQuery(String SQL, String dbName)
{
ResultSet rset = null ;
try {
Statement st = DatabaseManager.getConnection().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
rset = st.executeQuery(SQL);
//st.close();
}
catch (SQLException e) {
System.out.println(e.getMessage());
System.exit(0);
}
return rset;
}
public static void executeUpdate(String SQL, String dbName)
{
try {
Statement st = DatabaseManager.getConnection().createStatement();
st.executeUpdate(SQL);
st.close();
}
catch (SQLException e) {
System.out.println(e.getMessage());
System.exit(0);
}
}
}
The problem is my code work perfect at the start but when time past it becomes really slow. What caused that problem and how can i fix that?
At starting time my application handles around 20 queries per second, after 1 hour of running it reaches to 10 queries per second and after 3 days of running it reaches to 1 query per 10 seconds!!
P.S: My application is a single user application that makes many queries through database.
P.S: Here is my JVM parameters in eclipse.ini:
--launcher.XXMaxPermSize
512M
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
512m
--launcher.defaultAction
openFile
--launcher.appendVmargs
-vmargs
-Dosgi.requiredJavaVersion=1.6
-Xms500m
-Xmx4G
-XX:MaxHeapSize=4500m
Unfortunately database is remote and I have not any monitoring access to it for finding out what is going on there.
Here is the example of my usage:
String count="select count(*) as counter from TSN";
ResultSet rscount=DatabaseManager.executeQuery(count, "SNfinal");
if(rscount.next()) {
numberofNodes=rscount.getInt("counter");
}
What caused that problem and how can i fix that?
The main problem that you have here is in the executeQuery() method.
You are not closing the Statement, I suppose that you have commented the line st.close() because you need the ResultSet open
for further processing.
I can see that your idea is to avoid see duplicate JDBC code in your application, but this is not the right approach.
The rule is: close the ResultSet and after that, close the Statement,
otherwise you are not releasing resources correctly and you expose to the kind of problem that you are describing.
Here you can find a good explanation about how to close resources correctly (take in mind that in your case you don´t need
to close the connection)
Edit:
An example could be
try{
Statement st = DatabaseManager.getConnection().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
ResultSet rsCount = st.executeQuery(count); //count="select count(*) as counter from TSN";
if(rsCount.next()) {
numberofNodes=rscount.getInt("counter");
}
} catch (SQLException e) {
//log exception
} finally {
rsCount.close();
st.close();
}
You should consider using a disconnected resultset like a CachedRowSet http://docs.oracle.com/javase/1.5.0/docs/api/javax/sql/rowset/CachedRowSet.html
public static ResultSet executeQuery(String SQL, String dbName)
{
CachedRowSetImpl crs = new CachedRowSetImpl();
ResultSet rset = null ;
Statement st = null;
try {
st = DatabaseManager.getConnection().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY);
rset = st.executeQuery(SQL);
crs.populate(rset);
}
catch (SQLException e) {
System.out.println(e.getMessage());
System.exit(0);
}finally{
rset.close();
st.close();
}
return crs;
}
CachedRowSet implements ResultSet so it should behave like a ResultSet.
http://www.onjava.com/pub/a/onjava/2004/06/23/cachedrowset.html
In addition to these changes, I would recommend you use a pooled datasource to get connections and close them instead of holding on to one open connection.
http://brettwooldridge.github.io/HikariCP/
Or if you arent java7, bonecp or c3po.
EDIT:
To answer your question, this solves your problem because CachedRowSetImpl doesnt stay connected to the database while in use.
This allows you to close your Resultset and Statement after you've populated the CachedRowSetImpl.
Hope that answers your question.
Although Connection Manager would close Statement and Resultset automatically, but it would be better if you close them immediately.
There's nothing else in your code will effect your single thread task, so I bet there must be something wrong in your database. Try to find out if there's any database locking or wrong column index. And also have a look at database query status, find out where's the bottleneck.
I have query regarding fetching jdbc connection from pool in sub method.Following are two method i came across suggest me which one best to avoid connection leakage and tell if any other solution.
Method 1:
getConnection is method which return Connection.
void testMain(){
Connection conn = getConnection();
subMethod(conn)
conn.close();
}
void subMethod(connection conn){
// use jdbc connection
return;
}
Method2:
void testMain(){
Connection conn = getConnection();
subMethod()
conn.close();
}
void subMethod(){
Connection conn = getConnection();
conn.close();
return;
}
The place where you need a Connection should get the connection.
The way you ensure that no resources are "leaked" is by using java 7's try-with-resource syntax:
public String fetchSomeData() {
try (Connection conn = getConnection()) { // This line, with this syntax, will ensure that it is automatically closed in an invisible "finally" block
// Do what you need to do with the data, return it or something else
} catch (SQLException e) {
// No need to do clean up here, log the exception or do whatever you want.
}
}
You can use try-with-resource syntax on any objects that implement AutoCloseable interface. That includes Connection, Statement, and Resultset among others.
If you need to do a transaction you might want to initialize the Connection in a method, and then pass that Connection to different other methods that adds to the transaction, and then commit it. If that's the case, you can do:
public String fetchSomeDataInTransactionStyle() {
try (Connection conn = getConnection()) { // This line, with this syntax, will ensure that it is automatically closed in an invisible "finally" block
conn.setAutocommit(false);
addSomethingToTransaction(conn);
addSomethingMore(conn);
conn.commit();
} catch (SQLException e) {
// No need to do clean up here, log the exception or do whatever you want.
}
}
I've written code which parses a file which contains MAC Addresses and Flag values to be updated in an Oracle table. However, since this process will be run on thousands of record I wish to split up the workload and update the database simultaneously. I am unsure of the best way to implement this as I am a beginner at concurrency. I've been reading up and looking at sample code, but it is still very ambiguous and unclear to me.
My first idea was to split the list into 10 segments but it became overcomplicated and convoluted with Lists of Lists and the sort...
I'm just looking for a nudge in the right direction...
Attached is my current code:
import java.io.*;
import java.util.*;
import java.text.ParseException;
import java.text.ParseException;
import java.lang.String;
import java.sql.*;
import java.lang.Class;
import oracle.jdbc.*;
import oracle.jdbc.driver.OracleDriver;
public class Process{
public FlagProcess(){
running = false;
}
public static List<String> readFile(String filename) throws IOException {
BufferedReader macAddresses = null;
List<String> info = new ArrayList<String>();
try {
macAddresses = new BufferedReader(new FileReader(filename));
String line = null;
while ((line = macAddresses.readLine()) != null) {
//Process the data, here we just print it out
System.out.println(line);
String[] bufferArray = line.split("\\|");
String mac = bufferArray[0];
String value = bufferArray[1];
System.out.println("MAC: " + mac);
System.out.println("PPV Value: " + value);
info.add(mac);
info.add(value);
}
} catch (FileNotFoundException ex) {
ex.printStackTrace();
} finally {
//Close the BufferedReader
try {
if (macAddresses != null)
macAddresses.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
return info;
}
public static Connection getConnection() throws SQLException,ClassNotFoundException{
DriverManager.registerDriver(new oracle.jdbc.OracleDriver());
String URL = "jdbc:oracle:thin://#xxxxxxxxxx:1521:xxxxxx";
Connection conn = DriverManager.getConnection(URL, "username", "password");
System.out.println("Connection established...");
return conn;
}
public static void freeConnection(Connection conn) throws SQLException {
try {
if (conn != null) {
conn.setAutoCommit(true);
}
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
System.out.println("FlagProcess.freeConnection() - got exception while closing connection.");
e.printStackTrace();
}
}
}
}
public static synchronized void updateDatabase(String mac, String value, Connection conn) throws SQLException
{
String update = "UPDATE device set FLAG = ? where IDENTIFICATION = ?";
System.out.println(update);
try{
PreparedStatement pstmt = conn.prepareStatement(update);
pstmt.setString(1, value);
pstmt.setString(2, mac);
int x = pstmt.executeUpdate();
System.out.println("Update complete.");
}
catch(Exception e)
{
e.printStackTrace();
}
finally{
freeConnection(conn);
}
}
public static void main(String [] args){
PPVFlagProcess pfp = new PPVFlagProcess();
try{
List<String> info= readFile("values");
String mac = info.get(0);
String value = info.get(1);
Connection conn = pfp.getConnection();
pfp.updateDatabase(mac, value, conn);
pfp.freeConnection(conn);
}
catch(Exception e){
e.printStackTrace();
}
}
}
Any help would be greatly appreciated, Thank you.
This sounds like premature optimization to me. With the concurrent approach you would increase performance on the client side. But I would expect your bottleneck to be on the database side (network, database cpu, lock contention, disc io). With the concurrent approach this could even result in worse performance.
So if you want to get this stuff fast I would look into sqlldr and stuff.
And before anything: Get a simple solution working and then look for the bottleneck.
I can't see any concurrency in your code, if you want a better performance, set batch size on your prepared statement, and execute statements once a while (for example after 20 records).
If you worry about the performance of your code, start with getting rid of the autocommit. You explicitly set it to true (the (cruel) default). Your code will already be a lot faster by setting autocommit to false.
Next thing to do is to use bulk loading. See 23 Performance Extensions, use the Oracle way to perform batches, not the standard way. Do this and you may ask yourself: why did I do it so difficult.
In Oracle there are a few
things not to do:
autocommit
connect/disconnect for every small element of work
use dynamic (unprepared) SQL
I think, you are slowing down the hole thing, by executing it multithreaded. Use prepared statements and a batchupdate instead.
The System.out are a bad idea, if performance is important.