In my Java application I am establishing KDB connection using
private void initConnection() throws KException, IOException {
conn = new c(host, port);
conn.tz = TimeZone.getTimeZone(CONNECTION_TIMEZONE);
}
Then I use conn for inserting data asynchronously in tables successfully.
KDB server is restarted once in week and at that time as connection is reset by server I am getting an exception java.net.SocketException.
Using the above exception I can again re-establish lost connection but this is not a decent solution as I lose batch etc. and I don’t want connection exception to be thrown at the time of insertion.
Instead, is there a call-back/listener/Exception method in KDB API where API can inform java application upfront if a connection is closed by server / lost?
Why don't you just do a
SELECT 1;
on your connection.
If it's open it'll run.
If it's closed it should throw an error because of the failing network connection.
You'd run this test only in the moments that your kbd server is scheduled for restart.
Personally, on the scheduled moments of restarting the kbd database I'd dump the data into an sqlite database for holding, and then when the frame for rebooting is over and connection is re-established dump the contents of the sqlite database into the kbd database.
If connection is lost, I would expect conn object to throw IOException or its successor, when you try to hit KDB.
Ie, you should catch IOException and
invoke initConnection method to re-establish connection
re-execute query
Would be nice to set reasonable threshold of retries either. Otherwise, if KDb is down, the code will keep trying to connect to it.
This answer is related (and possibly adds on) to Anton Dovzhenko's answer above, but hopefully the below helps you as well.
If we take a look at kx's documentation for reconnecting to a q process automatically, we can observe the following code snippet from ReconnectionExample.java below.
//initiate reconnect loop (possibly within a catch block).
while (true) {
try {
System.err.println("Connection failed - retrying..");
//Wait a bit before trying to reconnect
Thread.sleep(5000);
qConnection = qConnFactory.getQConnection();
System.out.println("Connection re-established! Resuming..");
//Exit loop
break;
} catch (IOException | KException e1) {
//resume loop if it fails
continue;
}
…
}
That would be one way to try and reconnect; within a catch block. However, to answer your questions in greater detail:
To alert the rest of your application, do so in the catch block. One way to go about doing this is to invoke another call back method to alert that the connection is now down.
To reconnect, do this as well in your catch block as well. Be careful to only do this at a specified interval, though. The reason why I mention that is because when a Java application tries to reconnect, it will create a new c object.
This should help your application handle reconnecting more efficiently, and notifying any other dependencies appropriately.
Related
We are currently dealing with the function, that has to work partially with the database, and partially with a service, whice operations are time-consuming. So, generally speaking, here is a transactional method, that has a code like this:
Connection conn = null;
try {
conn = getConnection(); // This I get from the connection pool
Employee emp = queryDatabase(id);
// Point A - ??
Response resp = makeLongTimeServiceCall(emp);
// Point B - ??
anotherQueryDatabase(resp);
} catch (Exception e) {
throw e; // And this also rolls back the transaction
} finally {
// If conn is not closed, close it, so there would be no leaks
}
So the big question is - should I close the connection at point A and then get it again from the connection pool at point B, so that other servlets could use that connection while I interact with the service? My guess is that I should, but will this hurt my transaction?
In many circumstances: yes, closing and reopening the connection sounds good. However you need to understand the implication of doing them in two separate transactions (by closing and re-opening the connection you are inherently doing them in separate transaction).
If another user invokes an operation that changes the state of your data at Point B will the end result still be correct?
If you don't have a transaction you can close the connection and ask for a new connection when needed.
Remember that if you are using a connection pool closing a connection will not really close the connection, but only flag it as reusable by other threads.
Database connections shouldn't be left open. Open the connections just when you need to execute a query and close it as early as possible. So my solution is, YES.
if your application doesn't close connection properly may lead to some issues like the
connection pool maxing out.
Applications Not Properly Closing Connections:
When writing an application to use the WebSphere Application Server(WAS) datasource, the best way is fallowing get/use/close pattern.
1.get
-This is when the application makes a request to get a connection from the pool.
The application will first lookup the datasource name and then do a getConnection() call.
2.use
-This is when the application executes a query and waits for a response
3.close
-This is the final stage when the application has received the response from the database and is now done using that connection.
-The application should call close() in a finally block to close out the connection and return it to the free pool.
If your application does not follow this pattern, you may see connections staying open for long periods of time. This is especially seen when the application does not close the connection properly. After the response is received from the database, the application should be calling close in a finally block. If the application does not explicitly close the connection, that connection will stay open forever in WAS until that server is restarted. This is known as a connection leak. To resolve this type of issue, you will have to modify your application to close every connection
for further information: https://www.ibm.com/support/pages/common-reasons-why-connections-stay-open-long-period-time-websphere-application-server
I have a program that performs frequent requests to a database using a single-threaded workflow. The requests require obtaining connections within a second or less of each other, but one connection should always be closed before a new one is opened.
However, over time my debugging shows that the connections rack up rather quickly. It seems that connections start multiplying, and eventually reach the point where the program hangs because the connection pool's maximum (currently using BoneCP) is filled. I have the maximum set at 15, but it had similar behaviour when it was at the default number (8 I believe).
I am using try-with-resource statements, similar to the one below, to close the connections.
//Connection example
//Connection pool is configured.
DataSource source = new DataSource();
try (Connection con = source.getConnection();
PreparedStatement stmt = con.prepareStatement(query) {
//Do SQL stuff
} catch (SQLException e) {
//Handle Exception
}
If I understand try with resources right, this should automatically close the connection, and return it to the pool. Since this didn't work, I tried adding a con.close() statement in an attempt to better handle the connections. This quantitatively seemed to help, but I'm not sure it did anything in reality.
Why are so many connections being created, and how can I better manage my connections so I don't create so many?
I am supporting some legacy code and it's chugged along fine until recently. I am looking for if there is a setting for JDBC Oracle thin connection where I can specify idle timeout via Java (no connection pooling)? A lot of resources online refer to connection pooling... is it even possible in my case (to specify idle timeout, in a non-pooling situation)? Or is idle time a setting on the specific DB user account?
Updates + Questions
I was able to log in as the user, and ran a query to try to find out resource limits. select * from USER_RESOURCE_LIMITS; However everything came back "UNLIMITED". Is it possible for another value (say from the JDBC connection) to override the "UNLIMITED"?
So the job holds onto the connection, while we actively query another system via DB links for a good duration of ~2+ hours... Now, why would the idle timeout even come into play?
Update #2
We switched to a different account (that has the same kind of DB link setup) and the job was able to finish like it did before. Which sort of points to something wonky with the Oracle user profile? But like I said, querying USER_RESOURCE_LIMITS shows both users to have "UNLIMITED" idle time. DBA pretty confirmed that too. What else could be causing this difference?
Update #3
Stack trace and such.
java.sql.SQLException: ORA-02396: exceeded maximum idle time, please connect again
ORA-06512: at line 1
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:316)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:282)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:639)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:184)
at oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:873)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1086)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2984)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3076)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4273)
at com.grocery.stand.Helper.getAccess(Helper.java:216)
at com.grocery.stand.fruitbasket.Dao.getPriceData(Dao.java:216)
at com.grocery.stand.fruitbasket.Dao.getPricees(Dao.java:183)
at com.grocery.stand.fruitbasket.UpdatePrice.updateAllFruitPrices(UpdatePrice.java:256)
at com.grocery.stand.fruitbasket.UpdatePrice.main(UpdatePrice.java:58)
SQL Exception while getting Data from SYSTEM_B
Exception while updating pricing : ORA-01012: not logged on
Exception in thread "main" java.sql.SQLException: ORA-01012: not logged on
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:316)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:277)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:272)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:129)
at oracle.jdbc.driver.T4CConnection.do_rollback(T4CConnection.java:478)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:1045)
at com.grocery.stand.Helper.rollBack(Helper.java:75)
at com.grocery.stand.fruitbasket.UpdatePrice.updatePartNumbers(UpdatePrice.java:291)
at com.grocery.stand.fruitbasket.UpdatePrice.main(UpdatePrice.java:58)
Connection Code
public static Connection openConnection() throws SQLException {
String userName = propBundle.getString(DB_UID);
String password = propBundle.getString(DB_PWD);
String url = propBundle.getString(DB_URL);
Connection conn = null;
try {
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
conn = (Connection) DriverManager.getConnection(url, userName,
password);
conn.setAutoCommit(false);
} catch (SQLException sqle) {
sqle.printStackTrace(System.out);
throw sqle;
}
return conn;
}
Error occurs on line execute()
public static void getSystemAccess(Connection dbConnection) throws SQLException {
try {
CallableStatement authStmt = null;
String authorize = "CALL ABC.ACCESS_PROCEDURE#some_db_link()";
authStmt = dbConnection.prepareCall(authorize);
authStmt.execute();
authStmt.close();
} catch (SQLException sqle1) {
sqle1.printStackTrace();
throw new SQLException(sqle1.getMessage());
}
}
I'm not sure that I understand the question you're asking.
The error you are getting indicates that the Oracle user that you are using to connect to the database has a profile configured (in Oracle) that limits the amount of time the connection can be idle. Oracle is killing your connection when the connection remains idle too long. Normally, the solution to this sort of problem would be to go to the DBA and ask for the idle time to be increased or to look through your code and see why the connection is open and unused for so long. If you were using a connection pool (which it doesn't appear you are), it would make sense for some connections to remain open and idle for long periods of time. Since it doesn't appear that you are using a connection pool, the question is whether it makes sense for the application to hold open the connection for long periods of time without doing anything. If the application opens a connection when the user logs in at 9am and doesn't close it until the user shuts down at 5pm, it may make sense to adjust the IDLE_TIME setting for this user in the database. Otherwise, you may want to investigate whether it makes logical sense for the application to hold open the database connection so long without doing something or whether the application can be modified to close the connection when it is no longer needed.
I have a bukkit plugin (minecraft) that requires a connection to the database.
Should a database connection stay open all the time, or be opened and closed when needed?
The database connection must be opened only when its needed and closed after doing all the necessary job with it. Code sample:
Prior to Java 7:
Connection con = null;
try {
con = ... //retrieve the database connection
//do your work...
} catch (SQLException e) {
//handle the exception
} finally {
try {
if (con != null) {
con.close();
}
} catch (SQLException shouldNotHandleMe) {
//...
}
}
Java 7:
try (Connection con = ...) {
} catch (SQLException e) {
}
//no need to call Connection#close since now Connection interface extends Autocloseable
But since manually opening a database connection is too expensive, it is highly recommended to use a database connection pool, represented in Java with DataSource interface. This will handle the physical database connections for you and when you close it (i.e. calling Connection#close), the physical database connection will just be in SLEEP mode and still be open.
Related Q/A:
Java Connection Pooling
Some tools to handle database connection pooling:
BoneCP
c3po
Apache Commons DBCP
HikariCP
Depends on what are your needs.
Creating a connection takes some time, so if you need to access database frequently it's better to keep the connection open. Also it's better to create a pool, so that many users can access database simultaneously(if it's needed).
If you need to use this connection only few times you may not keep it open, but you will have delay when you would like to access database. So i suggest you to make a timer that will keep connection open for some time(connection timeout).
You need to close your connections after each query executions.Sometimes you need to execute multiple queries at the same time because the queries are hanging from each other.Such as "first insert task then assign it to the employees".At this time execute your queries on the same transaction and commit it, if some errors occur then rollback.By default autocommit is disabled in JDBC. Example
Use connection pooling.If you are developing a webapplication then use App Server connection pooling.App server will use the same pooling for each of your applications so you can control the connection count from the one point.Highly recommend the Apache Tomcat Connection pooling.Example
As an additional info:
Connection, Statement and ResultSet.
1.If you close connection you don't need close statement or resultset.Both of them will be closed automatically
2.If you close Statement it will close ResultSet also
3.if you use try-with-resources like this:
try (Connection con = ...) {
} catch (SQLException e) {
}
it will close the connection automatically.Because try-with-resources require autoclosable objects and Connection is autocloseable.You can see the details about try-with-resources here
Actually, it's all matter on how you write your application! It's an art, but sadly everyone takes a tutorial for a good practice like Microsoft's tutorials.
If you know what you are coding, then you keep your connection open for the lifetime of the application. It's simple, not because you have to go at work in the morning that everyday we have to build a special route just for you! You take that single route or 2 or 4 like everyone does! You judge for the traffics and you build 2, 4 or 6 routes as needed. If there is traffic with these 4 or 6 routes, you wait!
Happy coding.
The Connection should be opened only when required. If it is open before the actual need, it reduces one active connection from the connection pool..so it ultimately effects the users of the application.
So,it is always a better practice to open connection only when required and closing it after completion of process.
Always try puttting you connection close logic inside the finally block that will ensure that your connection will be closed,even if any exception occurs in the application
finally
{
connection.close()
}
I am passing Resultset object to each thread. Each thread is connecting to the database and inserting data. Untill thread 110 it is working fine. After it crosses 111 thread it throws the above exception.
I am using oracle 11g.
My sample Thread code is:
class MyThreadClass implements Runnable
{
public Connection connection;
public Statement statement2;
public ResultSet rs2;
public String cookie;
public MyThreadClass(ResultSet rs1)
{
rs2=rs1;
}
public void run()
{
try
{
cookie=rs2.getString("COOKIE");
driver = "oracle.jdbc.driver.OracleDriver";
url = "jdbc:oracle:thin:#127.0.0.1:1521:xx";
/* connection
statement2.executeUpdate("INSERT INTO visit_header VALUES ('"+cookie+"')");
}
I am not getting how to handle this exception.
Your multi-threaded application is opening too many Connections/Sessions. Hence, the listener is dropping and blocking new connections for a while.
Check your DB resource usage first:
SELECT * FROM v$resource_limit WHERE resource_name IN ('processes','sessions');
Check to see if your MAX_UTILIZATION for either your Processes or Sessions is getting too close to the LIMIT_VALUE. If yes, you should either:
Use DB Connection pooling to share Connection objects between threads. Or,
Increase the number of processes/sessions that Oracle can service simultaneously.
Actually, Connection Pooling (#1) should always be done. An application cannot scale up otherwise. Check Apache Commons DBCP for details. For #2, open a new SQL*Plus session as SYSTEM and run:
ALTER system SET processes=<n-as-per-number-of-threads> scope=spfile;
to increase backend concurrency. Then RESTART the Database. IMPORTANT!
I guess the database just don't accept more connections from your host. If I understand your question right you are making maybe 100 threads which each connects to the database in short time. Maybe you don't even close the connection correctly, or the accesses are lasting so long that a huge amount of connections are opened. The database have a limit to which it accepts connections.
You should definitely reduce the number of connections by some clever technique. Maybe reduce the number of concurrent threads and/or use a connection pool.
Try this solution at your end. It worked for me.
Close the connection in try/catch block and just after closing the connection,
write-
Thread.sleep(1000);
In this case you can write it as-
finally {
try {
if (conn != null && !conn.isClosed())
{
conn.close();
Thread.sleep(1000);
}
}
catch (SQLException e) {
e.printStackTrace();}
}