I have two application servers distinctly located in two different data centers, running the application in active-active mode. The application db is also hosted between the same two data center in active-passive mode. I am receiving connection reset errors from my application server which is on the other datacenter, when connecting to the DB. These connection reset errors are intermittent and no ORA/Java exception codes are provided with it. ht e datacenter diagram is provided here
enter image description here
java.sql.BatchUpdateException: Io exception: Connection reset
A network appliance somewhere between your app server and the Database may kill the socket based on inactivity. This happens with large connection pools where all connections aren't used frequently. It could be resolved by turning keep_alive on all the JDBC connections. To do so set the JDBC property "oracle.net.keepAlive" to "true".
Related
We are using H2 started as database server process and listening on standard TCP/IP port 9092.
Our application is deployed in a Tomcat using connection pooling. We do a purge during idle time which at the end results in closing all connections to H2. From time to time we observe errors when the application tries to open the connection to H2 again:
SCHEDULERSERVICE schedule: Exception: Database may be already in use: "Waited for database closing longer than 1 minute". Possible solutions: close all other connection(s); use the server mode [90020-199]
org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database may be already in use: "Waited for database closing longer than 1 minute". Possible solutions: close all other connection(s); use the server mode [90020-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:617)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:205)
at org.h2.message.DbException.get(DbException.java:181)
at org.h2.engine.Engine.openSession(Engine.java:209)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178)
at org.h2.engine.Engine.createSession(Engine.java:161)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:160)
at java.lang.Thread.run(Thread.java:748)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:617)
at org.h2.engine.SessionRemote.done(SessionRemote.java:607)
at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:143)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:431)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:317)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:169)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:148)
at org.h2.Driver.connect(Driver.java:69)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
The problem occurs when the Tomcat connection pool closes all idle connection (unused) and one connection still in use is closed afterwards.
The next attempt to open a new connection fails, a retry is successfully after some wait time.
Under which circumstances does this exception happen?
What does the exception mean?
Are there any recommendations to follow to avoid the problem?
It sounds to me that H2 does a database close after the last connection has been closed.
When does the database close occure?
How can database closures been controlled?
Thx in advance
Thorsten
Embedded database in web applications needs careful handling of its lifecycle.
You can add a javax.servlet.ServletContextListener implementation (marked with #WebListener annotation or included into web.xml) and add explicit database shutdown to its contextDestroyed() methods.
You can force database shutdown here with connection.createStatement().execute("SHUTDOWN"). If your application needs to write something to database during unload, it should do it before that command.
Without the explicit shutdown H2 closes the database when all connections are closed, if some other behavior wasn't configured explicitly (with parameters in JDBC URL, for example). For example, DB_CLOSE_DELAY sets the additional delay, maybe your application uses that setting and therefore H2 doesn't close the database immediately, or application doesn't close all connections immediately.
Anyway, when you're trying to update the web application of the fly, Tomcat tries to initialize the new version before its old version is unloaded. If H2 is in classpath of the web application itself, the new version will be unable to connect to the database during short period of time when the new version is already online but the old version isn't unloaded yet.
If you don't like it, you can run the standalone H2 Server process and use remote connections to it in your web applications.
Another option is to move H2 to the classpath of Tomcat itself and configure the connection pool as resource in the server.xml, in that case it shouldn't be affected by the lifecycle of your applications.
In both these cases you shouldn't use the SHUTDOWN command.
UPDATED
With client-server connections to a remote server such exception means that server decided to close the database because there are no active connection. This operation can't be interrupted and reverted in the middle. On attempt to open a new connection to the same database during this process it waits at most for 1 minute for completion of this process to re-open the database again. This timeout is not configurable.
There are two possible solutions.
DB_CLOSE_DELAY setting can be used with some large value in seconds. When all connections are closed, database will stay online for the specified number of seconds. -1 also can be used to set an infinite timeout.
You can try to speed up the shutdown process, but you have to figure out what takes so much time by yourself. The file compaction procedure is limited to 200 milliseconds by default, it may take a longer time, but I think it shouldn't be that long. Maybe you have a lot of temporary objects or uncommitted data. Maybe you have a very high fragmentation of database file. It's hard to say what's going wrong without further investigation.
As per IBM documentation:
Purge Policy
Specifies how to purge connections when a stale connection or fatal connection error is detected.
Valid values are EntirePool and FailingConnectionOnly.
Question:
How/When does the server get to know that a connection got staled? Does it purge the pool as soon as (immediately?) any connection goes stale or it happens as per the reap time?
Say if the reap time is 180 seconds. Let's say the reap thread last ran at 3:05 PM and a connection goes stale at 3:06 PM, will the server purge the pool at 3:06 PM itself or the purge will happen only at 3:08 PM ? Is there a risk of clients getting staled connection objects between 3:06 and 3:08 ?
The IBM document i'm referring to is:
https://www.ibm.com/support/knowledgecenter/en/ssw_i5_54/rzamy/50/admin/help/udat_conpoolset.html
Stale connections are identified in the following ways:
JDBC operation is performed that raises SQLRecoverableException or SQLNonTransientConnectionException, or a general SQLException with a SQL State or error code that the application server has built-in knowledge of. For specific lists of SQL states and error codes, refer to the SQLState mappings in DatabaseHelper and its various subclasses per database.
JDBC driver's ConnectionEventListener.connectionErrorOccurred signals the application server that a connection has gone bad.
When the application server learns that a connection has gone bad, it does not return that connection to the pool. Subsequent requests outside of the sharing scope would never get that same connection.
Purge Policy determines what the application server does with the other connections that are in the pool at the time that a stale connection occurred. The application server can aggressively purge all connections from the pool (EntirePool option), or it can leave the others there (FailingConnectionOnly option), or it can check all connections in the pool before allowing them to be handed out (ValidateAllConnections option).
Note that the property values above are for WebSphere Application Server Liberty. If using traditional, ValidateAllConnections is done as the combination of FailingConectionOnly plus defaultPretestOptimizationOverride=true.
We are getting this error sporadically. With the same TNS, we are able to make proper connections to the database. But we see this in the logs while make connections some times. Following is the stack trace. This is db connection to Oracle from a Linux machine and java application Any help is appreciated.
java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:439)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:165)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:297)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:221)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:157)
at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:94)
at oracle.jdbc.pool.OracleImplicitConnectionCache.makeCacheConnection(OracleImplicitConnectionCache.java:1567)
at oracle.jdbc.pool.OracleImplicitConnectionCache.getCacheConnection(OracleImplicitConnectionCache.java:478)
at oracle.jdbc.pool.OracleImplicitConnectionCache.getConnection(OracleImplicitConnectionCache.java:347)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:404)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:189)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:165)
try following
(obvious) IP address is incorrect - try PING
The port is not open, or is blocked by a firewall - try TELNET
The DB listener is not running or is binding to a different network
interface - again, TELNET should confirm this (also use Oracle client
tools to connect)
No local ports are available for the out-going connection
(unlikely) - only if you're making thousands of connections, or
creating hundreds of new connections every minute.
Seems the connection pool runs out of connections...
When DBMS listener's incoming request buffer is overloaded by many simultaneous connection requests. It will fail some of them.
you can have the thread sleep a bit (half-second to a second or so) between successive connection requests. After that, don't close connections until they're broken. Keep them and re-use them.
Check https://forums.oracle.com/forums/thread.jspa?messageID=2540479, maybe you must change listener.ora's file host parameter to your host parameter. You can check that what parameter is your hostname in windows, cmd>hostname
I have a question related to the scenario when connecting from a Java application using the Microsoft JDBC Driver 4.0 to a SQL Server 2014 with AlwaysOn Availability Groups set up for high availability.
With this set up, we will be connecting to an availability group listener (specified in the db connecting string instead of any particular instance), so that the DB fail-over etc. is handled gracefully by the listener and it tries to connect to the next available instance behind the scenes if current primary goes down in the AG cluster.
Question(s) I have is,
In the data-source that is configured on the j2ee application server side (we use WebSphere), what happens to those connections already pooled by the data-source?
When a database goes down, though the AG listener would try to reconnect on the db side to the next available DB, will the AG Listener also through the jdbc driver send an event or something to the data-source created on the app server and make sure those connections that are already pooled by the datasource to be discarded and have it create new ones so that transactions on the application side wont fail (though they might for a while till new connections are created and fail over is successful) or the java application has to find out only after requesting it from the datasource?
WebSphere Application Server is able to cope with bad connections and removes them from the pool. Exactly when this happens depends on some configurable options and on how fully the Microsoft JDBC driver takes advantage of the javax.sql.ConnectionEventListener API to send notifications to the application server. In the ideal case where a JDBC driver sends the connectionErrorOccurred event immediately for all connections, WebSphere Application Server responds by removing all of these connections from the pool and by marking any connection that is currently in-use as bad so that it does not get returned to the pool once the application closes the handle. Lacking this, WebSphere Application Server will discover the first bad connection upon next use by the application. It is discovered either by a connectionErrorOcurred event that is sent by the JDBC driver at that time, or lacking that, upon inspecting the SQLState/error code of an exception for known indicators of bad connections. WebSphere Application Server then goes about purging bad connections from the pool according to the configured Purge Policy. There are 3 options:
Purge Policy of Entire Pool - all connections are removed from
the pool and in-use connections marked as bad so that they are not
pooled.
Purge Policy of Failing Connection Only - only the
specific connection upon which the error actually occurred is
removed from the pool or marked as bad and not returned to the pool
Purge Policy of Validate All Connections - all connections are
tested for validity (Connection.isValid API) and connections found
to be bad are removed from the pool or marked as bad and not
returned to the pool. Connections found to be valid remain in the
pool and continue to be used.
I'm not sure from your description if you are using WebSphere Application Server traditional or Liberty. If traditional, there is an additional option for pre-testing connections as they are handed out of the pool, but be aware that turning this on can have performance implications.
That said, the one thing to be aware of is that regardless of any of the above, your application will always need to be capable of handling the possibility of errors due to bad connections (even if the connection pool is cleared, connections can go bad while in use) and respond by requesting a new connection and retrying the operation in a new transaction.
Version 4 of that SQL Server JDBC driver is old and doesn't know anything about the always on feature.
Any data source connection pool can be configured to check the status of the connection from the pool prior to doling it out to the client. If the connection cannot be used the pool will create a new one. That's true of all vendors and versions. I believe that's the best you can do.
In my application I use connection to Oracle, when connection lost and I try to re-connect I receive exception:
java.sql.SQLException: Io exception: Broken pipe
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:124)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:161)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:273)
at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:540)
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:264)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:196)
For recover I need to restart application, does it possible recover without restart?
Thanks.
Followings could be the possibilities which could be causing the exception:
Network problem: That is the network between the database and application server causing the physical connection to be dropped after a period of time. It's probably due to a firewall running behind the network which is configured to kill db connections after a specified period of time. You may consider a workaround to maintain the connection alive all the time simply by re-configuring your application server. For Tomcat, you may try adding validationQuery="select 'validationQuery' from dual in the Tomcat datasource conf file (context.xml)
The connections to the database server are being reset and the client is not notified by the database driver. The problem in this case is that the Oracle driver is discovering that it's socket to the DBMS somehow (firewall again, maybe?) has been closed by the other end. You may consider setting your connection timeout (in the pool) shorter than the network/DB server timeout as a solution.