from time to time I have the following error in my logs:
Error querying database. Cause: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
The error may exist in org.sonar.core.issue.db.ActionPlanMapper
The error may involve org.sonar.core.issue.db.ActionPlanMapper.findByKeys
The error occurred while executing a query
How should I adjust my connection pool settings so that this doesn't occur anymore?
sonar.jdbc.maxActive=25
sonar.jdbc.maxIdle=5
sonar.jdbc.minIdle=2
sonar.jdbc.maxWait=15000
sonar.jdbc.minEvictableIdleTimeMillis=600000
sonar.jdbc.timeBetweenEvictionRunsMillis=30000
According to http://jira.codehaus.org/browse/SONAR-188, you can try to update your connection settings in file: conf/sonar.properties
with adding "autoreconnect".
Example:
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8?autoReconnect=true
Related
my java code is like:
logger.info("start");
getJdbcTemplate().execute("call " + procedureName + "()");
and I got the exception:
org.springframework.dao.DataAccessResourceFailureException: StatementCallback; SQL [call PRMI_UPDATE_USER_LOGIN_INFO()]; Io ERROR: Connection reset; nested exception is java.sql.SQLException: Io ERROR: Connection reset
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:257)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:407)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:428)
Maybe it's caused by the long time waiting. I found that it printed "start" in log and after about 5 minutes I got the exception.
update at 2013-03-13:
I got that exception not only at calling oracle stored procedure but at druid's 'JdbcUtil.close(...)':
com.alibaba.druid.util.JdbcUtils.close:81 - close connection error
java.sql.SQLRecoverableException: Io Error: Connection reset
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:263)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:521)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:500)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3509)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:167)
at com.alibaba.druid.filter.stat.StatFilter.connection_close(StatFilter.java:254)
at com.alibaba.druid.filter.FilterChainImpl.connection_close(FilterChainImpl.java:163)
at com.alibaba.druid.proxy.jdbc.ConnectionProxyImpl.close(ConnectionProxyImpl.java:115)
at com.alibaba.druid.util.JdbcUtils.close(JdbcUtils.java:79)
at com.alibaba.druid.pool.DruidDataSource.shrink(DruidDataSource.java:1876)
at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:1694)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:150)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:180)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:169)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1034)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1010)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:97)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:487)
The druid's JdbcUtil.close method is quite simple:
public static void close(Connection x) {
if (x == null) {
return;
}
try {
x.close();
} catch (Exception e) {
LOG.debug("close connection error", e);
}
}
the source code is :
https://github.com/alibaba/druid/blob/master/src/main/java/com/alibaba/druid/util/JdbcUtils.java
It should wait as long as it is needed. Forget about various hacks which try to "detect" deadlock based on timeout delay.
you should find also some ORA-XXXX error. Io ERROR: Connection reset does not look like Oracle error message, there should be some error number attached to it
the timeout 5 minutes is very strange value. Theoretically this can be setup also on database side. As profile parameter CPU_PER_CALL but in such a case you should get an error: ORA-02393: exceeded call limit on CPU usage. And you connection should NOT be lost
theoretically you can also have problems which dead connection detection, but 5 minutes timeout is too short for that
another possible source can be ORA-600 error. Oracle internal error, maybe your session process crashed and therefore TCP connection was lost
you should contact your local DBAs and ask then for cooperation. They should help you better than anonymous people on the Internet forum.
Maybe it's caused by the long time waiting
No it is not caused due to that
As Java Doc says about DataAccessResourceFailureException
Data access exception thrown when a resource fails completely: for
example, if we can't connect to a database using JDBC.
After running my application, i am getting this error after around 5 mins.
Even though i am returning the resource after use, i keep getting this.
I have built jedis-2.2.2-SNAPSHOT.jar from the jedis code base, since its not released yet
I had set the minIdle = 100, maxIdle=200 & maxActive=200. At the time of this exception, the connection count to redis was 122 from my application
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:42)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
at redis.clients.util.Pool.getResource(Pool.java:40)
... 6 more
Did you check that redis is still up & running ?
If not, investigate why it died.
try a redis-cli in a terminal if you can. "info" would give you more details.
I am getting sporadic errors from a java service that is listening to OracleAQ.
It seems to be happening each night, and I can't be sure what is going on. Could it really be a database connection problem ?
Or does the "Dequeue failed" suggest that it was connected and something else happened ?
Here is the exception below :
[2013-11-04 18:16:16,508] WARN org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'MYCOMPANY_INFO_QUEUE' - trying to recover. Cause: JMS-120: Dequeue failed; nested exception is java.sql.SQLException: Io exception: Socket read timed out
oracle.jms.AQjmsException: JMS-120: Dequeue failed
at oracle.jms.AQjmsError.throwEx(AQjmsError.java:311)
at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:2234)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:1028)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:951)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:929)
at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:781)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985)
at java.lang.Thread.run(Thread.java:662)
[Linked-exception]
java.sql.SQLException: Io exception: Socket read timed out
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:976)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329)
at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:1732)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:1028)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:951)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:929)
at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:781)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985)
at java.lang.Thread.run(Thread.java:662)
[2013-11-04 18:16:16,569] INFO org.springframework.jms.listener.DefaultMessageListenerContainer - Successfully refreshed JMS Connection
The jms receive timeout should be configured in seconds (while the db timeout is in milliseconds). So make sure your jms value is less. For example, here is my working spring config:
<bean id="xxxJmsTemplate" class="org.springframework.jms.core.JmsTemplate102">
<property name="connectionFactory" ref="xxxJmsConnectionFactory"/>
<property name="defaultDestinationName" value="some_queue"/>
<property name="receiveTimeout" value="10"/><!-- seconds -->
</bean>
PS: The special Spring constant RECEIVE_TIMEOUT_NO_WAIT value of -1 does not seem to work for this setting. But setting a reasonably short time in seconds should do the trick.
I suggest looking at your dequeue options for wait time.
import oracle.AQ.AQDequeueOption;
...
AQDequeueOption options = null;
options = new AQDequeueOption();
options.setWaitTime(AQDequeue.WAIT_NONE);
//WAIT_NONE = do not wait if messages are not available
//WAIT_FOREVER = waits "forever"; default value
...
The WAIT_FOREVER setting is default and will wait until a message is available on the queue; however this holds the database connection open.
I believe this is the reason why you experience the errors "sporadically"; most of the time messages are being enqueued and running smoothly; and then when messages are not enqueued (each night) your database connection is timing out.
There is on error that appears in my log file every time:
com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: Socket closed
STACKTRACE:
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1994)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3250)
at com.mysql.jdbc.Connection.execSQL(Connection.java:3179)
at com.mysql.jdbc.Statement.executeQuery(Statement.java:1207)
at com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
It happens always in the same point of the code, when the application does a specific query in the MySQL database.
The query is LIKE THIS:
SELECT M.*, O.id
FROM order_message M
INNER JOIN orders O ON M.order_id = O.id
WHERE O.seller_id = 14224 AND M.sender_user_id <> 14224
ORDER BY M.creation_date DESC
LIMIT 5
I noticed (by EXPLAINING this query) that it always use temporary/filesort to execute. All indexes are properly created and I think these is no way to improve this, but I suspect the query performance or resource utilization is causing the exception error.
I am using amazon RDS
The problem was that my connection pool was configured in a way that any database connection that took longer than 10 seconds would be dropped by the connection pool (c3p0). I was using the unreturnedConnectionTimeout parameter.
The c3p0 documentation page discourage to use this parameter. Ideally, all connections should be properly closed (and thus returned to the pool)
http://www.mchange.com/projects/c3p0/#unreturnedConnectionTimeout
I have increased the parameter to 60 seconds and the problem get solved.
I am using DB2 with JDBC and the below code throws SQLException when I try to run for the first time after I left the connection idle for a few minutes. From the second time onward it would work exactly the way it has to.
ResultSet.next()
Any ideas on what would cause the exception?
Exception trace:
FFDC Exception:com.ibm.db2.jcc.am.lo SourceId:com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next
ProbeId:2624 Reporter:
com.ibm.ws.rsadapter.jdbc.WSJccResultSet#b080b08
com.ibm.db2.jcc.am.lo: The current transaction was rolled back because of error "-30108".. SQLCODE=-1476, SQLSTATE=40506, DRIVER=3.57.110
at com.ibm.db2.jcc.am.bd.a(bd.java:663)
at com.ibm.db2.jcc.am.bd.a(bd.java:60)
at com.ibm.db2.jcc.am.bd.a(bd.java:127)
at com.ibm.db2.jcc.am.am.b(am.java:3760)
at com.ibm.db2.jcc.t4.eb.h(eb.java:278)
at com.ibm.db2.jcc.t4.eb.a(eb.java:239)
at com.ibm.db2.jcc.t4.eb.c(eb.java:31)
at com.ibm.db2.jcc.t4.u.a(u.java:32)
at com.ibm.db2.jcc.t4.j.Zb(j.java:259)
at com.ibm.db2.jcc.am.am.X(am.java:3554)
at com.ibm.db2.jcc.t4.d.f(d.java:1881)
at com.ibm.db2.jcc.am.gc.a(gc.java:200)
at com.ibm.db2.jcc.t4.d.a(d.java:109)
at com.ibm.db2.jcc.am.am.c(am.java:366)
at com.ibm.db2.jcc.am.am.next(am.java:293)
at com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next(WSJdbcResultSet.java:3120)
The "outer" SQL Code, -1476, indicates that the transaction was rolled back because of the "inner" SQL Code, -30108, which says:
A connection failed but has been re-established. Special register
settings might have been replayed. Host name or IP address of the new
connection: host-name. Service name or port number of the new
connection: service-name. Reason code: reason-code.
I would guess that your connection timed out, and it dropped.