We Keep Coding

Queries take too long after MySQL fail-over

I have an application that uses MySQL fail-over via a connection url:
jdbc:mysql://172.17.0.4:3306,172.17.0.3:3306/db_name?autoReconnect=true&failOverReadOnly=false
Now when primary db is down then it should move to secondary connection and application flow should work as expected.
Now the problem happens when moving to secondary db. It takes too long to move/execute the queries, causing the flow to take much longer than expected.
I have checked already with db and there's no slow query issues. I guess it's something to do with fail-over and with checking the states. So, any idea what might be causing this issue or how to resolve this delay?
I am also using c3p0 to manage the connections pools. Already tried with initialTimeout and maxReconnects but no luck so far.
DataSource
<bean id="productionDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${jdbc.driver}"/>
<property name="jdbcUrl" value="${jdbc.url}"/>
<property name="user" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
<property name="description" value="integration_ds"/>
<!-- configuration pool via c3p0-->
<property name="acquireIncrement" value="${datasource.acquireIncrement}"/>
<property name="idleConnectionTestPeriod" value="${datasource.idleConnectionTestPeriod}"/>
<!-- seconds -->
<property name="maxPoolSize" value="${datasource.maxPoolSize}"/>
<property name="maxStatements" value="${datasource.maxStatements}"/>
<property name="maxStatementsPerConnection" value="${datasource.maxStatementsPerConnection}"/>
<property name="minPoolSize" value="${datasource.minPoolSize}"/>
<property name="initialPoolSize" value="${datasource.initialPoolSize}"/>
<property name="maxIdleTime" value="${datasource.maxIdleTime}"/>
<property name="acquireRetryAttempts" value="${datasource.acquireRetryAttempts}"/>
<property name="acquireRetryDelay" value="${datasource.acquireRetryDelay}"/>
<property name="breakAfterAcquireFailure" value="${datasource.breakAfterAcquireFailure}"/>
<property name="debugUnreturnedConnectionStackTraces" value="true"/>
</bean>
Properties
datasource.acquireIncrement=1
datasource.idleConnectionTestPeriod=1000
datasource.maxPoolSize=10
datasource.maxStatements=600
datasource.minPoolSize=5
datasource.initialPoolSize=5
datasource.maxIdleTime=7200
#datasource.acquireRetryAttempts=5
datasource.acquireRetryAttempts=1
#datasource.acquireRetryDelay=5000
#datasource.acquireRetryDelay=1000
datasource.acquireRetryDelay=100
datasource.breakAfterAcquireFailure=false
datasource.maxStatementsPerConnection=3
datasource.checkoutTimeout=100
DAO
private static final String findByAppIdHql = "select app from AppImpl app where app.appId = ?";
final Query query = sf.getCurrentSession().createQuery(findByAppIdHql).setString(0, appId);
query.setCacheable(true);
query.setCacheRegion("app_query_cache");
query.setCacheMode(CacheMode.NORMAL);
List<App> apps = query.list();

Issue with CachingConnectionFactory and DefaultMessageListenerContainer

It is mentioned in the documentation of DefaultMessageListenerContainer class that it is not recommended to use CachingConnectionFactory with dynamic scaling. While searching, I have encountered following link:
Why DefaultMessageListenerContainer should not use CachingConnectionFactory?
Here found a comment from Gary Russell that
the problem is with caching consumers when using variable concurrency in the container; we can end up with a live consumer "stuck" in the cache".
We have used DefaultMessageListenerContainer and CachingConnectionFactory together so this is surely a problem from above link.
We are encountering problems with our application having following behaviour:
TCP ZeroWindow network congestion
TCP RESET from application server to MQ
DB connection grows during the issue while different transactions halt
Messages in certain queues gets built up
We have following code configuration :
In ibmmq-context.xml file:
<!-- WebSphere MQ Connection Factory -->
<bean id="appMqConnectionFactory" class="com.ibm.mq.jms.MQConnectionFactory">
<property name="hostName">
<value>${ibmmq.ip}</value>
</property>
<property name="port">
<value>${ibmmq.port}</value>
</property>
<property name="queueManager">
<value>${ibmmq.queuemanager}</value>
</property>
<property name="channel">
<value>${ibmmq.channel}</value>
</property>
<property name="clientReconnectOptions">
<util:constant static-field="com.ibm.msg.client.wmq.WMQConstants.WMQ_CLIENT_RECONNECT"/>
</property>
<property name="transportType" ref="appTransport"/>
</bean>
<!-- A cached connection -->
<bean id="appCachedConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="appMqConnectionFactory"/>
<property name="sessionCacheSize" value="${jms.session.cachesize}"/>
</bean>
<!-- Use native MQ classes. -->
<bean id="appTransport" class="org.springframework.beans.factory.config.FieldRetrievingFactoryBean">
<property name="staticField">
<value>com.ibm.mq.jms.JMSC.MQJMS_TP_CLIENT_MQ_TCPIP</value>
</property>
</bean>
In jms-context file:
<bean id="bankListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="cachedConnectionFactory" />
<property name="destination" ref="transactionResponseDestination" />
<property name="messageListener" ref="thirdpartyService" />
<property name="autoStartup" value="false"/>
<property name="taskExecutor" ref="listenerExecutor"/>
<property name="concurrency" value="20-30"/>
</bean>
There are 6 such listeners like bankListener and each of the listeners has concurrency value, varies from 10-40
<bean id="listenerExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
<property name="maxPoolSize" value="140"/>
<property name="corePoolSize" value="100"/>
<property name="queueCapacity" value="30"/>
<property name="threadNamePrefix" value="jms-listener-task-"/>
<property name="threadGroupName" value="jms-listener-tasks"/>
</bean>
and jms-context.xml file uses ibmmq-context.xml file.
And to note, we have used IBM MQ 7.1, Spring 4.2.8, spring-integration-core as 4.3.1.RELEASE and JBoss EAP 6.4.10
We are planning to fix this by following way:
<bean id="bankListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="appMqConnectionFactory" />
<property name="destination" ref="transactionResponseDestination" />
<property name="messageListener" ref="thirdpartyService" />
<property name="autoStartup" value="false"/>
<property name="taskExecutor" ref="listenerExecutor"/>
<property name="concurrency" value="20-30"/>
</bean>
My request:
Please review the configuration and let me know is there anything else to be changed.
Could you please also explain our application behaviour(above 4 points - a to d) with our current configuration with CachingConnectionFactory and DefaultMessageListenerContainer
Thanks in advance for your help.

Try setting:
cachingConnectionFactory.setCacheConsumers(false);
Or in Spring it should be:
<bean id="appCachedConnectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
...
<property name="cacheConsumers" value="false"/>
...
</bean>

Hibernate: why c3p0 timeout not working?

I work on web java application using hibernate, and i always get memory low errors for tons of java inactive sessions on my oracle db. i tried to closing this unused sessions with c3p0 timeout config but it's not working at all.
this is my hibernate config and my open sessions after several minutes:
(the commented code below is for when i tried using common dbcp refer to one of answers for similar questions and timeout times set to 30 seconds for test.)
<property name="hibernate.dialect">org.hibernate.dialect.OracleDialect</property>
<property name="hibernate.connection.driver_class">oracle.jdbc.OracleDriver</property>
<property name="hibernate.connection.url">jdbc:oracle:thin:#localhost:test</property>
<property name="hibernate.connection.username">tams_test</property>
<property name="hibernate.connection.password">test1234</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.connection.isolation">2</property>
<property name="org.hibernate.flushMode">COMMIT</property>
<property name="hibernate.cache.use_query_cache">false</property>
<property name="hibernate.transaction.auto_close_session">true</property>
<property name="hibernate.order_updates">true</property>
<property name="hibernate.order_inserts">true</property>
<!-- <property name="hibernate.dbcp.initialSize">3</property>
<property name="hibernate.dbcp.maxIdle">50</property>
<property name="hibernate.dbcp.minIdle">0</property>
<property name="hibernate.dbcp.maxWait">180000</property>
<property name="hibernate.dbcp.maxConnLifetimeMillis">30000</property>
<property name="hibernate.dbcp.defaultQueryTimeout">1</property>-->
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_size">150</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.timeout">30</property>
<property name="hibernate.c3p0.idle_test_period">30</property>
<property name="hibernate.c3p0.idleConnectionTestPeriod">30</property>
<property name="c3p0.testConnectionOnCheckout">true</property>
<property name="testConnectionOnCheckin">true</property>
<property name="c3p0.maxConnectionAge">30</property>
<property name="maxIdleTimeExcessConnections">30</property>
<property name="connection.provider_class">
org.hibernate.connection.C3P0ConnectionProvider
</property>
open sessions on oracle db after several minutes

I faced your problem too, and hibernate didn't want to apply my parameters of c3p0, so I changed my datasource and it completly solved my problem of inactive sessions.
Try to setup your dataSource bean like this:
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="oracle.jdbc.driver.OracleDriver"/>
<property name="jdbcUrl" value="jdbc:oracle:thin:#xxxx:1521:xxx"/>
<property name="user" value="xx"/>
<property name="password" value="xxxx"/>
<property name="maxStatements" value="1000"/>
<property name="maxIdleTime" value="30"/>
<property name="maxPoolSize" value="100"/>
<property name="minPoolSize" value="10"/>
<property name="initialPoolSize" value="10"/>
<property name="idleConnectionTestPeriod" value="20"/>
</bean>

jdbc connection string with c3p0

I have jdbc configured with c3p0.
However, I am worried about possible conflicts, since some of the parameters in jdbc string and c3p0 are similar.
Here is my jdbc string:
jdbc:mysql://1.1.1.1:3306/db?useSSL=true&requireSSL=true&connectTimeout=15000&socketTimeout=30000&autoReconnect=true
We decided to include connectTimeout, socketTimeout, autoReconnect
because otherwise it took too long to switch to the replica if master crashes. ( We are using MySQL RDS Multi-AZ ).
Here are my c3p0 properties:
<property name="acquireIncrement" value="3"/>
<property name="minPoolSize" value="5"/>
<property name="maxPoolSize" value="10"/>
<property name="maxIdleTime" value="3600"/>
<!-- 1 hour -->
<property name="maxConnectionAge" value="7200"/>
<property name="maxIdleTimeExcessConnections" value="600"/>
<property name="idleConnectionTestPeriod" value="180"/>
<property name="testConnectionOnCheckin" value="true"/>
<property name="testConnectionOnCheckout" value="false"/>
Any ideas / suggestions?
Maybe someone has production-ready configuration?
Best Regards,
Maksim

Configure spring datasource for hibernate and #Transactional

At this moment I'm using DriverManagerDataSource with #Transactional annotation to manage transactions. But all transactions are very very slow, probably because data source open and close connection to db each time.
What data source should I use to speed up transaction?

I am using in my application combination of two approaches. the first one is c3p0 connection pooling, its almost the same solution as chkal sugested. The second approach is to use Spring lazyConnectionDataSourceProxy, which creates lazy loading proxy that loads connection only if you hit the database. This is very useful, when you have second level cache and you are only reading cached data and queries - database wont be hit, and you don't need to acquire connection (which is pretty expensive).
<bean name="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="driverClass" value="${jdbc.driverClassName}" />
<property name="jdbcUrl" value="${jdbc.url}" />
<property name="user" value="${jdbc.username}" />
<property name="password" value="${jdbc.password}" />
<!-- Pool properties -->
<property name="minPoolSize" value="5" />
<property name="initialPoolSize" value="10" />
<property name="maxPoolSize" value="50" />
<property name="maxStatements" value="50" />
<property name="idleConnectionTestPeriod" value="120" />
<property name="maxIdleTime" value="1200" />
</bean>
<bean name="lazyConnectionDataSourceProxy" class="org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy">
<property name="targetDataSource" ref="dataSource" />
</bean>

DriverManagerDataSource isn't actually a connection pool and should only be used for testing. You should try BasicDataSource from Apache Commons DBCP. Something like:
<bean id="dataSource" destroy-method="close"
class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="${jdbc.driverClassName}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
</bean>