We Keep Coding

All the threads are waiting on some barrier condition

I have class like this.
class EmployeeTask implements Runnable {
public void run(){
PayDAO payDAO = new payDAO(session);
String newSalary= payDAO.getSalary(empid);
String newTitle= payDAO.getTitle(empid);
EmployeeDAO employeeDAO = new EmployeeDAO(session);
List<Employee> employees = employeeDAO.getEmployees();
employees.parallelStream().foreach(employee-> employeeDAO.add(newSalary, newTitle,employee));
}
}
When I run the above code for one thread, it completes DB operation in 1 second and returns.
When I run it using parallelStream(), it will submit 8 requests, all 8 threads will wait for eight seconds and then return. Which means the server is executing the requests sequentially instead of parallel.
I checked the java jetty logs
Theread-1 Insert …
Theread-2 Insert …
…
Theread-8 Insert …
After eight seconds, that is 1 second per request
Theread-1 Insert … <= update 1
Theread-2 Insert … <= update 1
…
Theread-8 Insert … <= update 1
the same thing continues.
This clearly tells, all the threads are getting blocked on one single resource, either the Datasource from my client java has only one connection so all the eight threads are getting blocked or the server is executing the requests one after the other.
I checked MaxPooledConnections — gives 20, MaxPooledConnections
PerNode - gives 5 default values.
I think the Vertica DB server is fine, maybe client is not having DatasourceConnection pooling, How do I enable Java side DatasourceConnection pooling. Any Code examples?
currently I am using mybatis ORM, with following datasource
<environments default=“XX”>
<environment id=“XX”>
<transactionManager type=“JDBC”/>
<dataSource type="POOLED">
<property name="driver" value="com.vertica.jdbc.Driver"/>
<property name="url" value="jdbc:vertica://host:port/schema”/>
<property name="username" value=“userName”/>
<property name="password" value=“password”/>
<property name="poolMaximumActiveConnections" value=“50”/>
<property name="poolMaximumIdleConnections" value=“10”/>
</dataSource>
</environment>
</environments>
What is the barrier condition all the threads are waiting on, my guess is Datasource connection. Any help is appreciated
Thanks.

Acquired Connections grows abruptly leading to server crash

I have a Java application that schedules a cron job after every 1 min. It runs on Glassfish 4. We are using Hibernate with JTA Entity Manager which is container managed for executing the queries on SQL Server database.
JDBC Connection Pool Settings are:
Initial and Minimum Pool Size:16
Maximum Pool Size:64
Pool Resize Quantity:4
Idle Timeout:300
Max Wait Time:60000
JDBC Connection Pool Statistics after 22 Hours run:
NumConnUsed 0count
NumConnAcquired 14404count
NumConnReleased 14404count
NumConnCreated 16count
NumConnFree 16count
The number of acquired connections keeps on incrementing and the Glassfish 4 crashes after around 10 days with below exception.
RAR5117 : Failed to obtain/create connection from connection pool [ com.beonic.tiv5 ]. Reason : com.sun.appserv.connectors.internal.api.PoolingException: java.lang.RuntimeException: Got exception during XAResource.start:
Please suggest how to avoid Glassfish crash.

finally
{
em = null;
ic = null;
}
I think here is the problem you are never commiting or closing the transacction
Giving this example and documentation of JTA check 5.2.2
// BMT idiom
#Resource public UserTransaction utx;
#Resource public EntityManagerFactory factory;
public void doBusiness() {
EntityManager em = factory.createEntityManager();
try {
// do some work
...
utx.commit();
}
catch (RuntimeException e) {
if (utx != null) utx.rollback();
throw e; // or display error message
}
finally {
em.close();
}
This is the correct way of doing a transacction. But you are only nulling the values and nothing more, that's why you your pools and not being closed
Here is more documentation about Transactions

It's hard to tell what is the real cause of the problem, but the problem might be that all your connections have become stale because not used for a long time.
It is a good practice to set up connection validation, which ensures that connections are reopened when closed by the external server.
There is a thorough article about connection pools in Glassfish/Payara, checkout especially the section about Connection validation (using Derby DB in the example):
To turn on connection validation :
asadmin set
resources.jdbc-connection-pool.test-pool.connection-validation-method=custom-validation
asadmin set
resources.jdbc-connection-pool.test-pool.validation-classname=
org.glassfish.api.jdbc.validation.DerbyConnectionValidation
asadmin
set
resources.jdbc-connection-pool.test-pool.is-connection-validation-required=true

Jedis Get Data: JedisConnectionFailureException iterating a section of code over long period of time

So I have a code that gets value from Redis using Jedis Client. But at a time, the Redis was at maximum connection and these exceptions were getting thrown:
org.springframework.data.redis.RedisConnectionFailureException
Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229)
...
org.springframework.data.redis.RedisConnectionFailureException
java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36)
...
org.springframework.data.redis.RedisConnectionFailureException
Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229)
When I check an AppDynamics analysis of this scenario, I saw some iteration of some calls over a long period of time (1772 seconds). The calls are shown in the snips.
Can anyone explain what's happening here? And why Jedis didn't stop after the Timeout setting (500ms)? Can I prevent this from happening for long?
This is what my Bean definitions for the Jedis look like:
<bean id="jedisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory" p:host-name="100.90.80.70" p:port="6382" p:timeout="500" p:use-pool="true" p:poolConfig-ref="jedisPoolConfig" p:database="3" />
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="1000" />
<property name="maxIdle" value="10" />
<property name="maxWaitMillis" value="500" />
<property name="testOnBorrow" value="true" />
<property name="testOnReturn" value="true" />
<property name="testWhileIdle" value="true" />
<property name="numTestsPerEvictionRun" value="10" />
</bean>

I'm not familiar with the AppDynamics output. I assume that's a cumulative view of Threads and their sleep times. So Threads get reused and so the sleep times add up. In some cases, a Thread gets a connection directly, without any waiting and in another call the Thread has to wait until the connection can be provided. The wait duration depends on when a connection becomes available, or the wait limit is hit.
Let's have a practical example:
Your screenshot shows a Thread, which waited 172ms. Assuming the sleep is only called within the Jedis/Redis invocation path, the Thread waited 172ms in total to get a connection.
Another Thread, which waited 530ms looks to me as if the first attempt to get a connection wasn't successful (which explains the first 500ms) and on a second attempt, it had to wait for 30ms. It could also be that it waited 2x for 265ms.
Sidenote:
1000+ connections could severely limit scalability. Spring Data Redis also supports other drivers which don't require pooling but work with fewer connections (see Spring Data Redis Connectors and here).

Too many connections using c3p0 and hibernate in MySql

I'm using c3p0 for my connection pooling. I've configured min connections as 100 and max size as 2000. I'm just writing a simple insert program to check how many connections are active in workbench. But, I'm getting the following error
java.sql.SQLException: Data source rejected establishment of connection, message from server: "Too many connections"
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:650)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1808)
at com.mysql.jdbc.Connection.<init>(Connection.java:452)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:411)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
My Hibernate.cfg.xml is as follows
<!-- c3p0 Connection pool config -->
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">100</property>
<property name="hibernate.c3p0.max_size">2000</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<property name="hibernate.c3p0.validate">true</property>
My Java program is
public static void main(String[] args) {
// TODO Auto-generated method stub
Transaction tx = null;
SessionFactory factory = HibernateUtil.getSessionFactory();
Session session = factory.openSession();
tx = session.beginTransaction();
System.setProperty("net.sf.ehcache.skipUpdateCheck", "true");
Employee e = new Employee(2,"Richard");
session.save(e);
try {
Thread.sleep(20000);
} catch (Exception e2) {
e2.printStackTrace();
}
tx.commit();
session.close();
System.out.println("Great! Student was saved");
}
It works fine when the min size is 5 and max size is 20. Do I need to do any changes in MySQL workbench?

The whole purpose of a database connection pool (like c3p0) is to optimise the use of resources versus the database.
If you had 6000 users with 6000 connections, you would quickly exhaust the available connections and errors would result.
Instead, a connection pool allows your application to "borrow" database connections from the pool, and return them after use.
So even though you have potentially 6000 users, the moments of time when multiple users are doing operations that operate versus the database concurrently at that moment in time would be a small fraction of that.
I would suggest to try this as a more reasonable value:
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">100</property>
After setting it up like this, I would run the application and watch the number of connections versus the database. If the 100 connections are at any time exhausted, you could look at tuning it upwards.
But I suspect that 100 concurrent connections will be enough - remember that they are "borrowed" from the c3p0 connection pool.
Documentation: What is c3p0? (connection pooling)

What is causing this JMS error connecting to OracleAQ?

I am getting sporadic errors from a java service that is listening to OracleAQ.
It seems to be happening each night, and I can't be sure what is going on. Could it really be a database connection problem ?
Or does the "Dequeue failed" suggest that it was connected and something else happened ?
Here is the exception below :
[2013-11-04 18:16:16,508] WARN org.springframework.jms.listener.DefaultMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'MYCOMPANY_INFO_QUEUE' - trying to recover. Cause: JMS-120: Dequeue failed; nested exception is java.sql.SQLException: Io exception: Socket read timed out
oracle.jms.AQjmsException: JMS-120: Dequeue failed
at oracle.jms.AQjmsError.throwEx(AQjmsError.java:311)
at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:2234)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:1028)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:951)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:929)
at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:781)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985)
at java.lang.Thread.run(Thread.java:662)
[Linked-exception]
java.sql.SQLException: Io exception: Socket read timed out
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:976)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329)
at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:1732)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:1028)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:951)
at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:929)
at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:781)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985)
at java.lang.Thread.run(Thread.java:662)
[2013-11-04 18:16:16,569] INFO org.springframework.jms.listener.DefaultMessageListenerContainer - Successfully refreshed JMS Connection

The jms receive timeout should be configured in seconds (while the db timeout is in milliseconds). So make sure your jms value is less. For example, here is my working spring config:
<bean id="xxxJmsTemplate" class="org.springframework.jms.core.JmsTemplate102">
<property name="connectionFactory" ref="xxxJmsConnectionFactory"/>
<property name="defaultDestinationName" value="some_queue"/>
<property name="receiveTimeout" value="10"/><!-- seconds -->
</bean>
PS: The special Spring constant RECEIVE_TIMEOUT_NO_WAIT value of -1 does not seem to work for this setting. But setting a reasonably short time in seconds should do the trick.

I suggest looking at your dequeue options for wait time.
import oracle.AQ.AQDequeueOption;
...
AQDequeueOption options = null;
options = new AQDequeueOption();
options.setWaitTime(AQDequeue.WAIT_NONE);
//WAIT_NONE = do not wait if messages are not available
//WAIT_FOREVER = waits "forever"; default value
...
The WAIT_FOREVER setting is default and will wait until a message is available on the queue; however this holds the database connection open.
I believe this is the reason why you experience the errors "sporadically"; most of the time messages are being enqueued and running smoothly; and then when messages are not enqueued (each night) your database connection is timing out.