I have class like this.
class EmployeeTask implements Runnable {
public void run(){
PayDAO payDAO = new payDAO(session);
String newSalary= payDAO.getSalary(empid);
String newTitle= payDAO.getTitle(empid);
EmployeeDAO employeeDAO = new EmployeeDAO(session);
List<Employee> employees = employeeDAO.getEmployees();
employees.parallelStream().foreach(employee-> employeeDAO.add(newSalary, newTitle,employee));
}
}
When I run the above code for one thread, it completes DB operation in 1 second and returns.
When I run it using parallelStream(), it will submit 8 requests, all 8 threads will wait for eight seconds and then return. Which means the server is executing the requests sequentially instead of parallel.
I checked the java jetty logs
Theread-1 Insert …
Theread-2 Insert …
…
Theread-8 Insert …
After eight seconds, that is 1 second per request
Theread-1 Insert … <= update 1
Theread-2 Insert … <= update 1
…
Theread-8 Insert … <= update 1
the same thing continues.
This clearly tells, all the threads are getting blocked on one single resource, either the Datasource from my client java has only one connection so all the eight threads are getting blocked or the server is executing the requests one after the other.
I checked MaxPooledConnections — gives 20, MaxPooledConnections
PerNode - gives 5 default values.
I think the Vertica DB server is fine, maybe client is not having DatasourceConnection pooling, How do I enable Java side DatasourceConnection pooling. Any Code examples?
currently I am using mybatis ORM, with following datasource
<environments default=“XX”>
<environment id=“XX”>
<transactionManager type=“JDBC”/>
<dataSource type="POOLED">
<property name="driver" value="com.vertica.jdbc.Driver"/>
<property name="url" value="jdbc:vertica://host:port/schema”/>
<property name="username" value=“userName”/>
<property name="password" value=“password”/>
<property name="poolMaximumActiveConnections" value=“50”/>
<property name="poolMaximumIdleConnections" value=“10”/>
</dataSource>
</environment>
</environments>
What is the barrier condition all the threads are waiting on, my guess is Datasource connection. Any help is appreciated
Thanks.
Related
I am very new in c3p0 integration...
I have these settings with c3p0-0.9.5-pre5.jar, hibernate-c3p0-3.5.6-Final.jar, hibernate-core-3.5.6-Final.jar and mchange-commons-java-0.2.6.3.jar jars like below...
<property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.c3p0.min_size">1</property>
<property name="hibernate.c3p0.max_size">5</property>
<property name="hibernate.c3p0.timeout">40</property>
<property name="hibernate.c3p0.idle_test_period">30</property>
<!--<property name="hibernate.c3p0.max_statements">50</property>-->
<property name="hibernate.c3p0.maxStatementsPerConnection">5</property> <!--Instead of max_statements-->
<property name="hibernate.c3p0.validate">true</property>
<property name="hibernate.connection.pool_size">25</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<property name="hibernate.c3p0.automaticTestTable">con_test</property>
<property name="hibernate.c3p0.privilegeSpawnedThreads">true</property>
<property name="hibernate.c3p0.contextClassLoaderSource">library</property>
<property name="hibernate.c3p0.maxAdministrativeTaskTime">30</property>
<property name="hibernate.c3p0.numHelperThreads">20</property>
The problem is, application generates thousands of threads and keep those as Time waited Threads.
I have print some of those threads by a loop ..
"<br/>" + c++ +". "+ t.getState() + " (" + t.isAlive() + ") : " + t.getName();
results is below...
147. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#0
148. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#3
149. WAITING (true) : Reference Handler
150. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#2
151. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#8
152. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#19
153. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|3b0c81d2]-HelperThread-#17
154. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#3
155. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#1
156. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|70cef37d]-HelperThread-#19
This is increasing very first when retrieve data from database by application.
App developed by Java, Struts-1, Hibernate, Oracle(BD).
How can I remove/kill these threads
One way or some other, if you are seeing thousand of these Threads, you are leaking DataSources. That is, your application is constructing c3p0 DataSources, each of which has its own complement of Threads, then it is losing or dereferencing or replacing them without first close()ing them.
A pooled DataSource should be constructed once, placed somewhere with shared availability, and reused over and over again. If, unusually, a DataSource needs to be reconstructed for some reason, you need to close() c3p0 DataSources or their Threads will live forever.
Perhaps the most common error that leads to this sort of thing are applications that hot-redeploy. If on app initialization a DataSource is created, in a shutdown hook in the redeploy cycle you must take car that the same DataSource gets destroyed.
Note that in the list of Threads above, you show many Threads from different DataSources (as they have different identity tokens after the common VMID portion before the |). You are definitely creating but not close()ing lots of DataSources.
So I have a code that gets value from Redis using Jedis Client. But at a time, the Redis was at maximum connection and these exceptions were getting thrown:
org.springframework.data.redis.RedisConnectionFailureException
Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229)
...
org.springframework.data.redis.RedisConnectionFailureException
java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36)
...
org.springframework.data.redis.RedisConnectionFailureException
Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229)
When I check an AppDynamics analysis of this scenario, I saw some iteration of some calls over a long period of time (1772 seconds). The calls are shown in the snips.
Can anyone explain what's happening here? And why Jedis didn't stop after the Timeout setting (500ms)? Can I prevent this from happening for long?
This is what my Bean definitions for the Jedis look like:
<bean id="jedisConnectionFactory" class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory" p:host-name="100.90.80.70" p:port="6382" p:timeout="500" p:use-pool="true" p:poolConfig-ref="jedisPoolConfig" p:database="3" />
<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxTotal" value="1000" />
<property name="maxIdle" value="10" />
<property name="maxWaitMillis" value="500" />
<property name="testOnBorrow" value="true" />
<property name="testOnReturn" value="true" />
<property name="testWhileIdle" value="true" />
<property name="numTestsPerEvictionRun" value="10" />
</bean>
I'm not familiar with the AppDynamics output. I assume that's a cumulative view of Threads and their sleep times. So Threads get reused and so the sleep times add up. In some cases, a Thread gets a connection directly, without any waiting and in another call the Thread has to wait until the connection can be provided. The wait duration depends on when a connection becomes available, or the wait limit is hit.
Let's have a practical example:
Your screenshot shows a Thread, which waited 172ms. Assuming the sleep is only called within the Jedis/Redis invocation path, the Thread waited 172ms in total to get a connection.
Another Thread, which waited 530ms looks to me as if the first attempt to get a connection wasn't successful (which explains the first 500ms) and on a second attempt, it had to wait for 30ms. It could also be that it waited 2x for 265ms.
Sidenote:
1000+ connections could severely limit scalability. Spring Data Redis also supports other drivers which don't require pooling but work with fewer connections (see Spring Data Redis Connectors and here).
I'm using c3p0 for my connection pooling. I've configured min connections as 100 and max size as 2000. I'm just writing a simple insert program to check how many connections are active in workbench. But, I'm getting the following error
java.sql.SQLException: Data source rejected establishment of connection, message from server: "Too many connections"
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:650)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1808)
at com.mysql.jdbc.Connection.<init>(Connection.java:452)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:411)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
My Hibernate.cfg.xml is as follows
<!-- c3p0 Connection pool config -->
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">100</property>
<property name="hibernate.c3p0.max_size">2000</property>
<property name="hibernate.c3p0.timeout">300</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.idle_test_period">3000</property>
<property name="hibernate.c3p0.validate">true</property>
My Java program is
public static void main(String[] args) {
// TODO Auto-generated method stub
Transaction tx = null;
SessionFactory factory = HibernateUtil.getSessionFactory();
Session session = factory.openSession();
tx = session.beginTransaction();
System.setProperty("net.sf.ehcache.skipUpdateCheck", "true");
Employee e = new Employee(2,"Richard");
session.save(e);
try {
Thread.sleep(20000);
} catch (Exception e2) {
e2.printStackTrace();
}
tx.commit();
session.close();
System.out.println("Great! Student was saved");
}
It works fine when the min size is 5 and max size is 20. Do I need to do any changes in MySQL workbench?
The whole purpose of a database connection pool (like c3p0) is to optimise the use of resources versus the database.
If you had 6000 users with 6000 connections, you would quickly exhaust the available connections and errors would result.
Instead, a connection pool allows your application to "borrow" database connections from the pool, and return them after use.
So even though you have potentially 6000 users, the moments of time when multiple users are doing operations that operate versus the database concurrently at that moment in time would be a small fraction of that.
I would suggest to try this as a more reasonable value:
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">100</property>
After setting it up like this, I would run the application and watch the number of connections versus the database. If the 100 connections are at any time exhausted, you could look at tuning it upwards.
But I suspect that 100 concurrent connections will be enough - remember that they are "borrowed" from the c3p0 connection pool.
Documentation: What is c3p0? (connection pooling)
We are using Oracle as the database for our Web application. The application runs well most of the time, but we get this "No more data to read from socket" error.
Caused by: java.sql.SQLRecoverableException: No more data to read from socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1142)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1869)
at org.hibernate.loader.Loader.doQuery(Loader.java:718)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:270)
at org.hibernate.loader.Loader.doList(Loader.java:2449)
... 63 more
We use spring, hibernate and i have the following for the datasource in my applciation context file.
<bean class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close" id="dataSource">
<property name="driverClassName" value="${database.driverClassName}" />
<property name="url" value="${database.url}" />
<property name="username" value="${database.username}" />
<property name="password" value="${database.password}" />
<property name="defaultAutoCommit" value="false" />
<property name="initialSize" value="10" />
<property name="maxActive" value="30" />
<property name="validationQuery" value="select 1 from dual" />
<property name="testOnBorrow" value="true" />
<property name="testOnReturn" value="true" />
<property name="poolPreparedStatements" value="true" />
<property name="removeAbandoned" value="true" />
<property name="logAbandoned" value="true" />
</bean>
I am not sure whether this is because of application errors, database errors or network errors.
We see the following on the oracle logs
Thu Oct 20 10:29:44 2011
Errors in file d:\oracle\diag\rdbms\ads\ads\trace\ads_ora_3836.trc (incident=31653):
ORA-03137: TTC protocol internal error : [12333] [4] [195] [3] [] [] [] []
Incident details in: d:\oracle\diag\rdbms\ads\ads\incident\incdir_31653\ads_ora_3836_i31653.trc
Thu Oct 20 10:29:45 2011
Trace dumping is performing id=[cdmp_20111020102945]
Thu Oct 20 10:29:49 2011
Sweep [inc][31653]: completed
Sweep [inc2][31653]: completed
Thu Oct 20 10:34:20 2011
Errors in file d:\oracle\diag\rdbms\ads\ads\trace\ads_ora_860.trc (incident=31645):
ORA-03137: TTC protocol internal error : [12333] [4] [195] [3] [] [] [] []
Incident details in: d:\oracle\diag\rdbms\ads\ads\incident\incdir_31645\ads_ora_860_i31645.trc
Thu Oct 20 10:34:21 2011
Oracle Version : 11.2.0.1.0
For errors like this you should involve oracle support. Unfortunately you do not mention what oracle release you are using. The error can be related to optimizer bind peeking. Depending on the oracle version different workarounds apply.
You have two ways to address this:
upgrade to 11.2
set oracle parameter _optim_peek_user_binds = false
Of course underscore parameters should only be set if advised by oracle support
We were facing same problem, we resolved it by increasing initialSize and maxActive size of connection pool.
You can check this link
Maybe this helps someone.
Another case: If you are sending date parameters to a parameterized sql, make sure you sent java.sql.Timestamp and not java.util.Date. Otherwise you get
java.sql.SQLRecoverableException: No more data to read from socket
Example statement:
In our java code, we are using org.apache.commons.dbutils and we have the following:
final String sqlStatement = "select x from person where date_of_birth between ? and ?";
java.util.Date dtFrom = new Date(); //<-- this will fail
java.util.Date dtTo = new Date(); //<-- this will fail
Object[] params = new Object[]{ dtFrom , dtTo };
final List mapList = (List) query.query(conn, sqlStatement, new MapListHandler(),params);
The above was failing until we changed the date parameters to be java.sql.Timestamp
java.sql.Timestamp tFrom = new java.sql.Timestamp (dtFrom.getTime()); //<-- this is OK
java.sql.Timestamp tTo = new java.sql.Timestamp(dtTo.getTime()); //<-- this is OK
Object[] params = new Object[]{ tFrom , tTo };
final List mapList = (List) query.query(conn, sqlStatement, new MapListHandler(),params);
This is a very low-level exception, which is ORA-17410.
It may happen for several reasons:
A temporary problem with networking.
Wrong JDBC driver version.
Some issues with a special data structure (on database side).
Database bug.
In my case, it was a bug we hit on the database, which needs to be patched.
Try two things:
Set in $ORACLE_HOME/network/admin/tnsnames.ora on the oracle server server=dedicated to server=shared to allow more than one connection at a time. Restart oracle.
If you are using Java this might help you: In java/jdk1.6.0_31/jre/lib/security/Java.security change securerandom.source=file:/dev/urandom to securerandom.source=file:///dev/urandom
I had the same problem. I was able to solve the problem from application side, under the following scenario:
JDK8, spring framework 4.2.4.RELEASE, apache tomcat 7.0.63, Oracle Database 11g Enterprise Edition 11.2.0.4.0
I used the database connection pooling apache tomcat-jdbc:
You can take the following configuration parameters as a reference:
<Resource name="jdbc/exampleDB"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testWhileIdle="true"
testOnBorrow="true"
testOnReturn="false"
validationQuery="SELECT 1 FROM DUAL"
validationInterval="30000"
timeBetweenEvictionRunsMillis="30000"
maxActive="100"
minIdle="10"
maxWait="10000"
initialSize="10"
removeAbandonedTimeout="60"
removeAbandoned="true"
logAbandoned="true"
minEvictableIdleTimeMillis="30000"
jmxEnabled="true"
jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer"
username="your-username"
password="your-password"
driverClassName="oracle.jdbc.driver.OracleDriver"
url="jdbc:oracle:thin:#localhost:1521:xe"/>
This configuration was sufficient to fix the error. This works fine for me in the scenario mentioned above.
For more details about the setup apache tomcat-jdbc: https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html
Downgrading the JRE from 7 to 6 fixed this issue for me.
Yes, as #ggkmath said, sometimes a good old restart is exactly what you need. Like when "contact the author and have him rewrite the app, meanwhile wait" is not an option.
This happens when an application is not written (yet) in a way that it can handle restarts of the underlying database.
In our case we had a query which loads multiple items with select * from x where something in (...)
The in part was so long for benchmark test.(17mb as text query). Query is valid but text so long. Shortening the query solved the problem.
I got this error then restarted my GlassFish server that held connection pools between my client app and the database, and the error went away. So, try restarting your application server if applicable.
I seemed to fix my instance by removing the parameter placeholder for a parameterized query.
For some reason, using these placeholders were working fine, and then they stopped working and I got the error/bug.
As a workaround, I substituted literals for my placeholders and it started working.
Remove this
where
SOME_VAR = :1
Use this
where
SOME_VAR = 'Value'
Seemed to be an issue with a view. JDBC query was using a view. I took a guess, recompiled the view and error is gone.
In my case the error occurs running a simple query like this in SQLdeveloper:
select count(1) from tabel1 inner join tabel2 on table1.id = table2.id_table1 ;
I solved the error so...
select
/*+OPT_PARAM('_index_join_enabled' 'false') */
count(1) from tabel1 inner join tabel2 on table1.id = table2.id_table1 ;
MySQL closes a connection after a certain time if nothing happens (8 hours by default). The time can be influenced by the wait_timeout variable in the configuration.
I have an Eclipse RCP application where I use EclipseLink as persistence framework and I get an error when the client exceeds the timeout:
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
...
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
...
com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException
...
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor...
I tried to set autoReconnect/autoReconnectForPools=true but this does not help.
Thanks
EDIT
In my persistence.xml I have the following properties set:
<property
name="eclipselink.jdbc.read-connections.max"
value="10" />
<property
name="eclipselink.jdbc.cache-statements"
value="true" />
<property
name="eclipselink.jdbc.read-connections.shared"
value="true" />
The rest of the configuration is done in the code:
Map<Object, Object> map = ...
map.put(PersistenceUnitProperties.JDBC_URL,...);
map.put(PersistenceUnitProperties.JDBC_USER,...);
map.put(PersistenceUnitProperties.JDBC_PASSWORD, ...);
map.put(PersistenceUnitProperties.JDBC_DRIVER, ...);
map.put(PersistenceUnitProperties.CLASSLOADER, this.getClass()
.getClassLoader());
map.put(PersistenceUnitProperties.TARGET_DATABASE, "MySQL");
entityManagerFactory = new PersistenceProvider()
.createEntityManagerFactory("...", map);
EclipseLink should auto reconnect dead connections. EclipseLink will trap the error, test the connection and if dead reconnect and possibly retry the query (if outside a transaction).
But this depends on what connection pooling you are using, what is your persistence.xml.
The easiest way to do this would be to spawn a thread that sends some sort of keepalive or simple query every hour or so. Here, we would leave a flag so the thread can be shut down on program exit, db change, etc. If it needs to respond faster to that type of shutdown, you can change the counter in the for loop and the sleep time.
boolean parentKilledMe = false;
while (!parentKilledMe){
//put query here
for (int x = 0; x < 360 && !parentKilledMe;x++){
try{
Thread.sleep(6000);
} catch (InterruptedException e) {
//your error handling here
}
}
}