I am using Spring Integration 2.0 with the below bean configuration for ThreadPoolTaskExecutor.
<bean id="MyTaskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" destroy-method="destroy" >
<property name="corePoolSize" value="25" />
<property name="maxPoolSize" value="25" />
<property name="queueCapacity" value="0" />
</bean>
I am starting to see the below error message in the application log. Need to understand what is happening in ThreadPoolTaskExecutor.
05-Mar-2020 11:34:36,389 | ERROR | | | | | | | [org.springframework.integration.channel.MessagePublishingErrorHandler:handleError:100] | failure occurred in messaging task
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor#40102708[Running, pool size = 25, active threads = 25, queued tasks = 0, completed tasks = 13487163]] did not accept task: org.springframework.integration.util.ErrorHandlingTaskExecutor$1#5de60ad1
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:244)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:174)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:51)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.springframework.integration.util.ErrorHandlingTaskExecutor$1#5de60ad1 rejected from java.util.concurrent.ThreadPoolExecutor#40102709[Running, pool size = 25, active threads = 25, queued tasks = 0, completed tasks = 13487163]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241)
... 11 more
Can some one identify the reason for this?
Env: JBoss AS 7.1.1.Final.
I have a WAR application using a data source taken from JBoss AS JNDI. When I shut down the server (Ctrl+C in the console), the application receives a shutdown command and starts to destroy its Spring context. However, I use a scheduler to perform some DB operations. When the application is closing, I want the tasks that are currently in the queue to be finished (but no new tasks are accepted - standard JDK Executor.shutdown() behaviour). This works fine when I undeploy the application without stopping the server. However, when I stop the whole server, the connection manager is closed before the application undeployment, which results in
14:31:51,604 INFO [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers
14:31:51,617 INFO [org.apache.coyote.http11.Http11Protocol] Stopping Coyote HTTP/1.1 on http-127.0.0.1-127.0.0.1-18080
14:31:51,638 INFO [org.hornetq.ra.HornetQResourceAdapter] HornetQ resource adapter stopped
14:31:51,653 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/MY-APP]] Closing Spring root WebApplicationContext
14:31:51,656 INFO [org.springframework.web.context.support.XmlWebApplicationContext] Closing Root WebApplicationContext: startup date [Tue Jul 09 14:30:56 CST 2013]; root of context hierarchy
14:31:51,659 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#2c591927: defining beans [<snipped>]; root of factory hierarchy
14:31:51,662 INFO [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.13.Final (HQ_2_2_13_FINAL_AS7, 122) [5f713ff6-5f86-11e2-a25d-1f3857764d50] stopped
14:31:51,673 INFO [MY-APP.Shutdown] Initializing shutdown. Already running tasks will be finished, new tasks will not be executed.
14:31:53,626 ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:575) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:818) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:874) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:882) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at com.my.app.scanner.db.SyncEntryRepository.deleteById(SyncEntryRepository.java:26) [classes:]
at com.my.app.rules.orphanentries.OrphanedEntriesProcessor.process(OrphanedEntriesProcessor.java:22) [classes:]
at com.my.app.routing.Router$RoutingWorker.performRouting(Router.java:49) [classes:]
at com.my.app.routing.Router$RoutingWorker.route(Router.java:32) [classes:]
at com.my.app.routing.Router.route(Router.java:18) [classes:]
at com.my.app.transformation.Transformation.perform(Transformation.java:21) [classes:]
at com.my.app.MyApp.run(MyApp.java:18) [classes:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [rt.jar:1.6.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
... 26 more
Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:321)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)
... 28 more
14:31:53,640 INFO [MY-APP.Shutdown] Shutdown complete
14:31:53,651 INFO [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'
14:31:53,656 INFO [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'
14:31:53,837 INFO [org.jboss.as.server.deployment] JBAS015877: Stopped deployment MY-APP.war in 2259ms
14:31:56,158 INFO [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService
14:31:56,158 INFO [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager
14:31:56,160 INFO [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 4567ms
In my standalone.xml I have
<subsystem xmlns="urn:jboss:domain:datasources:1.0">
<datasources>
<datasource jndi-name="java:/my/DS1" pool-name="My1" enabled="true" use-java-context="true">
<connection-url>jdbc:oracle:thin:#10.172.1.1:1521:ABCD</connection-url>
<driver>oracle</driver>
<pool>
<min-pool-size>1</min-pool-size>
<max-pool-size>10</max-pool-size>
<prefill>true</prefill>
<use-strict-min>false</use-strict-min>
</pool>
<security>
<user-name>xxx</user-name>
<password>xxx</password>
</security>
<timeout>
<idle-timeout-minutes>5</idle-timeout-minutes>
</timeout>
<statement>
<prepared-statement-cache-size>500</prepared-statement-cache-size>
<share-prepared-statements>true</share-prepared-statements>
</statement>
</datasource>
<drivers>
<driver name="oracle" module="com.oracle"/>
</drivers>
</datasources>
</subsystem>
then, I lookup the datasources in the code:
DataSource ds = new InitialContext().lookup("java:/my/DS1");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
The datasource has to be looked up programmatically and not hard-coded in spring.xml since there can be multiple data sources and their JNDI names are configured in external properties file that the application scans on startup.
Then, the jdbcTemplate is passed to a TableScanner that is invoked with Spring scheduling:
public TableScanner(JdbTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Scheduled(fixedDelay = 5000)
public void run() {
// query a table using jdbcTemplate and process the retrieved records, deleting them at the end of processing
}
When stopping of the server occurs during run() method invocation, all the retrieved records should be processed and deleted. When the application tries to delete them, the connection manager is already closed, hence the exception.
Is there any way to make the connection manager (or any other JBoss AS subsystem I might need) to wait until the application is stopped?
I had the same problem, and I have now found the cause described in https://issues.jboss.org/browse/WFLY-944 . The solution proposed there in order to avoid it, is to declare the jndi as a resource in your web application, so that it does not unbind from the jboss server before your application terminates.
To achieve this add the following part to your web.xml which sets a reference to your jboss jndi with name "jdbc/myDS".
<resource-ref>
<res-ref-name>jdbc/myDS</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<lookup-name>java:/my/DS1</lookup-name>
</resource-ref>
Then instead of looking up "java:/my/DS1" from your code part that points to the actual jndi, you should now instead look up "java:comp/env/jdbc/myDS" which will reference the resource you put in the web.xml that actually points to the actual jndi (that we previously specified as the 'lookup-name' xml property in the part added in the web.xml):
DataSource ds = (DataSource)
ctx.lookup("java:comp/env/jdbc/myDS");
I don't know which functionality is offered by Spring Scheduling, but if you had used an executor from the standard API, the solution would have been to add a ServletContextListener to your web application, which is invoked by the container when undeploying or stopping your application. In the contextDestroyed(ServletContextEvent sce) method, you would then invoke something like executor.shutdown(); executor.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS), which will stop the executor and wait for all pending tasks to finish. You may want to consider using a shorter timeout when invoking awaitTermination.
IMHO the better solution may be to avoid using Spring within Java EE applications, at least when exactly the same functionality is offered by the Java EE application server as well, as in this case.
Have you considered using EJB Scheduled bean? This way jboss will spawn the worker thread.
I found this JBoss AS issue, which reflects my problem. It appears that if the data sources are not bound in a static way, via #Resource, then the server does not know that the data source is still in use. I went with letting Spring manage the data sources on its own, which may not be the Java EE way, but gets the work done and is appropriate in my situation.
I have a multithreaded application and im using log4j for the logging. When i test my application on my windows 7/intel laptop, it works just fine. But when i deploy the same application to the production server(solaris/sparc), it works for 10 minutes then starts to slow down(very slow). I used profiler to see the threads state. There were a lot of threads blocked because of the log4j operations in the thread dump. Im also using async appender but coulnd't manage to get any successful result. Any guess?
Here are some dump:
"pool-7-thread-105" - Thread t#22939
java.lang.Thread.State: BLOCKED
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <794f2dae> (a org.apache.log4j.spi.RootLogger) owned by "pool-7-thread-112" t#22946
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.info(Category.java:666)
"pool-7-thread-112" - Thread t#22946
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
- locked <5fcbc329> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(Unknown Source)
- locked <19e9d0c5> (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
- locked <3680c465> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flush(Unknown Source)
at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
- locked <3fddded7> (a org.apache.log4j.ConsoleAppender)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
- locked <794f2dae> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.info(Category.java:666)
Also log4j.xml - version 1.2.17
<appender name="scripts" class="org.apache.log4j.DailyRollingFileAppender">
<param name="threshold" value="INFO"/>
<param name="file" value="log/script.log"/>
<param name="DatePattern" value="'.'yyyy-MM-dd"/>
<layout class="org.apache.log4j.EnhancedPatternLayout">
<param name="ConversionPattern"
value="%-6p[%-d{ISO8601}] [%t] %m (%F:%L) %n"/>
</layout>
</appender>
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<param name="threshold" value="INFO"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-6p[%-d{ISO8601}] [%t] %m (%F:%L) %n"/>
</layout>
</appender>
<appender name="ASYNC" class="org.apache.log4j.AsyncAppender">
<param name="BufferSize" value="5000"/>
<appender-ref ref="scripts"/>
</appender>
<root>
<priority value="debug"/>
<appender-ref ref="stdout"/>
<appender-ref ref="ASYNC"/>
</root>
So the threads that are trying to log are blocked. Why? Because a logging thread is currently blocked in java.io.FileOutputStream.writeBytes(Native Method). That strongly suggests that the thing the logging is going to (a file or whatever) cannot keep up with the amount of logging being done.
The way to troubleshoot this is to start with looking at where your logging is going. If it's going to a file, is that file on a fast, local filesystem? Is disk I/O an issue?
You may wish to use a logging scheme that doesn't block on I/O. It doesn't have to -- other threads can log to an internal queue while a thread is doing disk I/O. However, if the logging rate exceeds the I/O rate, eventually you have to stop logging from building up in memory somehow, and that requires either blocking threads or throwing log entries away. That said, this shouldn't be an issue unless you are either logging way too much, have a serious I/O performance issue (like logging over NFS on a crowded network), or have file buffering disabled somehow.
Judging from the stack dump, something is calling flush() ... so that is going to negate a lot of the "goodness" of using BufferedOutputStream. If you can figure out how to avoid the flushes the throughput will improve.
David Schwartz comments thus:
Windows is notorious for not really honoring file flushes as a way to improve its performance. Solaris and Linux really do full flushes. This might help explain the platform-dependent performance difference.
(I didn't know that. And yes it would.)
Turning off immediateFlush in the WriterAppender might help
Excellent advice.
I have a webservice - java/hibernate/mysql/innoDB.
I keep getting apperante deadlock exception.
I was looking around the net, and tried what was suggested but still have problems...
I am using the following in my hibernate cfg:
hibernate.cfg:
<property name="hibernate.connection.driver_class">org.gjt.mm.mysql.Driver</property>
<property name="connection.autoReconnect">true</property>
<property name="connection.autoReconnectForPools">true</property>
<property name="connection.is-connection-validation-required">true</property>
<property name="hibernate.connection.zeroDateTimeBehavior">convertToNull</property>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLInnoDBDialect</property>
<property name="hibernate.show_sql">false</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.format_sql">true</property>
<property name="hibernate.use_sql_comments">true</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.timeout">1800</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.acquire_increment">1</property>
<!-- this property forces the revalidation of a connection after the given amount of time (in secs) -->
<!-- it must be set to LESS than the wait_timout setting for the mysql server (this setting defaults to 28800 secs (8 hours)) -->
<property name="hibernate.c3p0.idle_test_period">2100</property>
c3p0.properties:
c3p0.properties
c3p0.testConnectionOnCheckout=true
c3p0.preferredTestQuery = SELECT 1;
I get the following exception:
[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#1e5161e1 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#1e5161e1 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#60dd0912 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4f7eae6c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#5ed5d3a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#22838de0
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#6ad77ed3
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:146)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2537)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2990)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1521)
com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:374)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:146)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2537)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2990)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1521)
com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:374)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:146)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2537)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2990)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1521)
com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:374)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#60dd0912 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#4f7eae6c (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask#5ed5d3a (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#1e5161e1 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
[WARN] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#1e5161e1 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#555214b9 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#7fcea971 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#58d67b44 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#6936daec
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#105ef1eb
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#495c83b2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#58ca40be
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#4660a77c
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:550)
com.mchange.v2.c3p0.impl.NewPooledConnection.cleanupUncachedStatements(NewPooledConnection.java:651)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:539)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:550)
com.mchange.v2.c3p0.impl.NewPooledConnection.cleanupUncachedStatements(NewPooledConnection.java:651)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:539)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:550)
com.mchange.v2.c3p0.impl.NewPooledConnection.cleanupUncachedStatements(NewPooledConnection.java:651)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:539)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#555214b9 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#7fcea971 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
[WARN] Task com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#58d67b44 (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
What are you trying to do? Seems to me there's something that's keeping your database connections open. As a result c3p0 is trying to get more connections to handle the extra requests.
I need to schedule a task using Spring 3.1.
I'm using Roo's with ActiveRecord pattern for entities.
Method to be scheduled is very simple, in a dedicated service bean:
public void sessionCleanup() {
featureSessionManager.sessionCleanup();
}
...configured as:
<bean id="scheduledTaskService" class="it.blablabla.ScheduledTaskService">
[...dependencies...]
</bean>
If I schedule it using #Scheduled(fixedRate=120000) and enabling annotation-driven scheduling with <task:annotation-driven/> everything works as expected.
But if I want to schedule the same method using XML configuration...
<task:scheduler id="scheduler" pool-size="1"/>
<task:scheduled-tasks scheduler="scheduler" >
<task:scheduled ref="scheduledTaskService" method="sessionCleanup" fixed-rate="120000"/>
</task:scheduled-tasks>
...application throws an exception on initialization.
2012-07-24 12:26:56,293 [pool-2-thread-1] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 'scheduler'
2012-07-24 12:26:56,671 [scheduler-1] ERROR org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.IllegalStateException: Entity manager has not been injected (is the Spring Aspects JAR configured as an AJC/AJDT aspects library?)
at it.blablabla.model.AbstractFeatureSession_Roo_Jpa_ActiveRecord.entityManager_aroundBody0(AbstractFeatureSession_Roo_Jpa_ActiveRecord.aj:38)
at it.blablabla.model.AbstractFeatureSession_Roo_Jpa_ActiveRecord.ajc$interMethod$it.blablabla_model_AbstractFeatureSession_Roo_Jpa_ActiveRecord$it.blablabla_model_AbstractFeatureSession$entityManager(AbstractFeatureSession_Roo_Jpa_ActiveRecord.aj:1)
at it.blablabla.model.AbstractFeatureSession.entityManager(AbstractFeatureSession.java:1)
at it.blablabla.model.AbstractFeatureSession_Roo_Jpa_ActiveRecord.ajc$interMethodDispatch1$it.blablabla_model_AbstractFeatureSession_Roo_Jpa_ActiveRecord$it.blablabla_model_AbstractFeatureSession$entityManager(AbstractFeatureSession_Roo_Jpa_ActiveRecord.aj)
at it.blablabla.model.FeatureSession.updateAsDroppedAllExpiredAndNotDropped_aroundBody10(FeatureSession.java:245)
at it.blablabla.model.FeatureSession.updateAsDroppedAllExpiredAndNotDropped(FeatureSession.java:1)
at it.blablabla.service.FeatureSessionManagementServiceImpl.dropExpiredSessions(FeatureSessionManagementServiceImpl.java:154)
at it.blablabla.service.FeatureSessionManagementServiceImpl.sessionCleanup(FeatureSessionManagementServiceImpl.java:178)
at it.blablabla.task.ScheduledTaskWorker.sessionCleanup(ScheduledTaskWorker.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Apparently it fails to inject EntityManager into entities instantiated by the task.
(Roo's ActiveRecords are #Configurable and EntityManager is injected on instance creation)
This happens either in Tomcat or in a JUnit.