We Keep Coding

Getting Hazelcast NullPointerException for HTTP session, when user tries to login into web application for the for first time

Getting Hazelcast NullPointerException for HTTP session for first time when user tries to login into our application.
Our application is using Spring framework (without spring boot).
we configured below property at server level.
HAZELCAST_CLUSTER_HOST_LIST= <server hostname>
We are using wildfly server and right now there is only one one member (only one node) in Hazelcast cluster.
In log we observe hazel cast started properly.
2022-10-04 12:12:25,790 INFO [com.hazelcast.internal.cluster.ClusterService] (ServerService Thread Pool -- 88) ([]) [128.191.161.63]:5701 [TEST-CLUSTER] [4.2]
Members {size:1, ver:1} [
Member [128.191.161.63]:5701 - c12838bc-17d9-43c0-b8f2-58573d262453 this
]
We also observe below error in log
2022-10-04 12:29:16,355 INFO [com.hazelcast.web.ClusteredSessionService] (default task-1) ([]) Retrying the connection!!
2022-10-04 12:29:16,355 INFO [stdout] (default task-1) INFO [com.hazelcast.web.ClusteredSessionService] (default task-1) ([]) Retrying the connection!!
2022-10-04 12:29:16,357 INFO [com.hazelcast.web.HazelcastInstanceLoader] (default task-1) ([]) Using existing Hazelcast instance with name [${jboss.host.name}] for session replication
2022-10-04 12:29:16,357 INFO [stdout] (default task-1) INFO [com.hazelcast.web.HazelcastInstanceLoader] (default task-1) ([]) Using existing Hazelcast instance with name [${jboss.host.name}] for session replication
2022-10-04 12:29:16,357 WARN [com.hazelcast.web.ClusteredSessionService] (default task-1) ([]) Cannot connect to Hazelcast server: Hazelcast instance with name [${jboss.host.name}] could not be found.
What I observe is
com.hazelcast.internal.cluster.ClusterService class is showing as cluster created successfully.
but
com.hazelcast.web.ClusteredSessionService class is showing error
Hazelcast instance with name [${jboss.host.name}] could not be found
Finally this is causing below NullPointerException, when we trying to use session Object.
java.lang.NullPointerException: null
at com.hazelcast.web.ClusteredSessionService.setAttribute(ClusteredSessionService.java:227) ~[hazelcast-wm-4.0.jar:4.0]
at com.hazelcast.web.HazelcastHttpSession.setAttribute(HazelcastHttpSession.java:101) ~[hazelcast-wm-4.0.jar:4.0]
at SSOLoginController.login(SSOLoginController.java:51) ~[classes:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.15.jar:5.3.15]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.15.jar:5.3.15]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.15.jar:5.3.15]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.15.jar:5.3.15]

Rabbitmq queue is not automatically created on server start

I get the following exception when starting my server on one of the environments. However, it works fine on the another environment.
Caused by: org.springframework.amqp.AmqpIllegalStateException: Fatal exception on listener startup
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:342)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:363)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
... 54 more
Caused by: org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:474)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
Note: I have the RabbitAdmin defined in the spring xml definitions as follows:
<bean id="admin" class="org.springframework.amqp.rabbit.core.RabbitAdmin">
<constructor-arg ref="rabbitConnectionFactory"/>
</bean>
I am not sure why the queues will get autocreated on one environment and not with the other.Any help with this will be great.
<rabbit:queue name="blah.blah.queue">
<rabbit:queue-arguments>
<entry key="x-ha-policy" value="all"/>
</rabbit:queue-arguments>
</rabbit:queue>
<rabbit:direct-exchange name="blah.blah.exchange">
<rabbit:bindings>
<rabbit:binding queue="blah.blah.queue" key="blah.blah.routing.key"/>
</rabbit:bindings>
</rabbit:direct-exchange>
*Added the queue declaration
Here are some more error logs:
2018-01-04 17:57:46,782 [instance=01] [SimpleAsyncTaskExecutor-1] ERROR org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer (SimpleMessageListenerContainer.java:562) - Consumer received fatal exception on startup
org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:231)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:527)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)
... 2 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 11 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:474)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:315)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
.Adding some debug logs.. I have not yet been able to figure out the issue.
2018-01-18 16:16:21,583 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:188) - Creating cached Rabbit Channel from AMQChannel(amqp://guest#10.221.57.217:5672/,1)
2018-01-18 16:16:21,596 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.core.RabbitTemplate (RabbitTemplate.java:625) - Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest#10.221.57.217:5672/,1)
2018-01-18 16:16:21,596 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.core.RabbitAdmin (RabbitAdmin.java:387) - declaring Exchange 'blah.blah.exchange'
2018-01-18 16:16:22,439 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:354) - Detected closed channel on exception. Re-initializing: null
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.context.support.DefaultLifecycleProcessor (DefaultLifecycleProcessor.java:170) - Starting bean 'blahblahlistener' of type [class org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer]
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.aop.framework.JdkDynamicAopProxy (JdkDynamicAopProxy.java:117) - Creating JDK dynamic proxy: target source is SingletonTargetSource for target object [org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1#45ecefcc]
2018-01-18 16:16:22,652 [instance=01] [RMI TCP Connection(3)-127.0.0.1] DEBUG org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer (AbstractMessageListenerContainer.java:361) - Starting Rabbit listener container.
2018-01-18 16:16:22,653 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.listener.BlockingQueueConsumer (BlockingQueueConsumer.java:198) - Starting consumer Consumer: tag=[null], channel=null, acknowledgeMode=AUTO local queue size=0
2018-01-18 16:16:23,078 [instance=01] [SimpleAsyncTaskExecutor-1] DEBUG org.springframework.amqp.rabbit.connection.CachingConnectionFactory (CachingConnectionFactory.java:354) - Detected closed channel on exception. Re-initializing: null
2018-01-18 16:16:23,290 [instance=01] [SimpleAsyncTaskExecutor-1] WARN org.springframework.amqp.rabbit.listener.BlockingQueueConsumer (BlockingQueueConsumer.java:222) - Reconnect failed; retries left=2
java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:790)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:348)
at com.sun.proxy.$Proxy89.queueDeclarePassive(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:216)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:527)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'blah.blah.queue' in vhost '/', class-id=50, method-id=10), null, ""}
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:343)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:216)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 11 more
.

I was finally able to figure this out. Posting the answer here just in case it helps others who come across a similar situation:
I figured out that the app was declaring another pre-existing queue. However, this pre-existing queue had a dead-letter-routing-key defined in one environment and not in the other... inconsistency across environments. The xml did not have this argument defined. So it worked in the other environment and not in the 1st one. So before going ahead to create any of the non-existing queues, the rabbit channel was getting closed because the arguments mismatched.

IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener

We are using Wildfly8.2 server. The DB connection pool managed by Wildfly server is not able to reconnect (or to regenerate a valid/managed connection pool) after the connections are invalidated by the database server.
Please see the below error:
2016-05-03 11:35:43,662 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (NRG) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener#231fd521[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection#7ea688d0 connection handles=1 lastUse=1462286812044 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool#794f665c mcp=SemaphoreArrayListManagedConnectionPool#43e70a99[pool=DSNAME] xaResource=LocalXAResourceImpl#7482a7a7[connectionListener=231fd521 connectionManager=3d2232ff warned=false currentXid=null productName=Oracle productVersion=Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options jndiName=java:jboss/DSNAME] txSync=null]: javax.resource.spi.ResourceAdapterInternalException: Unexpected error<br>
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:644)<br>
at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.connectionError(BaseWrapperManagedConnection.java:610)<br>
at org.jboss.jca.adapters.jdbc.WrappedConnection.checkException(WrappedConnection.java:1640)
at org.jboss.jca.adapters.jdbc.WrappedStatement.checkException(WrappedStatement.java:1267)<br>
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:467)
at com.abc.app.beans.ReconcileInfo.getNextMessage(ReconcileInfo.java:412) [classes:]
at com.abc.app.beans.GroupInfo.sendMessage(GroupInfo.java:819) [classes:]<br>
at com.abc.app.app.GroupSendThread.send(GroupSendThread.java:112) <br>[classes:]
at com.ctsu.rdts.app.GroupSendThread.run(GroupSendThread.java:66) [classes:]
Caused by: java.lang.ThreadDeath<br>
at java.lang.Thread.stop(Thread.java:850) [rt.jar:1.8.0_31]
at com.abc.app.app.HttpDequeue.stopThread(HttpDequeue.java:876) [classes:]
at com.abc.app.app.HttpDequeue.clearLocks(HttpDequeue.java:397) [classes:]
at com.abc.app.app.HttpDequeue.run(HttpDequeue.java:383) [classes:]

JBoss AS 7 shutdown - connection manager closed before WAR deployment stopped

Env: JBoss AS 7.1.1.Final.
I have a WAR application using a data source taken from JBoss AS JNDI. When I shut down the server (Ctrl+C in the console), the application receives a shutdown command and starts to destroy its Spring context. However, I use a scheduler to perform some DB operations. When the application is closing, I want the tasks that are currently in the queue to be finished (but no new tasks are accepted - standard JDK Executor.shutdown() behaviour). This works fine when I undeploy the application without stopping the server. However, when I stop the whole server, the connection manager is closed before the application undeployment, which results in
14:31:51,604 INFO [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers
14:31:51,617 INFO [org.apache.coyote.http11.Http11Protocol] Stopping Coyote HTTP/1.1 on http-127.0.0.1-127.0.0.1-18080
14:31:51,638 INFO [org.hornetq.ra.HornetQResourceAdapter] HornetQ resource adapter stopped
14:31:51,653 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/MY-APP]] Closing Spring root WebApplicationContext
14:31:51,656 INFO [org.springframework.web.context.support.XmlWebApplicationContext] Closing Root WebApplicationContext: startup date [Tue Jul 09 14:30:56 CST 2013]; root of context hierarchy
14:31:51,659 INFO [org.springframework.beans.factory.support.DefaultListableBeanFactory] Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory#2c591927: defining beans [<snipped>]; root of factory hierarchy
14:31:51,662 INFO [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.13.Final (HQ_2_2_13_FINAL_AS7, 122) [5f713ff6-5f86-11e2-a25d-1f3857764d50] stopped
14:31:51,673 INFO [MY-APP.Shutdown] Initializing shutdown. Already running tasks will be finished, new tasks will not be executed.
14:31:53,626 ERROR [org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler] Unexpected error occurred in scheduled task.: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:575) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:818) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:874) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:882) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at com.my.app.scanner.db.SyncEntryRepository.deleteById(SyncEntryRepository.java:26) [classes:]
at com.my.app.rules.orphanentries.OrphanedEntriesProcessor.process(OrphanedEntriesProcessor.java:22) [classes:]
at com.my.app.routing.Router$RoutingWorker.performRouting(Router.java:49) [classes:]
at com.my.app.routing.Router$RoutingWorker.route(Router.java:32) [classes:]
at com.my.app.routing.Router.route(Router.java:18) [classes:]
at com.my.app.transformation.Transformation.perform(Transformation.java:21) [classes:]
at com.my.app.MyApp.run(MyApp.java:18) [classes:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:64) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53) [spring-context-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_45]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [rt.jar:1.6.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.6.0_45]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:137)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.2.3.RELEASE.jar:3.2.3.RELEASE]
... 26 more
Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/my/DS1
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:321)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:129)
... 28 more
14:31:53,640 INFO [MY-APP.Shutdown] Shutdown complete
14:31:53,651 INFO [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'
14:31:53,656 INFO [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] Closing JPA EntityManagerFactory for persistence unit 'default'
14:31:53,837 INFO [org.jboss.as.server.deployment] JBAS015877: Stopped deployment MY-APP.war in 2259ms
14:31:56,158 INFO [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService
14:31:56,158 INFO [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager
14:31:56,160 INFO [org.jboss.as] JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 4567ms
In my standalone.xml I have
<subsystem xmlns="urn:jboss:domain:datasources:1.0">
<datasources>
<datasource jndi-name="java:/my/DS1" pool-name="My1" enabled="true" use-java-context="true">
<connection-url>jdbc:oracle:thin:#10.172.1.1:1521:ABCD</connection-url>
<driver>oracle</driver>
<pool>
<min-pool-size>1</min-pool-size>
<max-pool-size>10</max-pool-size>
<prefill>true</prefill>
<use-strict-min>false</use-strict-min>
</pool>
<security>
<user-name>xxx</user-name>
<password>xxx</password>
</security>
<timeout>
<idle-timeout-minutes>5</idle-timeout-minutes>
</timeout>
<statement>
<prepared-statement-cache-size>500</prepared-statement-cache-size>
<share-prepared-statements>true</share-prepared-statements>
</statement>
</datasource>
<drivers>
<driver name="oracle" module="com.oracle"/>
</drivers>
</datasources>
</subsystem>
then, I lookup the datasources in the code:
DataSource ds = new InitialContext().lookup("java:/my/DS1");
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
The datasource has to be looked up programmatically and not hard-coded in spring.xml since there can be multiple data sources and their JNDI names are configured in external properties file that the application scans on startup.
Then, the jdbcTemplate is passed to a TableScanner that is invoked with Spring scheduling:
public TableScanner(JdbTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Scheduled(fixedDelay = 5000)
public void run() {
// query a table using jdbcTemplate and process the retrieved records, deleting them at the end of processing
}
When stopping of the server occurs during run() method invocation, all the retrieved records should be processed and deleted. When the application tries to delete them, the connection manager is already closed, hence the exception.
Is there any way to make the connection manager (or any other JBoss AS subsystem I might need) to wait until the application is stopped?

I had the same problem, and I have now found the cause described in https://issues.jboss.org/browse/WFLY-944 . The solution proposed there in order to avoid it, is to declare the jndi as a resource in your web application, so that it does not unbind from the jboss server before your application terminates.
To achieve this add the following part to your web.xml which sets a reference to your jboss jndi with name "jdbc/myDS".
<resource-ref>
<res-ref-name>jdbc/myDS</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
<lookup-name>java:/my/DS1</lookup-name>
</resource-ref>
Then instead of looking up "java:/my/DS1" from your code part that points to the actual jndi, you should now instead look up "java:comp/env/jdbc/myDS" which will reference the resource you put in the web.xml that actually points to the actual jndi (that we previously specified as the 'lookup-name' xml property in the part added in the web.xml):
DataSource ds = (DataSource)
ctx.lookup("java:comp/env/jdbc/myDS");

I don't know which functionality is offered by Spring Scheduling, but if you had used an executor from the standard API, the solution would have been to add a ServletContextListener to your web application, which is invoked by the container when undeploying or stopping your application. In the contextDestroyed(ServletContextEvent sce) method, you would then invoke something like executor.shutdown(); executor.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS), which will stop the executor and wait for all pending tasks to finish. You may want to consider using a shorter timeout when invoking awaitTermination.
IMHO the better solution may be to avoid using Spring within Java EE applications, at least when exactly the same functionality is offered by the Java EE application server as well, as in this case.

Have you considered using EJB Scheduled bean? This way jboss will spawn the worker thread.

I found this JBoss AS issue, which reflects my problem. It appears that if the data sources are not bound in a static way, via #Resource, then the server does not know that the data source is still in use. I went with letting Spring manage the data sources on its own, which may not be the Java EE way, but gets the work done and is appropriate in my situation.

Jboss 7.1 standalone socket/port error

System: OS X 10.7.X
When starting jboss from the terminal I get a bind exception without a port number being mentioned. Could anyone enlighten me?
11:52:27,635 ERROR [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-11) Error initializing endpoint: java.lang.Exception: Socket bind failed: [48] Address already in use
at org.apache.tomcat.util.net.AprEndpoint.init(AprEndpoint.java:578) [jbossweb-7.0.7.Final.jar:]
at org.apache.coyote.http11.Http11AprProtocol.init(Http11AprProtocol.java:121) [jbossweb-7.0.7.Final.jar:]
at org.apache.catalina.connector.Connector.init(Connector.java:983) [jbossweb-7.0.7.Final.jar:]
at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:267) [jboss-as-web-7.1.0.CR1b.jar:7.1.0.CR1b]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1824) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1759) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
at java.lang.Thread.run(Thread.java:680) [:1.6.0_29]
11:52:27,637 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-11) MSC00001: Failed to start service jboss.web.connector.http: org.jboss.msc.service.StartException in service jboss.web.connector.http: LifecycleException: Protocol handler initialization failed: java.lang.Exception: Socket bind failed: [48] Address already in use
at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:271)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1824) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1759) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_29]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_29]
at java.lang.Thread.run(Thread.java:680) [:1.6.0_29]
Caused by: LifecycleException: Protocol handler initialization failed: java.lang.Exception: Socket bind failed: [48] Address already in use
at org.apache.catalina.connector.Connector.init(Connector.java:985)
at org.jboss.as.web.WebConnectorService.start(WebConnectorService.java:267)
... 5 more

If you look in standalone.xml, there should be a section at the end with port numbers (from what i remember!). You could just check to see if any other process is using any of them.
More specifically, the stacktrace names the web connector, which should help narrow down which port this is. I would expect it to be the main HTTP port, which i think is 8080 by default.
It is really, really annoying that the exception doesn't specify the port.