We Keep Coding

HiveMQ Java library fails to automatically reconnect to broker

I'm using the HiveMQ library in my Java Spring application to connect to a Mosquitto instance as I find it more user-friendly compared to the Paho client. But something is going wrong with the automatic reconnection. From time to time the connection is lost and the application doesn't succeed in reconnecting (see logs 1). This can also be triggered by restarting the Mosquitto broker itself (see logs 2).
This is my client builder code with additional logging in the disconnect to check if the credentials are still correct:
client = MqttClient.builder()
.useMqttVersion5()
.identifier(identifier)
.serverHost(host)
.serverPort(port)
.sslWithDefaultConfig()
// https://www.hivemq.com/blog/hivemq-mqtt-client-features/reconnect-handling/
.automaticReconnectWithDefaultConfig()
.addDisconnectedListener(context -> logger.error("MQTT user {} with identifier {} on {}:{} has disconnected reason: {}",
username, identifier, host, port, context.getCause().getMessage()))
.buildAsync();
client.connectWith()
.simpleAuth()
.username(username)
.password(password.getBytes())
.applySimpleAuth()
.cleanStart(false)
.keepAlive(60)
.send();
1/ This is shown in my logs after the connection has been lost by the application itself:
2022-03-16 02:10:33.502 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for PINGRESP
2022-03-16 02:11:25.090 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
2022-03-16 02:12:27.200 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for CONNACK
2/ This is the logs after the broker has been restarted, some expected time-outs, but also in the end "not authorized":
2022-03-16 10:17:37.178 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Server closed connection without DISCONNECT.
2022-03-16 10:17:48.441 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:00.747 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:10.625 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: No route to host: ***/***:8883
2022-03-16 10:18:26.845 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:42.584 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
In both cases, the connection is back to normal with an application restart.
any ideas?

It appears that your question is answered in this issue:
If you set the username and password on the connect call, they will not be stored and reused when the client reconnects (for security reasons).
The following code (from the issue linked above) demonstrates the approach:
Mqtt3Client.builder()
.identifier("ePCR mobile-" + currentTimeMillis())
.serverHost(config.getHost())
.serverPort(config.getPort())
.automaticReconnectWithDefaultConfig()
.simpleAuth()
.username(config.getUsername())
.password(config.getPassword())
.applySimpleAuth()
.buildRx();

Resetting heartbeat timestamps because of huge system clock jump

I am running hazelcast application, and I am getting below error, after keeping my machine in sleep mode/log off for sometime.
2021-06-21 14:41:07.854 INFO 8288 --- [cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager
: [192.168.181.51]:5701 [APP] [4.2] System clock apparently jumped from 2021-06-21 14:10:28.569 to 2021-06-21 14:41:07.832 since last heartbeat (+1834263 ms)
2021-06-21 14:41:07.855 INFO 8288 --- [cached.thread-9] c.h.i.server.tcp.TcpServerConnection : [192.168.181.51]:5701 [APP] [4.2] Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Client heartbeat is timed out, closing connection to Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=true, connectionType=JVM, planeIndex=-1]. Now: 2021-06-21 14:41:07.833. LastTimePacketReceived: 2021-06-21 14:10:29.314
2021-06-21 14:41:07.915 WARN 8288 --- [cached.thread-7] c.h.i.c.impl.ClusterHeartbeatManager : [192.168.181.51]:5701 [APP] [4.2] Resetting heartbeat timestamps because of huge system clock jump! Clock-Jump: 1834263 ms, Heartbeat-Timeout: 60000 ms
2021-06-21 14:41:08.208 WARN 8288 --- [onMonitorThread] c.h.s.i.o.impl.InvocationMonitor : [192.168.181.51]:5701 [APP] [4.2] MonitorInvocationsTask delayed 1836451 ms
2021-06-21 14:41:08.213 WARN 8288 --- [onMonitorThread] c.h.s.i.o.impl.InvocationMonitor : [192.168.181.51]:5701 [APP] [4.2] BroadcastOperationControlTask delayed 1834623 ms
2021-06-21 14:41:08.539 INFO 8288 --- [cached.thread-9] c.h.i.server.tcp.TcpServerConnection : [192.168.181.51]:5701 [APP] [4.2] Connection[id=2, /127.0.0.1:5701->/127.0.0.1:5703, qualifier=null, endpoint=[127.0.0.1]:5703, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Client heartbeat is timed out, closing connection to Connection[id=2, /127.0.0.1:5701->/127.0.0.1:5703, qualifier=null, endpoint=[127.0.0.1]:5703, alive=true, connectionType=JVM, planeIndex=-1]. Now: 2021-06-21 14:41:08.539. LastTimePacketReceived: 2021-06-21 14:10:29.949
2021-06-21 14:41:08.551 WARN 8288 --- [ached.thread-36] c.h.i.cluster.impl.MulticastService : [192.168.181.51]:5701 [APP] [4.2] Sending multicast datagram failed. Exception message saying the operation is not permitted usually means the underlying OS is not able to send packets at a given pace. It can be caused by starting several hazelcast members in parallel when the members send their join message nearly at the same time.
java.net.NoRouteToHostException: No route to host: Datagram send failed
at java.net.TwoStacksPlainDatagramSocketImpl.send(Native Method) ~[na:1.8.0_251]
at java.net.DatagramSocket.send(Unknown Source) ~[na:1.8.0_251]
at com.hazelcast.internal.cluster.impl.MulticastService.send(MulticastService.java:291) ~[hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.internal.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:113) [hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.internal.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:75) [hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.internal.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:42) [hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.spi.impl.executionservice.impl.DelegateAndSkipOnConcurrentExecutionDecorator$DelegateDecorator.run(DelegateAndSkipOnConcurrentExecutionDecorator.java:77) [hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) [hazelcast-all-4.2.jar!/:4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_251]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_251]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-all-4.2.jar!/:4.2]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) [hazelcast-all-4.2.jar!/:4.2]
My client config is as below:
ClientConfig clientConfig = new ClientConfig();
clientConfig.setClusterName("abc");
clientConfig.getNetworkConfig().addAddress("localhost");
clientConfig.getNetworkConfig().setSmartRouting(true);
clientConfig.getNetworkConfig().addOutboundPortDefinition("5701-5720");
ClientConnectionStrategyConfig connectionStrategyConfig = clientConfig.getConnectionStrategyConfig();
ConnectionRetryConfig connectionRetryConfig = connectionStrategyConfig.getConnectionRetryConfig();
connectionRetryConfig.setInitialBackoffMillis(1000)
.setMaxBackoffMillis(60000)
.setMultiplier(2)
.setClusterConnectTimeoutMillis(1000)
.setJitter(0.2);
HazelcastClient hc = HazelcastClient.newHazelcastClient(clientConfig);
Please let me know what wrong configuration am I doing or why this is happening?

This is a common issue for socket based applications. Ideally you disable the sleep / power save mode. You can try using the SystemParametersInfo API:
SystemParametersInfo( SPI_SETPOWEROFFACTIVE, 0, NULL, 0 );
But typically this would be seen as ill behaved since you should disable the Power Off during installation of your application thus requesting permission.

Elasticsearch RestClient Connection reset by peer

I have in my AWS VPC a cluster of ES with 2 nodes. On top of those nodes I have a load balancer. In the same vpc I have a microservice that accesses Elasticsearch via RestHighLevelClient version 7.5.2 .
I create the client in the following manner :
public class ESClientWrapper {
#Getter
private RestHighLevelClient client;
public ESClientWrapper() throws IOException {
FileInputStream propertiesFile = new FileInputStream("/var/elastic.properties");
Properties properties = new Properties();
properties.load(propertiesFile );
RestClientBuilder builder = RestClient.builder(new HttpHost(
properties .getProperty("host"),
Integer.parseInt(properties.getProperty("port"))
));
this.client = new RestHighLevelClient(builder);
}
}
When my micro service doesn't get requests for a long time (12h..) there are occurrences when the first response that is sent (or a few after..) are getting the following error:
2020-09-09 07:03:13.106 INFO 1 --- [nio-8080-exec-1] c.a.a.services.CustomersMetadataService : Trying to add the following role : {role=a2}
2020-09-09 07:03:13.106 INFO 1 --- [nio-8080-exec-1] c.a.a.e.repositories.ESRepository : Trying to insert the following document to app-index : {role=a2}
2020-09-09 07:03:13.109 ERROR 1 --- [nio-8080-exec-1] c.a.a.e.dal.ESRepository : Failed to add customer : {role=a2}
java.io.IOException: Connection reset by peer
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:828) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:871) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
....
....
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.35.jar!/:9.0.35]
at java.base/java.lang.Thread.run(Thread.java:836) ~[na:na]
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:na]
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:na]
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[na:na]
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245) ~[na:na]
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[na:na]
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[na:na]
at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
... 1 common frames omitted
2020-09-09 07:06:55.109 INFO 1 --- [nio-8080-exec-2] c.a.a.services.MyService : Trying to add the following role : {role=a2}
2020-09-09 07:06:55.109 INFO 1 --- [nio-8080-exec-2] c.a.a.e.repositories.ESRepository : Trying to insert the following document to index app-index: {role=a2}
2020-09-09 07:06:55.211 INFO 1 --- [nio-8080-exec-2] c.a.a.e.dal.ESRepository : IndexResponse[index=app-index,type=_doc,id=x532323272533321870287,version=1,result=created,seqNo=70,primaryTerm=1,shards={"total":2,"successful":2,"failed":0}]
As you can see, 3 minutes after the failed request the next request was successfully handeled by ES. What can kill the request ? I checked Elasticsearch logs and didn't see any indication for killing connection. The MS is in the same vpc as elastic so it isn't passing through any firewall that might kill it.
I found the following issue in github that suggested to increase the default connection timeout but I'm wondering if the issue here is really a timeout problem and if increasing the default time is really the best solution..
Also, I found this bug opened in their repo regarding the same problem but without any answers.
UPDATE
I noticed that even after 10 minutes my service is up this happens. My service started and sent a query to ES and everything worked well. After 10 minutes I sent insert request and it failed on connection reset by peer.

In the end I didn't find a problem in my configuration/implementation. It seems like a bug in the implementation of Elasticsearch's RestHighLevelClient.
I implemented a retry mechanism that wraps the RestHighLevelClient and retries the query if I get the same error. I used Spring #Retry annotation for this solution.

I was facing the same issue. Everything worked fine, but after some time a single request got refused.
The solution (in my case) was to set the keepalive property of the tcp connection with:
final RestClientBuilder restClientBuilder = RestClient.builder(...);
restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultIOReactorConfig(IOReactorConfig.custom()
.setSoKeepAlive(true)
.build()))
Found here:
https://github.com/elastic/elasticsearch/issues/65213

SolrJ hanging when connecting to zookeeper

I have a local two instance Solr Cloud setup with a single zookeeper instance. I am trying to connect via SolrJ to execute a query however my code hangs for 2mins or so when executing the query and then fails. I have followed the basic example on the Solr wiki. The logs/code is below
2016-07-24 13:29:01.932 INFO 83666 --- [qtp699221219-28] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=localhost:2181 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.SolrZkClient$3#496eab9
2016-07-24 13:29:01.948 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager : Waiting for client to connect to ZooKeeper
2016-07-24 13:29:01.953 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-24 13:29:01.955 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to localhost/127.0.0.1:2181, initiating session
2016-07-24 13:29:01.967 INFO 83666 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1561cdd875e0004, negotiated timeout = 10000
2016-07-24 13:29:01.972 INFO 83666 --- [back-3-thread-1] o.a.solr.common.cloud.ConnectionManager : Watcher org.apache.solr.common.cloud.ConnectionManager#4bb95d56 name:ZooKeeperConnection Watcher:localhost:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2016-07-24 13:29:01.972 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ConnectionManager : Client is connected to ZooKeeper
2016-07-24 13:29:01.973 INFO 83666 --- [qtp699221219-28] o.apache.solr.common.cloud.SolrZkClient : Using default ZkACLProvider
2016-07-24 13:29:01.974 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Updating cluster state from ZooKeeper...
2016-07-24 13:29:01.990 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Loaded empty cluster properties
2016-07-24 13:29:01.995 INFO 83666 --- [qtp699221219-28] o.a.solr.common.cloud.ZkStateReader : Updated live nodes from ZooKeeper... (0) -> (2)
2016-07-24 13:31:24.653 ERROR 83666 --- [qtp699221219-28] o.a.s.client.solrj.impl.CloudSolrClient : Request to collection foo failed due to (0) java.net.ConnectException: Operation timed out, retry? 0
and my code is:
String zkHostString = "localhost:2181";
CloudSolrClient solr = new CloudSolrClient.Builder().withZkHost(zkHostString).build();
solr.setDefaultCollection("foo");
SolrQuery query = new SolrQuery();
query.set("q", "*:*");
QueryResponse response = null;
try {
response = solr.query(query);
} catch (SolrServerException e) {
return null;
}
//Do Something with the results...

Urgh, I'm an idiot, the zookeeper instance and solr instances are inside docker, the code posted above is not. So Zookeeper reported back the solr urls using the docker containers ip...The host needs to connect via localhost and not the docker container ip.
Eg: Zookeeper responds [http://172.17.0.5:8983/solr/foo_shard1_replica2, http://172.17.0.6:8984/solr/foo_shard1_replica1]
but my code needs to call [http://localhost:8983/solr/foo_shard1_replica2, http://localhost:8984/solr/foo_shard1_replica1]

Can a SocketException end of file be caused by low memory in the server?

Hi dear community of java addicts.
I was getting these exceptions in a CentOs VM, probably running with low RAM and then I noted that the time was not correctly synchronized between the other VM needed to communicate with my nice component....
I was wondering to know, When ? Why ? How ? A SocketException: end of file is produced in a LINUX server...
These are my logs:
2012-05-16 13:22:41,863 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][initDatabaseProperties] - Initializing database custom properties.
2012-05-16 13:22:41,864 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][initDatabaseProperties] - Setting NLS_DATE_FORMAT to : DD/MM/YYYY HH24:MI:SS
2012-05-16 13:22:47,096 [Timer-2] ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker][handleException] - Got marshalling exception, exiting
java.net.SocketException: end of file
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:685)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.ConnectionValidator.doCheckConnectionWithoutLease(ConnectionValidator.java:828)
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:345)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2012-05-16 13:22:47,288 [Thread-2624] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:22:47,329 [Thread-2625] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:22:51,146 [Timer-4] WARN [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][run] - org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask#7a7385ac: detected failure on control connection Thread[control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623],5,] (5c4o020-jlorp4-h29d35xs-1-h2aawvkq-l2t: requesting new control connection
2012-05-16 13:22:51,159 [controlConnectionRecreate:control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623]] ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][createControlConnection] - unable to get secondary locator
org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:613)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.getSecondaryLocator(BisocketClientInvoker.java:640)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:230)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask$1.run(BisocketServerInvoker.java:1048)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:465)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:913)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:602)
... 5 more
2012-05-16 13:22:51,161 [controlConnectionRecreate:control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623]] ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][run] - Unable to recreate control connection: InvokerLocator [null://ams-dev-bo.swissbytes.ch:11641/null]
java.io.IOException: unable to get secondary locator: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:235)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask$1.run(BisocketServerInvoker.java:1048)
2012-05-16 13:22:56,870 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - Query for the level[1] was executed.
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][doFilterResultsCotainCipAdress] - doFilterResultsCotainCipAdress() - Searching Cip Address: 1213194
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - cipAddress: 1213194 was NOT FOUND in Filter Results
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - Current result size is not on filter's range : filter1 => [ 1 , 10 ] vs 0
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.SimilarDebtorsProcessor][delegateFixedResult] - Fixed result GET_SIMILAR_DEBTORS totalSize: 1 -> fixedSize: 1
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.util.ListFragmenter][fragmentList] - Preparing [0] fragments in chunks of size [500]
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.util.ListFragmenter][fragmentList] - List fragment range[0-1]
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.SimilarDebtorsProcessor][processFragment] - Sending 'SimilarDebtors' message fragment[0-1]
2012-05-16 13:22:56,980 [jmsContainer-1] ERROR [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][onMessage] - JMSException caused by Message[ID:JBM-5a9ac639-f2a4-436c-8170-37378d8b606b], somenthing is wrong with the communication.
2012-05-16 13:22:56,982 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][onMessage] - Queue listener will be stopped.
2012-05-16 13:22:56,983 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][stopListener] - Listener successfully stopped
2012-05-16 13:22:56,984 [jmsContainer-1] DEBUG [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][errorTemplateSend] - Trying to send error message to AMS-WA error queue.
2012-05-16 13:22:56,985 [jmsContainer-1] DEBUG [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][errorTemplateSend] - Adapter id: 1
2012-05-16 13:22:57,014 [jmsContainer-1] ERROR [org.jboss.jms.client.container.ClosedInterceptor][invoke] - ClosedInterceptor.ClientSessionDelegate[ioy8-vyl6fa2h-1-hbl1g92h-qrmrca-a50o4c5]: method getTransacted() did not go through, the interceptor is CLOSED
2012-05-16 13:22:57,016 [jmsContainer-1] ERROR [org.springframework.jms.listener.DefaultMessageListenerContainer][rollbackOnExceptionIfNecessary] - Application exception overridden by rollback exception
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is org.jboss.jms.exception.MessagingNetworkFailureException; nested exception is org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:292)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:474)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:548)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
at ch.swissbytes.cipadapter.jms.request.CipAdapterBean.errorTemplateSend(CipAdapterBean.java:226)
at ch.swissbytes.cipadapter.jms.request.CipAdapterBean.onMessage(CipAdapterBean.java:160)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:506)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:463)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:240)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.jboss.jms.exception.MessagingNetworkFailureException
at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at $Proxy0.createConnection(Unknown Source)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.doCreateConnection(UserCredentialsConnectionFactoryAdapter.java:174)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.createConnection(UserCredentialsConnectionFactoryAdapter.java:149)
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:316)
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:270)
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:215)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:461)
... 12 more
Caused by: org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
at org.jboss.remoting.Client.connect(Client.java:1804)
at org.jboss.remoting.Client.connect(Client.java:652)
at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
... 32 more
Caused by: java.lang.Exception: Error setting up client lease
at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
at org.jboss.remoting.Client.connect(Client.java:1800)
... 37 more
Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:613)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
... 39 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:465)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:913)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:602)
... 42 more
2012-05-16 13:22:57,031 [jmsContainer-1] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer][handleListenerException] - Execution of JMS message listener failed
javax.jms.IllegalStateException: The object is closed
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:157)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$getTransacted_N1613179584734032131.invokeNext(ClientSessionDelegate$getTransacted_N1613179584734032131.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.getTransacted(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.getTransacted(JBossSession.java:154)
at org.springframework.jms.listener.AbstractMessageListenerContainer.rollbackOnExceptionIfNecessary(AbstractMessageListenerContainer.java:574)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:442)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:240)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
at java.lang.Thread.run(Thread.java:662)
2012-05-16 13:22:57,072 [jmsContainer-1] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:23:35,981 [org.springframework.scheduling.timer.TimerFactoryBean#0] DEBUG [ch.swissbytes.cipadapter.services.tasks.CheckQueueListenerStatus][run] - Checking queueListener status
And error was originated in this method:
private void initDatabaseProperties(final Session session) {
logger.info("Initializing database custom properties.");
properties.getProperty(CommonConstants.CIP_DATE_FORMAT_PROP);
final String dateFormat = DateUtil.VIEW_DATE_FORMAT;
logger.info("Setting NLS_DATE_FORMAT to : " + dateFormat);
final String queryString = "ALTER SESSION SET NLS_DATE_FORMAT = '" + StringUtils.trim(dateFormat) + "'";
session.createSQLQuery(queryString).executeUpdate();
}

I can't answer for what exceptions JBoss throws, but in general EOS on a socket is caused by exactly one thing: receiving a FIN from the peer as the result of a close or shutdown output by the peer.