We Keep Coding

HiveMQ Java library fails to automatically reconnect to broker

I'm using the HiveMQ library in my Java Spring application to connect to a Mosquitto instance as I find it more user-friendly compared to the Paho client. But something is going wrong with the automatic reconnection. From time to time the connection is lost and the application doesn't succeed in reconnecting (see logs 1). This can also be triggered by restarting the Mosquitto broker itself (see logs 2).
This is my client builder code with additional logging in the disconnect to check if the credentials are still correct:
client = MqttClient.builder()
.useMqttVersion5()
.identifier(identifier)
.serverHost(host)
.serverPort(port)
.sslWithDefaultConfig()
// https://www.hivemq.com/blog/hivemq-mqtt-client-features/reconnect-handling/
.automaticReconnectWithDefaultConfig()
.addDisconnectedListener(context -> logger.error("MQTT user {} with identifier {} on {}:{} has disconnected reason: {}",
username, identifier, host, port, context.getCause().getMessage()))
.buildAsync();
client.connectWith()
.simpleAuth()
.username(username)
.password(password.getBytes())
.applySimpleAuth()
.cleanStart(false)
.keepAlive(60)
.send();
1/ This is shown in my logs after the connection has been lost by the application itself:
2022-03-16 02:10:33.502 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for PINGRESP
2022-03-16 02:11:25.090 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
2022-03-16 02:12:27.200 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Timeout while waiting for CONNACK
2/ This is the logs after the broker has been restarted, some expected time-outs, but also in the end "not authorized":
2022-03-16 10:17:37.178 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: Server closed connection without DISCONNECT.
2022-03-16 10:17:48.441 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:00.747 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:10.625 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: No route to host: ***/***:8883
2022-03-16 10:18:26.845 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: io.netty.channel.ConnectTimeoutException: connection timed out: ***/***:8883
2022-03-16 10:18:42.584 ERROR 1 --- [client.mqtt-1-2] MqttConfig : MQTT user *** with identifier SERVICE on ***:8883 has disconnected reason: CONNECT failed as CONNACK contained an Error Code: NOT_AUTHORIZED.
In both cases, the connection is back to normal with an application restart.
any ideas?

It appears that your question is answered in this issue:
If you set the username and password on the connect call, they will not be stored and reused when the client reconnects (for security reasons).
The following code (from the issue linked above) demonstrates the approach:
Mqtt3Client.builder()
.identifier("ePCR mobile-" + currentTimeMillis())
.serverHost(config.getHost())
.serverPort(config.getPort())
.automaticReconnectWithDefaultConfig()
.simpleAuth()
.username(config.getUsername())
.password(config.getPassword())
.applySimpleAuth()
.buildRx();

Hazelcast Kubernetes Warning Unknown Protocol

I was using Zookeeper Hazelcast discovery but now i have changed it to Hazelcast-Kubernetes. It seems it is working fine but some times it prints warning message like :
2020-10-16 13:45:27.434 WARN 1 --- [.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection : [10.131.6.73]:5701 [dev] [3.12.7] Connection[id=6, /10.131.6.73:5701->/10.131.6.1:58546, qualifier=null, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=6, /10.131.6.73:5701->/10.131.6.1:58546, qualifier=null, endpoint=null, alive=true, type=NONE], thread=hz.switch-data-analytics.IO.thread-in-1
java.lang.IllegalStateException: Unknown protocol: OPT
at com.hazelcast.nio.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:107)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:135)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:369)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:354)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:280)
at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:235)
and
2020-10-16 13:45:27.438 WARN 1 --- [.IO.thread-in-2] com.hazelcast.nio.tcp.TcpIpConnection : [10.131.6.73]:5701 [dev] [3.12.7] Connection[id=7, /10.131.6.73:5701->/10.131.6.1:58548, qualifier=null, endpoint=null, alive=false, type=NONE] closed. Reason: Exception in Connection[id=7, /10.131.6.73:5701->/10.131.6.1:58548, qualifier=null, endpoint=null, alive=true, type=NONE], thread=hz.switch-data-analytics.IO.thread-in-2
java.lang.IllegalStateException: TLS handshake header detected, but plain protocol header was expected.
at com.hazelcast.nio.tcp.UnifiedProtocolDecoder.loadProtocol(UnifiedProtocolDecoder.java:125)
at com.hazelcast.nio.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:87)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:135)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:369)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:354)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:280)
at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:235)
I could not find any protocol usage like OPT or etc.
What should i do to clear this warning ?
Note :
I am using openshift and and my spring boot application use hazelcast distributed cache in 3 pods under one namespace.
Here is my setting :
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(true)
.setProperty("service-name", applicationProperties.getHazelcast().getServiceName())
.setProperty("namespace", applicationProperties.getPod().getNamespace());

There is a similar issue report in this GH Issue. As described there, adding the following part might solve the issue:
- containerPort: 5701
In any case, it might be a bug of hazelcast-kubernetes. Feel free to add the "steps to reproduce" to the following Hazelcast Kubernetes GH Issue.

Elasticsearch RestClient Connection reset by peer

I have in my AWS VPC a cluster of ES with 2 nodes. On top of those nodes I have a load balancer. In the same vpc I have a microservice that accesses Elasticsearch via RestHighLevelClient version 7.5.2 .
I create the client in the following manner :
public class ESClientWrapper {
#Getter
private RestHighLevelClient client;
public ESClientWrapper() throws IOException {
FileInputStream propertiesFile = new FileInputStream("/var/elastic.properties");
Properties properties = new Properties();
properties.load(propertiesFile );
RestClientBuilder builder = RestClient.builder(new HttpHost(
properties .getProperty("host"),
Integer.parseInt(properties.getProperty("port"))
));
this.client = new RestHighLevelClient(builder);
}
}
When my micro service doesn't get requests for a long time (12h..) there are occurrences when the first response that is sent (or a few after..) are getting the following error:
2020-09-09 07:03:13.106 INFO 1 --- [nio-8080-exec-1] c.a.a.services.CustomersMetadataService : Trying to add the following role : {role=a2}
2020-09-09 07:03:13.106 INFO 1 --- [nio-8080-exec-1] c.a.a.e.repositories.ESRepository : Trying to insert the following document to app-index : {role=a2}
2020-09-09 07:03:13.109 ERROR 1 --- [nio-8080-exec-1] c.a.a.e.dal.ESRepository : Failed to add customer : {role=a2}
java.io.IOException: Connection reset by peer
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:828) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:871) ~[elasticsearch-rest-high-level-client-7.5.2.jar!/:7.5.2]
....
....
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.35.jar!/:9.0.35]
at java.base/java.lang.Thread.run(Thread.java:836) ~[na:na]
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:na]
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:na]
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[na:na]
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245) ~[na:na]
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[na:na]
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[na:na]
at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
... 1 common frames omitted
2020-09-09 07:06:55.109 INFO 1 --- [nio-8080-exec-2] c.a.a.services.MyService : Trying to add the following role : {role=a2}
2020-09-09 07:06:55.109 INFO 1 --- [nio-8080-exec-2] c.a.a.e.repositories.ESRepository : Trying to insert the following document to index app-index: {role=a2}
2020-09-09 07:06:55.211 INFO 1 --- [nio-8080-exec-2] c.a.a.e.dal.ESRepository : IndexResponse[index=app-index,type=_doc,id=x532323272533321870287,version=1,result=created,seqNo=70,primaryTerm=1,shards={"total":2,"successful":2,"failed":0}]
As you can see, 3 minutes after the failed request the next request was successfully handeled by ES. What can kill the request ? I checked Elasticsearch logs and didn't see any indication for killing connection. The MS is in the same vpc as elastic so it isn't passing through any firewall that might kill it.
I found the following issue in github that suggested to increase the default connection timeout but I'm wondering if the issue here is really a timeout problem and if increasing the default time is really the best solution..
Also, I found this bug opened in their repo regarding the same problem but without any answers.
UPDATE
I noticed that even after 10 minutes my service is up this happens. My service started and sent a query to ES and everything worked well. After 10 minutes I sent insert request and it failed on connection reset by peer.

In the end I didn't find a problem in my configuration/implementation. It seems like a bug in the implementation of Elasticsearch's RestHighLevelClient.
I implemented a retry mechanism that wraps the RestHighLevelClient and retries the query if I get the same error. I used Spring #Retry annotation for this solution.

I was facing the same issue. Everything worked fine, but after some time a single request got refused.
The solution (in my case) was to set the keepalive property of the tcp connection with:
final RestClientBuilder restClientBuilder = RestClient.builder(...);
restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultIOReactorConfig(IOReactorConfig.custom()
.setSoKeepAlive(true)
.build()))
Found here:
https://github.com/elastic/elasticsearch/issues/65213

Java/Scala remote HDFS usage

I'm trying to connect to remote HDFS cluster. I've read some documentation and getting started's but didn't find a best solution how to do that.
Situation: I have HDFS on xxx-something.com. I can connect to it via SSH and everything works.
But what I'm trying to do, get the files from it to my local machine.
What I've done:
I've created core-site.xml in my conf folder (I'm creating Play! application). There I've changed fs.default.name config to hdfs://xxx-something.com:8020 (not sure about the port).
Then I'm trying to launch a simple test:
val conf = new Configuration()
conf.addResource(new Path("conf/core-site.xml"))
val fs = FileSystem.get(conf)
val status = fs.listStatus(new Path("/data/"))
And I'm getting errors:
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.trash.interval; Ignoring.
13:56:09.012 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: hadoop.tmp.dir; Ignoring.
13:56:09.013 [specs2.DefaultExecutionStrategy1] WARN org.apache.hadoop.conf.Configuration - conf/core-site.xml:a attempt to override final parameter: fs.checkpoint.dir; Ignoring.
13:56:09.022 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://xxx-something.com:8021
13:56:09.059 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:226)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:213)
at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:53)
at org.apache.hadoop.net.NetUtils.<clinit>(NetUtils.java:62)
Thanks in advance!
Update:
probably the port was wrong. Now I set it to 22, I'm still getting same errors, but after 3 times it does say:
14:01:01.877 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - Connecting to xxx-something.com/someIp:22
14:01:02.187 [specs2.DefaultExecutionStrategy1] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva sending #0
14:01:02.188 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva: starting, having connections 1
14:01:02.422 [IPC Client (47) connection to xxx-something.com/someIp:22 from britva] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to xxx-something.com/someIp:22 from britva got value #1397966893
And afterwards:
Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
java.io.IOException: Call to xxx-something.com/someIp:22 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1103)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:122)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:33)
at HdfsSpec$$anonfun$1$$anonfun$apply$3.apply(HdfsSpec.scala:17)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.executeBody(MyNotifierRunner.scala:16)
at testingSupport.specs2.MyNotifierRunner$$anon$2$$anon$1.execute(MyNotifierRunner.scala:16)
Caused by: java.io.EOFException
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:745)
What does it mean?

You'll need to find the fs.default.name property in the $HADOOP_HOME/conf/core-site.xml on the server running the Name Node (HDFS master) to get the correct port. It might be 8020, or it could be something else. That's what you should use. Make sure there's no firewall between you and the server that disallows connections on the port.

Can a SocketException end of file be caused by low memory in the server?

Hi dear community of java addicts.
I was getting these exceptions in a CentOs VM, probably running with low RAM and then I noted that the time was not correctly synchronized between the other VM needed to communicate with my nice component....
I was wondering to know, When ? Why ? How ? A SocketException: end of file is produced in a LINUX server...
These are my logs:
2012-05-16 13:22:41,863 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][initDatabaseProperties] - Initializing database custom properties.
2012-05-16 13:22:41,864 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][initDatabaseProperties] - Setting NLS_DATE_FORMAT to : DD/MM/YYYY HH24:MI:SS
2012-05-16 13:22:47,096 [Timer-2] ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker][handleException] - Got marshalling exception, exiting
java.net.SocketException: end of file
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:685)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.ConnectionValidator.doCheckConnectionWithoutLease(ConnectionValidator.java:828)
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:345)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2012-05-16 13:22:47,288 [Thread-2624] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:22:47,329 [Thread-2625] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:22:51,146 [Timer-4] WARN [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][run] - org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask#7a7385ac: detected failure on control connection Thread[control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623],5,] (5c4o020-jlorp4-h29d35xs-1-h2aawvkq-l2t: requesting new control connection
2012-05-16 13:22:51,159 [controlConnectionRecreate:control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623]] ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][createControlConnection] - unable to get secondary locator
org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:613)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.getSecondaryLocator(BisocketClientInvoker.java:640)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:230)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask$1.run(BisocketServerInvoker.java:1048)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:465)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:913)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:602)
... 5 more
2012-05-16 13:22:51,161 [controlConnectionRecreate:control: Socket[addr=ams-dev-bo.swissbytes.ch/192.168.0.190,port=11641,localport=57623]] ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker][run] - Unable to recreate control connection: InvokerLocator [null://ams-dev-bo.swissbytes.ch:11641/null]
java.io.IOException: unable to get secondary locator: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:235)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask$1.run(BisocketServerInvoker.java:1048)
2012-05-16 13:22:56,870 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - Query for the level[1] was executed.
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][doFilterResultsCotainCipAdress] - doFilterResultsCotainCipAdress() - Searching Cip Address: 1213194
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - cipAddress: 1213194 was NOT FOUND in Filter Results
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.services.RuleSetExecutorImpl][runFilterRec] - Current result size is not on filter's range : filter1 => [ 1 , 10 ] vs 0
2012-05-16 13:22:56,871 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.SimilarDebtorsProcessor][delegateFixedResult] - Fixed result GET_SIMILAR_DEBTORS totalSize: 1 -> fixedSize: 1
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.util.ListFragmenter][fragmentList] - Preparing [0] fragments in chunks of size [500]
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.util.ListFragmenter][fragmentList] - List fragment range[0-1]
2012-05-16 13:22:56,872 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.SimilarDebtorsProcessor][processFragment] - Sending 'SimilarDebtors' message fragment[0-1]
2012-05-16 13:22:56,980 [jmsContainer-1] ERROR [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][onMessage] - JMSException caused by Message[ID:JBM-5a9ac639-f2a4-436c-8170-37378d8b606b], somenthing is wrong with the communication.
2012-05-16 13:22:56,982 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][onMessage] - Queue listener will be stopped.
2012-05-16 13:22:56,983 [jmsContainer-1] INFO [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][stopListener] - Listener successfully stopped
2012-05-16 13:22:56,984 [jmsContainer-1] DEBUG [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][errorTemplateSend] - Trying to send error message to AMS-WA error queue.
2012-05-16 13:22:56,985 [jmsContainer-1] DEBUG [ch.swissbytes.cipadapter.jms.request.CipAdapterBean][errorTemplateSend] - Adapter id: 1
2012-05-16 13:22:57,014 [jmsContainer-1] ERROR [org.jboss.jms.client.container.ClosedInterceptor][invoke] - ClosedInterceptor.ClientSessionDelegate[ioy8-vyl6fa2h-1-hbl1g92h-qrmrca-a50o4c5]: method getTransacted() did not go through, the interceptor is CLOSED
2012-05-16 13:22:57,016 [jmsContainer-1] ERROR [org.springframework.jms.listener.DefaultMessageListenerContainer][rollbackOnExceptionIfNecessary] - Application exception overridden by rollback exception
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is org.jboss.jms.exception.MessagingNetworkFailureException; nested exception is org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:292)
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:474)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:548)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
at ch.swissbytes.cipadapter.jms.request.CipAdapterBean.errorTemplateSend(CipAdapterBean.java:226)
at ch.swissbytes.cipadapter.jms.request.CipAdapterBean.onMessage(CipAdapterBean.java:160)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:506)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:463)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:240)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.jboss.jms.exception.MessagingNetworkFailureException
at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:198)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
at $Proxy0.createConnection(Unknown Source)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.doCreateConnection(UserCredentialsConnectionFactoryAdapter.java:174)
at org.springframework.jms.connection.UserCredentialsConnectionFactoryAdapter.createConnection(UserCredentialsConnectionFactoryAdapter.java:149)
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:316)
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:270)
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:215)
at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:461)
... 12 more
Caused by: org.jboss.remoting.CannotConnectException: Error setting up client lease upon performing connect.
at org.jboss.remoting.Client.connect(Client.java:1804)
at org.jboss.remoting.Client.connect(Client.java:652)
at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
... 32 more
Caused by: java.lang.Exception: Error setting up client lease
at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:508)
at org.jboss.remoting.Client.setupClientLease(Client.java:1912)
at org.jboss.remoting.Client.connect(Client.java:1800)
... 37 more
Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:613)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:474)
... 39 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:192)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:465)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:913)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:602)
... 42 more
2012-05-16 13:22:57,031 [jmsContainer-1] WARN [org.springframework.jms.listener.DefaultMessageListenerContainer][handleListenerException] - Execution of JMS message listener failed
javax.jms.IllegalStateException: The object is closed
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:157)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$getTransacted_N1613179584734032131.invokeNext(ClientSessionDelegate$getTransacted_N1613179584734032131.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.getTransacted(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.getTransacted(JBossSession.java:154)
at org.springframework.jms.listener.AbstractMessageListenerContainer.rollbackOnExceptionIfNecessary(AbstractMessageListenerContainer.java:574)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:442)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:240)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868)
at java.lang.Thread.run(Thread.java:662)
2012-05-16 13:22:57,072 [jmsContainer-1] WARN [org.jboss.remoting.Client][removeListener] - unable to remove remote callback handler: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://ams-dev-bo.swissbytes.ch:4457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&marshaller=org.jboss.jms.wireformat.JMSWireFormat&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
2012-05-16 13:23:35,981 [org.springframework.scheduling.timer.TimerFactoryBean#0] DEBUG [ch.swissbytes.cipadapter.services.tasks.CheckQueueListenerStatus][run] - Checking queueListener status
And error was originated in this method:
private void initDatabaseProperties(final Session session) {
logger.info("Initializing database custom properties.");
properties.getProperty(CommonConstants.CIP_DATE_FORMAT_PROP);
final String dateFormat = DateUtil.VIEW_DATE_FORMAT;
logger.info("Setting NLS_DATE_FORMAT to : " + dateFormat);
final String queryString = "ALTER SESSION SET NLS_DATE_FORMAT = '" + StringUtils.trim(dateFormat) + "'";
session.createSQLQuery(queryString).executeUpdate();
}

I can't answer for what exceptions JBoss throws, but in general EOS on a socket is caused by exactly one thing: receiving a FIN from the peer as the result of a close or shutdown output by the peer.