We Keep Coding

MQTT5 message is lost on HiveMQ Cloud when correlationData is added

In a Java server application, we want to use the correlationData, which is part of MQTT5, so we can use this in the reply message to link and validate it when the reply is received.
I'm using the hivemq-mqtt-client library 1.2.2 and connecting to HiveMQ Cloud.
The connection is made like this:
private Mqtt5AsyncClient client;
public MqttConfig(Environment environment) {
client = MqttClient.builder()
.useMqttVersion5()
.identifier("TestServer")
.serverHost(MQTT_SERVER)
.serverPort(MQTT_PORT)
.sslWithDefaultConfig()
.automaticReconnectWithDefaultConfig()
.buildAsync();
client.connectWith()
.simpleAuth()
.username(MQTT_USER)
.password(MQTT_PASSWORD.getBytes())
.applySimpleAuth()
.send()
.whenComplete((connAck, throwable) -> {
if (throwable != null) {
logger.error("Could not connect to MQTT: {}", throwable.getMessage());
} else {
logger.info("Connected to MQTT: {}", connAck.getReasonCode());
}
});
}
public Mqtt5AsyncClient getClient() {
return client;
}
Sending a message is done with this method:
mqttConfig.getClient()
.publishWith()
.topic(destinationTopic)
//.correlationData(correlationData.getBytes())
.responseTopic(responseTopic)
.payload(message.getBytes())
.qos(MqttQos.AT_LEAST_ONCE)
.send()
.whenComplete((result, throwable) -> {
if (throwable != null) {
logger.info("Error sending to '{}': {} - {}", destinationTopic, message, throwable.getMessage());
} else {
logger.info("Message sent to '{}': {} - {}", destinationTopic, message, result);
}
});
When monitoring the messages on http://www.hivemq.com/demos/websocket-client/ and on a subscriber, messages are only received when the line with `correlationData()' is commented, as you can see above.
Both with or without that line, the application logs a successful sending, e.g. with correlation data enabled:
Message sent to 'server/test': testMessage - MqttQos2Result{publish=MqttPublish{topic=server/test, payload=11byte, qos=EXACTLY_ONCE, retain=false, responseTopic=server/test/reply, correlationData=6byte}, pubRec=MqttPubRec{packetIdentifier=1}}
Any idea why the additional correlationData seems to cause that they are not shown on the websocket test page, and are not received on any of the subscribers?
As an experiment, I used the paho 5 library instead of the HiveMQ library with the following code, but had exactly the same behavior and needed to disable the line to see messages passing:
MqttProperties properties = new MqttProperties();
//properties.setCorrelationData(correlationData.getBytes());
MqttMessage mqttMessage = new MqttMessage();
mqttMessage.setQos(1);
mqttMessage.setPayload(message.getBytes());
mqttMessage.setProperties(properties);
try {
mqttConfig.getClient().publish(destinationTopic, mqttMessage);
logger.info("Message was sent to '{}': {}", destinationTopic, message);
} catch (MqttException ex) {
logger.error("Error sending to '{}': {} - {}", destinationTopic, message, ex.getMessage());
}

This behaviour has now been fixed by HiveMQ Cloud.

MQTT: Does paho store failed outbound messages

We are building a project in Java which is using MQTT to communicate with other services using Paho client library. While running the application, I observed that if the broker is down, Paho is not caching failed outbound messages.
Is there a way to make Paho cache failed outbound messages?
I am using Paho version 1.2.5
org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.5
Here is how I have configured it:
public void connect(MqttCallback mqttCallback, int maxInflight) {
try {
persistence = new MqttDefaultFilePersistence(mqttPersistenceLocation);
mqttClient = new MqttAsyncClient(broker, clientId, persistence);
connOpts = new MqttConnectOptions();
// always set cleanSession to false if QoS 1 or 2 is used.
connOpts.setCleanSession(false);
connOpts.setAutomaticReconnect(true);
connOpts.setMaxInflight(maxInflight);
mqttClient.setCallback(mqttCallback);
LOGGER.info("Connecting to broker: " + broker + " with maxInflight: " + maxInflight);
mqttClient.connect(connOpts).waitForCompletion();
LOGGER.info("Connected");
} catch (Exception e) {
LOGGER.error("Error while connecting to MQTT:" + e);
}
}

Paho MQTT client local address and port

My question is simple.
I want to know how to fix port client has.
According to Eclipse documents and IBM's, User has to fix broker address(this is absolutely natural). But There are no mentions about way of how to fix client site port.
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.javadoc.doc/WMQMQxrClasses/com/ibm/micro/client/mqttv3/MqttClient.html
MQTT must be also on TCP Layer, so I think it's possible to fix port.
If you have ideas, let me know.
Thanks

Under normal circumstance you don't set the source port for TCP connections, you just let the OS pick a random free port.
If you fix the source port then you can only ever run 1 instance of the client at a time on a given machine and if you get a connection failure you have to wait for the TCP stack to free that socket up again before you can reconnect.
If for some reason you REALLY NEED to fix the source port then you could probably write a custom javax.net.SocketFactory implementation that you can hard code the source port. Then pass this in as part of the MQTTConnectOptions object, but again I'm really struggling to come up with a reason this is a good idea.

You don't specify the client's port, as it is choosen by the OS, just as with any client to server connection, like you don't need to do that with a HTTP connection either.
You specify the IP address and port of the MQTT broker in the URL you pass in the form of tcp://<address>:<port>, for example tcp://localhost:4922.
The code below shows how I connect a Paho client in an OSGi context, where all connections parameters are retrieved from the bundle context.
private void configureMqtt(BundleContext context) throws IOException, MqttException {
String broker = context.getProperty("mqtt.broker");
if (broker == null) {
throw new ServiceException("Define mqtt.broker");
}
String client = context.getProperty("mqtt.clientname");
if (client == null) {
throw new ServiceException("Define mqtt.clientname");
}
String dir = context.getProperty("mqtt.persistence");
if (dir == null) {
dir = "mqtt";
};
File directory = context.getDataFile(dir);
directory.mkdirs();
logger.config(() -> String.format("MQTT broker: %s, clientname: %s persistence: %s", broker, client, directory.getAbsolutePath()));
connectOptions = new MqttConnectOptions();
connectOptions.setWill(GARAGE + "/door", new byte[0], 0, true);
String ka = context.getProperty("mqtt.keepalive");
if (ka != null) {
connectOptions.setKeepAliveInterval(Integer.valueOf(ka));
}
persistence = new MqttDefaultFilePersistence(directory.getCanonicalPath());
mqttClient = new MqttAsyncClient(broker, client, persistence);
mqttClient.setCallback(this);
connect();
}
private void connect() {
logger.fine("Connecting to MQTT broker");
try {
IMqttToken token = mqttClient.connect(connectOptions);
IMqttActionListener listener = new IMqttActionListener() {
#Override
public void onSuccess(IMqttToken asyncActionToken) {
logger.log(Level.INFO, "Connected to MQTT broker");
}
#Override
public void onFailure(IMqttToken asyncActionToken, Throwable exception) {
logger.log(Level.WARNING, "Could not connect to MQTT broker, retrying in 3 seconds", exception);
service.schedule(this::connect, 3, TimeUnit.SECONDS);
}
};
token.setActionCallback(listener);
} catch (MqttException e) {
logger.log(Level.SEVERE, "Cannot reconnect to MQTT broker, giving up", e);
}
}

Smack 4.1.0 GCM CCS stops responding after a while

Have implemented gcm ccs for chat module and i am able to send and receive messages. Below is the main connection module,
config = XMPPTCPConnectionConfiguration.builder()
.setServiceName("gcm-pesu.googleapis.com")
.setPort(GCM_PORT)
.setHost(GCM_SERVER)
.setCompressionEnabled(false)
.setConnectTimeout(30000)
.setSecurityMode(SecurityMode.ifpossible)
.setSendPresence(false)
.setSocketFactory(SSLSocketFactory.getDefault())
.build();
connection = new XMPPTCPConnection(config);
connection.connect();
Roster roster = Roster.getInstanceFor(connection);
roster.setRosterLoadedAtLogin(false);
connection.addConnectionListener(new LoggingConnectionListener());
// Handle incoming packets
connection.addAsyncStanzaListener(new MyStanzaListener(), new MyStanzaFilter());
// Log all outgoing packets
connection.addPacketInterceptor(new MyStanzaInterceptor(), new MyStanzaFilter());
connection.login(mProjectId + "#gcm.googleapis.com", mApiKey);
logger.info("logged in: " + mProjectId);
PingManager pm = PingManager.getInstanceFor(connection);
pm.setPingInterval(300);
pm.pingMyServer();
pm.registerPingFailedListener(new PingFailedListener() {
#Override
public void pingFailed() {
connection.disconnect();
logger.error("GCM CCS, Ping failed !!");
}
});
The problem i am running into is not receiving any message from GCM, sent by client device after a while. Though, the heartbeat looks normal and i do get pong from GCM even in that case. Is it something to do with SSL ?
Have handled connection draining case as follows,
String controlType = (String) jsonObject.get("control_type");
volatile boolean connectionDraining = false;
if ("CONNECTION_DRAINING".equals(controlType)) {
connectionDraining = true;
try {
connection.disconnect();
connect();
connectionDraining = false;
} catch (Exception e) {
logger.error("Error establishing new connection after draining ", e);
}
}

Implemented queue of channels when one of it is draining.
private Deque<Channel> channels;
protected void handleControlMessage(Map<String, Object> jsonObject) {
logger.info("Control message : " + jsonObject);
String controlType = (String) jsonObject.get("control_type");
if ("CONNECTION_DRAINING".equals(controlType)) {
connectionDraining = true;
}
}
Create new channel while sending message
public void sendDownstreamMessage(String jsonRequest) {
Channel channel = channels.peekFirst();
try {
if (channel.connectionDraining) {
synchronized (channels) {
channel = channels.peekFirst();
if (channel.connectionDraining) {
channels.addFirst(connect());
channel = channels.peekFirst();
}
}
}
channel.send(jsonRequest);
} catch (Exception e) {
logger.error("Message not sent. Error in connecting :", e);
}
}
GCM will take care of closing the other. This resolved the issue.

I believe you're facing a common case using gcm css that is not very visible in the documentation.
If you look in the doc, Control Messages you'll read:
Periodically, CCS needs to close down a connection to perform load balancing. Before it closes the connection, CCS sends a CONNECTION_DRAINING message to indicate that the connection is being drained and will be closed soon. "Draining" refers to shutting off the flow of messages coming into a connection, but allowing whatever is already in the pipeline to continue. When you receive a CONNECTION_DRAINING message, you should immediately begin sending messages to another CCS connection, opening a new connection if necessary. You should, however, keep the original connection open and continue receiving messages that may come over the connection (and ACKing them)—CCS handles initiating a connection close when it is ready.

Eclipse paho Mqtt:Getting java.io.EOF Exception

I am getting "java.io.eof" exception,when i am trying to subscribe mqtt client. I am using eclipse paho library and using mosquitto broker. I am not getting any answer of this,so please help me why this happens ?
Mqtt connection and subscribe
I am using this code for connecting and subscribing to mosquitto
private void buildClient(String clientId){
log.debug("Connecting... "+clientId);
try {
mqttClient = new MqttClient(envConfiguration.getBrokerUrl(), clientId,new MemoryPersistence());
System.out.println(mqttClient.isConnected());
} catch (MqttException e) {
log.debug("build client stopped due to "+e.getCause());
}
chatCallback = new ChatCallback(this.userService,this);
mqttClient.setCallback(chatCallback);
mqttConnectOptions = new MqttConnectOptions();
mqttConnectOptions.setCleanSession(true);
}
#Override
public void connect(String clientId,String topic) {
try{
if(mqttClient == null || !mqttClient.getClientId().equals(clientId)){
buildClient(clientId);
mqttClient.connect(mqttConnectOptions);
subscribe(clientId,topic);
}
}catch (Exception e) {
log.debug("connection attempt failed "+ e.getCause() + " trying...");
}
}
#Override
public void subscribe(String clientId,String topic) throws MqttException {
if(mqttClient != null && mqttClient.isConnected()){
mqttClient.subscribe(topic,0);
/*try {
log.debug("Subscribing... with client id :: " + clientId + "topic");
mqttClient.subscribe(topic,2);
} catch (MqttException e) {
log.debug("subscribing error.."+e.getLocalizedMessage());
}*/
}
}
}
And mqtt call back
#Override
public void connectionLost(Throwable arg0) {
log.debug("Connection lost... attampting retrying due to "
+ arg0);
arg0.printStackTrace();
// chatServiceimpl.connect();
}
#Override
public void deliveryComplete(IMqttDeliveryToken arg0) {
log.debug("delivered message" + arg0);
// TODO Auto-generated method stub
}
#Override
public void messageArrived(String arg0, MqttMessage arg1) throws Exception {
log.debug("Message recived..." + arg1.toString());
userService.saveChat(arg1.toString());
}
I am facing this error when i am subscribing to mosquitto
Error logs
2015-11-30/18:19:00.877 [MQTT Call: 25287] DEBUG c.s.s.ChatCallback: Message recived...{ "id":"37153topic25287T1448886285.79573", "from":"37153", "to":"25287", "chatBody":[{"type": "text", "message":"The fact "}]}
2015-11-30/18:19:00.878 [MQTT Call: 25287] DEBUG c.s.s.u.UserService: Saving chat...
2015-11-30/18:19:00.883 [MQTT Call: 25287] DEBUG c.s.s.u.UserService: Get user by id::37153
2015-11-30/18:19:00.885 [MQTT Call: 25287] DEBUG c.s.s.u.UserService: Get user by id::25287
2015-11-30/18:19:00.886 [MQTT Rec: 25287] DEBUG c.s.s.ChatCallback: Connection lost... attampting retrying due to Connection lost (32109) - java.io.EOFException
Connection lost (32109) - java.io.EOFException
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:138)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:56)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:100)
... 1 more
Mosquitto Logs
1448889230: Client 25287 disconnected.
1448889230: New client connected from 192.168.2.63 as 25287 (c0, k60).
1448889231: New connection from 192.168.2.242 on port 1883.
1448889231: Client 25287 already connected, closing old connection.
1448889231: Client 25287 disconnected.
1448889231: New client connected from 192.168.2.242 as 25287 (c1, k60).
1448889231: New connection from 192.168.2.63 on port 1883.
1448889231: Client 25287 already connected, closing old connection.
1448889231: Client 25287 disconnected.
1448889231: New client connected from 192.168.2.63 as 25287 (c0, k60).
1448889269: New connection from 192.168.2.242 on port 1883.

You have multiple clients connecting to the broker with the same clientid, this is not allowed and as one connects the broker will disconnect the currently connected client.
If both clients have automatic reconnection logic then they will just continue to kick each other off.
Change the client id on one of the clients.

As hardillb mentioned above, you have multiple clients connecting. Server (mosquitto) will disconnect the old connection when it receives a connect request from the same client again.
use the isConnected() method on MqttClient object to know if its connected. for eg.
if (! m_client.isConnected()) {
// reconnect
}