We Keep Coding

Azure ServiceBusSessionReceiverAsyncClient - Mono instead of Flux

I have a Spring Boot app, where I receive one single message from a Azure Service Bus queue session.
The code is:
#Autowired
ServiceBusSessionReceiverAsyncClient apiMessageQueueIntegrator;
.
.
.
Mono<ServiceBusReceiverAsyncClient> receiverMono = apiMessageQueueIntegrator.acceptSession(sessionid);
Disposable subscription = Flux.usingWhen(receiverMono,
receiver -> receiver.receiveMessages(),
receiver -> Mono.fromRunnable(() -> receiver.close()))
.subscribe(message -> {
// Process message.
logger.info(String.format("Message received from quque. Session id: %s. Contents: %s%n", message.getSessionId(),
message.getBody()));
receivedMessage.setReceivedMessage(message);
timeoutCheck.countDown();
}, error -> {
logger.info("Queue error occurred: " + error);
});
As I am receiving only one message from the session, I use a CountDownLatch(1) to dispose of the subscription when I have received the message.
The documentation of the library says that it is possible to use Mono.usingWhen instead of Flux.usingWhen if I only expect one message, but I cannot find any examples of this anywhere, and I have not been able to figure out how to rewrite this code to do this.
How would the pasted code look if I were to use Mono.usingWhen instead?

Thank you conniey. Posting your suggestion as an answer to help other community members.
By default receiveMessages() is a Flux because we imagine the messages from a session to be "infinitely long". In your case, you only want the first message in the stream, so we use the next() operator.
The usage of the countdown latch is probably not the best approach. In the sample, we had one there so that the program didn't end before the messages were received. .subscribe is not a blocking call, it sets up the handlers and moves onto the next line of code.
Mono<ServiceBusReceiverAsyncClient> receiverMono = sessionReceiver.acceptSession("greetings-id");
Mono<ServiceBusReceivedMessage> singleMessageMono = Mono.usingWhen(receiverMono,
receiver -> {
// Anything you wish to do with the receiver.
// In this case we only want to take the first message, so we use the "next" operator. This returns a
// Mono.
return receiver.receiveMessages().next();
},
receiver -> Mono.fromRunnable(() -> receiver.close()));
try {
// Turns this into a blocking call. .block() waits indefinitely, so we have a timeout.
ServiceBusReceivedMessage message = singleMessageMono.block(Duration.ofSeconds(30));
if (message != null) {
// Process message.
}
} catch (Exception error) {
System.err.println("Error occurred: " + error);
}
You can refer to GitHub issue:ServiceBusSessionReceiverAsyncClient - Mono instead of Flux

Java: Azure Service Bus Queue Receiving messsages with sessions

I'm writing code in java (using Azure SDK for Java), I have a Service bus queue that contains sessionful messages. I want to receive those messages and process them to another place.
I make a connection to the Queue by using QueueClient, and then I use registerSessionHandler to process through the messages (code below).
The problem is that whenever a message is received, I can print all details about it including the content, but it is printed 10 times and after each time it prints an Exception.
(printing 10 times: I understand that this is because there is a 10 times retry policy before it throws the message to the Dead letter queue and goes to the next message.)
The Exception says
> USERCALLBACK-Receiver not created. Registering a MessageHandler creates a receiver.
The output with the Exception
But I'm sure that the SessionHandler does the same thing as MessageHandler but includes support for sessions, so it should create a receiver since it receives messages. I have tried to use MessageHandler but it won't even work and stops the whole program because it doesn't support sessionful messages, and the ones I receive have sessions.
My problem is understanding what the Exception wants me to do, and how can I fix the code so it won't give me any exceptions? Does anyone have suggestions on how to improve the code? or other methods that do the same thing?
QueueClient qc = new QueueClient(
new ConnectionStringBuilder(connectionString),
ReceiveMode.PEEKLOCK);
qc.registerSessionHandler(
new ISessionHandler() {
#Override
public CompletableFuture<Void> onMessageAsync(IMessageSession messageSession, IMessage message) {
System.out.printf(
"\nMessage received: " +
"\n --> MessageId = %s " +
"\n --> SessionId = %s" +
"\n --> Content Type = %s" +
"\n --> Content = \n\t\t %s",
message.getMessageId(),
messageSession.getSessionId(),
message.getContentType(),
getMessageContent(message)
);
return qc.completeAsync(message.getLockToken());
}
#Override
public CompletableFuture<Void> OnCloseSessionAsync(IMessageSession iMessageSession) {
return CompletableFuture.completedFuture(null);
}
#Override
public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
System.out.println("\n Exception " + exceptionPhase + "-" + throwable.getMessage());
}
},
new SessionHandlerOptions(1, true, Duration.ofMinutes(1)),
Executors.newSingleThreadExecutor()
);
(The getMessageContent(message) method is a separate method, for those interested:)
public String getMessageContent(IMessage message){
List<byte[]> content = message.getMessageBody().getBinaryData();
StringBuilder sb = new StringBuilder();
for (byte[] b : content) {
sb.append(new String(b)
);
}
return sb.toString();
}

For those who wonder, I managed to solve the problem!
It was simply done by using Azure Functions ServiceBusQueueTrigger, it will then listen to the Service bus Queue and process the messages. By setting isSessionsEnabled to true, it will accept sessionful messages as I wanted :)
So instead of writing more than 100 lines of code, the code looks like this now:
public class Function {
#FunctionName("QueueFunction")
public void run(
#ServiceBusQueueTrigger(
name = "TriggerName", //Any name you choose
queueName = "queueName", //QueueName from the portal
connection = "ConnectionString", //ConnectionString from the portal
isSessionsEnabled = true
) String message,
ExecutionContext context
) {
// Write the code you want to do with the message here
// Using the variable messsage which contains the messageContent, messageId, sessionId etc.
}
}

Some messages are lost in SQS

I'm sending and receiving messages with SQS and Java SDK. Almost all messages are working fine, but some of them are lost and I can't understand why. This is the code to send a message:
final SendMessageRequest msg = new SendMessageRequest(
this.queueUrl, data.toString()
).withMessageGroupId(projectId);
final Map<String, MessageAttributeValue> attrs = new HashMap<>(1);
// message signature - HMAC calculated from message body
final String signature = // calculating signature
attrs.put(
"signature",
new MessageAttributeValue()
.withDataType("String")
.withStringValue(signature)
);
attrs.put(
"project",
new MessageAttributeValue()
.withDataType("String")
.withStringValue(projectId)
);
attrs.put(
"priority",
new MessageAttributeValue()
.withDataType("String")
.withStringValue(priority)
);
msg.setMessageDeduplicationId(
String.format("%s:%s", projectId, signature)
);
msg.setMessageAttributes(attrs);
Logger.debug(this, "sending message: %s", msg);
final SendMessageResult res = this.sqs.sendMessage(msg);
Logger.info(
this,
"message '%s' (%s) was sent: %s",
data.id(), data.type(), res
);
and the code for receiving (running in a loop):
final List<Message> messages = this.sqs.receiveMessage(
new ReceiveMessageRequest(url)
.withMessageAttributeNames(
"project", "signature", "expires", "priority"
)
.withVisibilityTimeout(
(int) Duration.ofMinutes(2L).getSeconds()
)
.withMaxNumberOfMessages(8)
).getMessages();
Logger.info(this, "received %d messages", messages.size());
for (final Message message : messages) {
Logger.debug(this, "received message: %s", message);
// actual logic here
this.sqs.deleteMessage(
new DeleteMessageRequest()
.withQueueUrl(this.queue)
.withReceiptHandle(message.getReceiptHandle())
);
}
The problem is that I'm able to receive some messages, but some not (always same type of data data.type()). The code to send and receive is the same for all messages.
App logs:
sending message: {QueueUrl: https://sqs.us-east-1.amazonaws.com/0000000/my-queue.fifo, MessageBody:some-unique-body,MessageAttributes: {priority={StringValue: HIGH,StringListValues: [],BinaryListValues: [],DataType: String}, signature={StringValue: EXTvx7WWrZ7uTU63szJ2C4VN/6ZOiw/wKL83qW7V3i0=,StringListValues: [],BinaryListValues: [],DataType: String}, project={StringValue: PMO,StringListValues: [],BinaryListValues: [],DataType: String}},MessageDeduplicationId: PMO:EXTvx7WWrZ7uTU63szJ2C4VN/6ZOiw/wKL83qW7V3i0=,MessageGroupId: PMO}
message 'e29baf85-7be4-449b-824f-e405c59cf7c4' ([test]) was send: {MD5OfMessageBody: 9d6e98e0e85c8f5ca7cc4c23378dc14b,MD5OfMessageAttributes: fe94ccb1b405588e0691c91392d2c8ea,MessageId: c8ce957a-93c8-49ef-9a08-9e12cb5952b4,SequenceNumber: 18848975946990575872}
receiving messages: limit=8; timout=2m
received 0 messages
receiving messages: limit=8; timout=2m
received 0 messages
receiving messages: limit=8; timout=2m
received 0 messages
receiving messages: limit=8; timout=2m
received 0 messages
I checked that deduplication-id of these messages are always different and it has different content. How can I debug this issue deeper?
Update:
It seems my queue is full of messages, but I can't receive them:
$ aws sqs get-queue-attributes --queue-url="$QURL" --attribute-names=ApproximateNumberOfMessages
{
"Attributes": {
"ApproximateNumberOfMessages": "1490"
}
}
It's strange, since some messages are delivered in a few seconds even with full queue.
Update2:
I tried to use long-polling, but it didn't help.

Check whether is there any inflight requests for that queue in AWS console, as i can see you are using the same messagegroupid for all the messages, so the messages will be processed sequentially, if any one message is received by any consumer, until the acknowledgement of the message or visibility timeout expires, no other consumers will receive any of the messages.

HTTP Node.js Java API

I am creating a Node.js Java backend. The Node.js middleware receives HTTP requests from an Android application and then relays it to the Java code. The reason for choosing this technologies is to create a highly scalable backend from scratch.
I want the Node.js api to receive the HTTP requests, pass it to the Java-side of the backend, the Java code does its calculations, sends back the result to the Node.js API and then finishes the process by sending the result back to the Android application.
I can receive and parse HTTP requests:
var BodyParser = require('body-parser');
var Express = require('express');
var JavaClient = require('./NodeJavaBridge.js');
var JavaClientInstance = new JavaClient();
var app = Express();
///// Receive message logic \\\\\
app.use(BodyParser.json());
app.post('/', function (request, response)
{
var task = request.body;
response.writeHead(200, { 'content-type': 'text/plain' });
var otherObject = { SomeData: 1234 };
var json = JSON.stringify({
data: otherObject
});
response.end(json);
});
console.log("START --> Java Client Instance");
JavaClientInstance.run();
app.listen(8080); //to port on which the express server listen
console.log("Server listening on: " + 8080);
I can also send and receive data between Node.js and Java:
var Util = require('util');
var EventEmitter = require('events').EventEmitter;
var ChildProc = require('child_process');
var JavaClient = function () {
var _self = this;
// The child process object we get when we spawn the java process
var _javaSpawn = null;
// buffer for receiving messages in part and piecing them together later
var _receiveBuffer = null;
// The location of java and the - we're making these public because maybe
// we want to change them in the user of this module.
_self.javaPath = 'java';
_self.jarPath = 'C:/Dev/Backend_Java.jar';
_self.verbose = true;
// list of events emitted - for informational purposes
_self.events = [
'spawn', 'message', 'exception', 'unknown', 'sent', 'java_error',
// Response messages that then become events themselves
'Error', 'Hello', 'Info'
];
/**
* Attach our own event handler to reply to the hello message.
* This is just a convenience part of the protocol so that clients don't have to do it.
* Also connects if connection data was supplied.
*/
_self.on('Hello', function () {
_self.sendHello();
});
/**
* Executes the java process to begin sending and receiving communication
*/
_self.run = function () {
// Invoke the process
_javaSpawn = ChildProc.spawn(_self.javaPath, ['-jar', _self.jarPath]);
// Wire up events
_javaSpawn.stdout.on('data', onData);
_javaSpawn.stderr.on('data', onJavaError);
_javaSpawn.on('exit', function (code) {
console.log("The java program exited with code " + code + ".");
});
// Emit our own event to indicate to others that we have spawned
_self.emit('spawn', _javaSpawn);
}
// sends the hello request message
_self.sendHello = function () {
sendMessage(
{
messageName : 'Hello',
version : '1.1'
});
}
// sends a message that will be echoed back as an Info message
_self.sendEcho = function (message) {
sendMessage(
{
messageName : "Echo",
message : message
});
}
// sends a message telling the java app to exit
_self.sendGoodbye = function () {
sendMessage(
{
"messageName" : "Goodbye"
});
}
/**
* Sends a message object as a JSON encoded string to the java application for processing.
*/
function sendMessage(aMsg)
{
// convert to json and prepare buffer
var aJsonString = JSON.stringify(aMsg);
var lByteLength = Buffer.byteLength(aJsonString);
var lMsgBuffer = new Buffer(4 + lByteLength);
// Write 4-byte length, followed by json, to buffer
lMsgBuffer.writeUInt32BE(lByteLength, 0);
lMsgBuffer.write(aJsonString, 4, aJsonString.length, 'utf8');
// send buffer to standard input on the java application
_javaSpawn.stdin.write(lMsgBuffer);
_self.emit('sent', aMsg);
}
/**
* Receive data over standard input
*/
function onData(data)
{
// Attach or extend receive buffer
_receiveBuffer = (null == _receiveBuffer) ? data : Buffer.concat([_receiveBuffer, data]);
// Pop all messages until the buffer is exhausted
while (null != _receiveBuffer && _receiveBuffer.length > 3)
{
var size = _receiveBuffer.readInt32BE(0);
// Early exit processing if we don't have enough data yet
if ((size + 4) > _receiveBuffer.length)
{
break;
}
// Pull out the message
var json = _receiveBuffer.toString('utf8', 4, (size + 4));
// Resize the receive buffer
_receiveBuffer = ((size + 4) == _receiveBuffer.length) ? null : _receiveBuffer.slice((size + 4));
// Parse the message as a JSON object
try
{
var msgObj = JSON.parse(json);
// emit the generic message received event
_self.emit('message', msgObj);
// emit an object-type specific event
if ((typeof msgObj.messageName) == 'undefined')
{
_self.emit('unknown', msgObj);
}
else
{
_self.emit(msgObj.messageName, msgObj);
}
}
catch (ex)
{
_self.emit('exception', ex);
}
}
}
/**
* Receive error output from the java process
*/
function onJavaError(data)
{
_self.emit('java_error', data.toString());
}
}
// Make our JavaClient class an EventEmitter
Util.inherits(JavaClient, EventEmitter);
// export our class
module.exports = JavaClient;
My problem: How do I let the POST request send a request to my JavaClient instance, wait for a response and then send it back to origin (Android app).
Here is an example of how I am trying to get the logic working:
var client = require('./JavaClient');
var instance = new client();
instance.on('message', function(msg) {
console.log('Received a message...');
console.log(msg);
});
instance.on('sent', function(msg) {
console.log('Sent a message...');
console.log(msg);
});
instance.on('Info', function(msg) {
console.log("Received info");
console.log(msg.message);
});
(function() {
// Start it up (Hello exchanges happen)
instance.run();
// Receive acknowledgement of hello
instance.once('Info', function() {
// Try echoing something
instance.sendEcho("ECHO!");
});
})();
If I should make something more clear please let me know (it's really late and I assume that my writing capabilities is taking a dive). I would appreciate any answer/suggestion/thisisabadidea type of comments.
Thanks!

var Util = require('util');
var EventEmitter = require('events').EventEmitter;
var ChildProc = require('child_process');
var JavaClient = function () {
var _self = this;
// The child process object we get when we spawn the java process
var _javaSpawn = null;
// buffer for receiving messages in part and piecing them together later
var _receiveBuffer = null;
// The location of java and the - we're making these public because maybe
// we want to change them in the user of this module.
_self.javaPath = 'java';
_self.jarPath = 'C:/Dev/Backend_Java.jar';
_self.verbose = true;
// list of events emitted - for informational purposes
_self.events = [
'spawn', 'message', 'exception', 'unknown', 'sent', 'java_error',
// Response messages that then become events themselves
'Error', 'Hello', 'Info'
];
/**
* Attach our own event handler to reply to the hello message.
* This is just a convenience part of the protocol so that clients don't have to do it.
* Also connects if connection data was supplied.
*/
_self.on('Hello', function () {
_self.sendHello();
});
/**
* Executes the java process to begin sending and receiving communication
*/
_self.run = function () {
// Invoke the process
_javaSpawn = ChildProc.spawn(_self.javaPath, ['-jar', _self.jarPath]);
// Wire up events
_javaSpawn.stdout.on('data', onData);
_javaSpawn.stderr.on('data', onJavaError);
_javaSpawn.on('exit', function (code) {
console.log("The java program exited with code " + code + ".");
});
// Emit our own event to indicate to others that we have spawned
_self.emit('spawn', _javaSpawn);
}
// sends the hello request message
_self.sendHello = function () {
sendMessage(
{
messageName : 'Hello',
version : '1.1'
});
}
// sends a message that will be echoed back as an Info message
_self.sendEcho = function (message) {
sendMessage(
{
messageName : "Echo",
message : message
});
}
// sends a message telling the java app to exit
_self.sendGoodbye = function () {
sendMessage(
{
"messageName" : "Goodbye"
});
}
/**
* Sends a message object as a JSON encoded string to the java application for processing.
*/
function sendMessage(aMsg)
{
// convert to json and prepare buffer
var aJsonString = JSON.stringify(aMsg);
var lByteLength = Buffer.byteLength(aJsonString);
var lMsgBuffer = new Buffer(4 + lByteLength);
// Write 4-byte length, followed by json, to buffer
lMsgBuffer.writeUInt32BE(lByteLength, 0);
lMsgBuffer.write(aJsonString, 4, aJsonString.length, 'utf8');
// send buffer to standard input on the java application
_javaSpawn.stdin.write(lMsgBuffer);
_self.emit('sent', aMsg);
}
/**
* Receive data over standard input
*/
function onData(data)
{
// Attach or extend receive buffer
_receiveBuffer = (null == _receiveBuffer) ? data : Buffer.concat([_receiveBuffer, data]);
// Pop all messages until the buffer is exhausted
while (null != _receiveBuffer && _receiveBuffer.length > 3)
{
var size = _receiveBuffer.readInt32BE(0);
// Early exit processing if we don't have enough data yet
if ((size + 4) > _receiveBuffer.length)
{
break;
}
// Pull out the message
var json = _receiveBuffer.toString('utf8', 4, (size + 4));
// Resize the receive buffer
_receiveBuffer = ((size + 4) == _receiveBuffer.length) ? null : _receiveBuffer.slice((size + 4));
// Parse the message as a JSON object
try
{
var msgObj = JSON.parse(json);
// emit the generic message received event
_self.emit('message', msgObj);
// emit an object-type specific event
if ((typeof msgObj.messageName) == 'undefined')
{
_self.emit('unknown', msgObj);
}
else
{
_self.emit(msgObj.messageName, msgObj);
}
}
catch (ex)
{
_self.emit('exception', ex);
}
}
}
/**
* Receive error output from the java process
*/
function onJavaError(data)
{
_self.emit('java_error', data.toString());
}
}
// Make our JavaClient class an EventEmitter
Util.inherits(JavaClient, EventEmitter);
// export our class
module.exports = JavaClient;

var client = require('./JavaClient');
var instance = new client();
instance.on('message', function(msg) {
console.log('Received a message...');
console.log(msg);
});
instance.on('sent', function(msg) {
console.log('Sent a message...');
console.log(msg);
});
instance.on('Info', function(msg) {
console.log("Received info");
console.log(msg.message);
});
(function() {
// Start it up (Hello exchanges happen)
instance.run();
// Receive acknowledgement of hello
instance.once('Info', function() {
// Try echoing something
instance.sendEcho("ECHO!");
});
})();

PUSH Notifications for >1000 devices through GCM Server(Java)

I have a GCM-backend Java server and I'm trying to send to all users a notification msg. Is my approach right? To just split them into 1000 each time before giving the send request? Or is there a better approach?
public void sendMessage(#Named("message") String message) throws IOException {
int count = ofy().load().type(RegistrationRecord.class).count();
if(count<=1000) {
List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).limit(count).list();
sendMsg(records,message);
}else
{
int msgsDone=0;
List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).list();
do {
List<RegistrationRecord> regIdsParts = regIdTrim(records, msgsDone);
msgsDone+=1000;
sendMsg(regIdsParts,message);
}while(msgsDone<count);
}
}
The regIdTrim method
private List<RegistrationRecord> regIdTrim(List<RegistrationRecord> wholeList, final int start) {
List<RegistrationRecord> parts = wholeList.subList(start,(start+1000)> wholeList.size()? wholeList.size() : start+1000);
return parts;
}
The sendMsg method
private void sendMsg(List<RegistrationRecord> records,#Named("message") String message) throws IOException {
if (message == null || message.trim().length() == 0) {
log.warning("Not sending message because it is empty");
return;
}
Sender sender = new Sender(API_KEY);
Message msg = new Message.Builder().addData("message", message).build();
// crop longer messages
if (message.length() > 1000) {
message = message.substring(0, 1000) + "[...]";
}
for (RegistrationRecord record : records) {
Result result = sender.send(msg, record.getRegId(), 5);
if (result.getMessageId() != null) {
log.info("Message sent to " + record.getRegId());
String canonicalRegId = result.getCanonicalRegistrationId();
if (canonicalRegId != null) {
// if the regId changed, we have to update the datastore
log.info("Registration Id changed for " + record.getRegId() + " updating to " + canonicalRegId);
record.setRegId(canonicalRegId);
ofy().save().entity(record).now();
}
} else {
String error = result.getErrorCodeName();
if (error.equals(Constants.ERROR_NOT_REGISTERED)) {
log.warning("Registration Id " + record.getRegId() + " no longer registered with GCM, removing from datastore");
// if the device is no longer registered with Gcm, remove it from the datastore
ofy().delete().entity(record).now();
} else {
log.warning("Error when sending message : " + error);
}
}
}
}

Quoting from Google Docs:
GCM is support for up to 1,000 recipients for a single message. This capability makes it much easier to send out important messages to your entire user base. For instance, let's say you had a message that needed to be sent to 1,000,000 of your users, and your server could handle sending out about 500 messages per second. If you send each message with only a single recipient, it would take 1,000,000/500 = 2,000 seconds, or around half an hour. However, attaching 1,000 recipients to each message, the total time required to send a message out to 1,000,000 recipients becomes (1,000,000/1,000) / 500 = 2 seconds. This is not only useful, but important for timely data, such as natural disaster alerts or sports scores, where a 30 minute interval might render the information useless.
Taking advantage of this functionality is easy. If you're using the GCM helper library for Java, simply provide a List collection of registration IDs to the send or sendNoRetry method, instead of a single registration ID.

We can not send more than 1000 push notification at time.I searched a lot but not result then i did this with same approach split whole list in sub lists of 1000 items and send push notification.