We Keep Coding

Java: Azure Service Bus Queue Receiving messsages with sessions

I'm writing code in java (using Azure SDK for Java), I have a Service bus queue that contains sessionful messages. I want to receive those messages and process them to another place.
I make a connection to the Queue by using QueueClient, and then I use registerSessionHandler to process through the messages (code below).
The problem is that whenever a message is received, I can print all details about it including the content, but it is printed 10 times and after each time it prints an Exception.
(printing 10 times: I understand that this is because there is a 10 times retry policy before it throws the message to the Dead letter queue and goes to the next message.)
The Exception says
> USERCALLBACK-Receiver not created. Registering a MessageHandler creates a receiver.
The output with the Exception
But I'm sure that the SessionHandler does the same thing as MessageHandler but includes support for sessions, so it should create a receiver since it receives messages. I have tried to use MessageHandler but it won't even work and stops the whole program because it doesn't support sessionful messages, and the ones I receive have sessions.
My problem is understanding what the Exception wants me to do, and how can I fix the code so it won't give me any exceptions? Does anyone have suggestions on how to improve the code? or other methods that do the same thing?
QueueClient qc = new QueueClient(
new ConnectionStringBuilder(connectionString),
ReceiveMode.PEEKLOCK);
qc.registerSessionHandler(
new ISessionHandler() {
#Override
public CompletableFuture<Void> onMessageAsync(IMessageSession messageSession, IMessage message) {
System.out.printf(
"\nMessage received: " +
"\n --> MessageId = %s " +
"\n --> SessionId = %s" +
"\n --> Content Type = %s" +
"\n --> Content = \n\t\t %s",
message.getMessageId(),
messageSession.getSessionId(),
message.getContentType(),
getMessageContent(message)
);
return qc.completeAsync(message.getLockToken());
}
#Override
public CompletableFuture<Void> OnCloseSessionAsync(IMessageSession iMessageSession) {
return CompletableFuture.completedFuture(null);
}
#Override
public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
System.out.println("\n Exception " + exceptionPhase + "-" + throwable.getMessage());
}
},
new SessionHandlerOptions(1, true, Duration.ofMinutes(1)),
Executors.newSingleThreadExecutor()
);
(The getMessageContent(message) method is a separate method, for those interested:)
public String getMessageContent(IMessage message){
List<byte[]> content = message.getMessageBody().getBinaryData();
StringBuilder sb = new StringBuilder();
for (byte[] b : content) {
sb.append(new String(b)
);
}
return sb.toString();
}

For those who wonder, I managed to solve the problem!
It was simply done by using Azure Functions ServiceBusQueueTrigger, it will then listen to the Service bus Queue and process the messages. By setting isSessionsEnabled to true, it will accept sessionful messages as I wanted :)
So instead of writing more than 100 lines of code, the code looks like this now:
public class Function {
#FunctionName("QueueFunction")
public void run(
#ServiceBusQueueTrigger(
name = "TriggerName", //Any name you choose
queueName = "queueName", //QueueName from the portal
connection = "ConnectionString", //ConnectionString from the portal
isSessionsEnabled = true
) String message,
ExecutionContext context
) {
// Write the code you want to do with the message here
// Using the variable messsage which contains the messageContent, messageId, sessionId etc.
}
}

Flow hangs when IdempotentReceiverInterceptor discards the message(after 4-th message)

I have following flow:
return flow -> flow.channel(inputChannel())
...
.gateway(childFlow, addMyInterceptor(str)); // by name
}
Consumer<GatewayEndpointSpec> addMyInterceptor(String objectIdHeader) {
return endpointSpec -> endpointSpec.advice(addMyInterceptorInternal(objectIdHeader))
.errorChannel(errorChannel());
}
default IdempotentReceiverInterceptor addMyInterceptorInternal(String header) {
MessageProcessor<String> headerSelector = message -> headerExpression(header).apply(message);
var interceptor = new IdempotentReceiverInterceptor(new MetadataStoreSelector(headerSelector, idempotencyStore()));
interceptor.setDiscardChannel(idempotentDiscardChannel());
return interceptor;
}
When IdempotentReceiverInterceptor encounters that message is duplicated - I see that application hangs on after 4-th duplicated message. I understand that it is because gateway expected response(like here: PubSubInboundChannelAdapter stops to receive messages after 4th message) but I don't have any ideas how to return result from interceptor.
Could you please explain it for me?

As long as all channels are direct (default) - i.e. no async handoffs in the flow using queue or executor channels, set the gateway's replyTimeout to 0 when the flow might not return a reply

Why doesn't this thread pool execute HTTP requests simultaneously?

I wrote a few lines of code which will send 50 HTTP GET requests to a service running on my machine. The service will always sleep 1 second and return a HTTP status code 200 with an empty body. As expected the code runs for about 50 seconds.
To speed things up a little I tried to create an ExecutorService with 4 threads so I could always send 4 requests at the same time to my service. I expected the code to run for about 13 seconds.
final List<String> urls = new ArrayList<>();
for (int i = 0; i < 50; i++)
urls.add("http://localhost:5000/test/" + i);
final RestTemplate restTemplate = new RestTemplate();
final List<Callable<String>> tasks = urls
.stream()
.map(u -> (Callable<String>) () -> {
System.out.println(LocalDateTime.now() + " - " + Thread.currentThread().getName() + ": " + u);
return restTemplate.getForObject(u, String.class);
}).collect(Collectors.toList());
final ExecutorService executorService = Executors.newFixedThreadPool(4);
final long start = System.currentTimeMillis();
try {
final List<Future<String>> futures = executorService.invokeAll(tasks);
final List<String> results = futures.stream().map(f -> {
try {
return f.get();
} catch (InterruptedException | ExecutionException e) {
throw new IllegalStateException(e);
}
}).collect(Collectors.toList());
System.out.println(results);
} finally {
executorService.shutdown();
executorService.awaitTermination(10, TimeUnit.SECONDS);
}
final long elapsed = System.currentTimeMillis() - start;
System.out.println("Took " + elapsed + " ms...");
But - if you look at the seconds of the debug output - it seems like the first 4 requests are executed simultaneously but all other request are executed one after another:
2018-10-21T17:42:16.160 - pool-1-thread-3: http://localhost:5000/test/2
2018-10-21T17:42:16.160 - pool-1-thread-1: http://localhost:5000/test/0
2018-10-21T17:42:16.160 - pool-1-thread-2: http://localhost:5000/test/1
2018-10-21T17:42:16.159 - pool-1-thread-4: http://localhost:5000/test/3
2018-10-21T17:42:17.233 - pool-1-thread-3: http://localhost:5000/test/4
2018-10-21T17:42:18.232 - pool-1-thread-2: http://localhost:5000/test/5
2018-10-21T17:42:19.237 - pool-1-thread-4: http://localhost:5000/test/6
2018-10-21T17:42:20.241 - pool-1-thread-1: http://localhost:5000/test/7
...
Took 50310 ms...
So for debugging purposes I changed the HTTP request to a sleep call:
// return restTemplate.getForObject(u, String.class);
TimeUnit.SECONDS.sleep(1);
return "";
And now the code works as expected:
...
Took 13068 ms...
So my question is why does the code with the sleep call work as expected and the code with the HTTP request doesn't? And how can I get it to behave in the way I expected?

From the information, I can see this is the most probable root cause:
The requests you make are done in parallel but the HTTP server which fulfils these request handles 1 request at a time.
So when you start making requests, the executor service fires up the requests concurrently, thus you get the first 4 at same time.
But the HTTP server can respond to requests one at a time i.e. after 1 second each.
Now when 1st request is fulfilled the executor service picks another request and fires it and this goes on till last request.
4 request are blocked at HTTP server at a time, which are being served serially one after the other.
To get a Proof of Concept of this theory what you can do is use a messaging service (queue) which can receive concurrently from 4 channels an test. That should reduce the time.

ServiceActivator 'randomly' unsubscribing from PublishSubscribeChannel

I have a method annotated with #ServiceActivator("CH1"), where "CH1" definition is:
#Bean(name = "CH1")
MessageChannel ch1() {
return new PublishSubscribeChannel
}
and other PollableChannels publishing to this channel via
#BidgeTo(value = "CH1", poller = #Poller("myPoller"))
Things seem to work fine most of the time; however, seemingly randomly the message handler unsubscribes from "CH1" and I see in the logs:
[DEBUG] (pool-2-thread-1) org.springframework.integration.dispatcher.BroadcastingDispatcher: No subscribers, default behavior is ignore
Now I know I can change the minSubscribers but I don't get why things seem to randomly unsubscribe? After this error it will go back to handling some messages fine. Does a message handler unsubscribe while handling messages or if the executor being used is full? I see no errors associated with this in the log nor and unsubscribe or update to subscriber counts to "CH1" in the logs.

That does not make sense. Please, share some test-case to reproduce from the Framework perspective.
The source code on the matter looks like:
if (dispatched == 0 && this.minSubscribers == 0 && logger.isDebugEnabled()) {
if (sequenceSize > 0) {
logger.debug("No subscribers received message, default behavior is ignore");
}
else {
logger.debug("No subscribers, default behavior is ignore");
}
}
where we can go to the sequence == 0 only in case of:
Collection<MessageHandler> handlers = this.getHandlers();
if (this.requireSubscribers && handlers.size() == 0) {
throw new MessageDispatchingException(message, "Dispatcher has no subscribers");
}
int sequenceSize = handlers.size();
Only the clue that your subscribers unsubscribes somehow...
I see that you have a DEBUG for your CH1, so would you mind to share DEBUG logs for entire org.springframework.integration when you see that error.
EDIT
Also note that whenever a subscriber is added/removed (e.g. when a consuming endpoint is started/stopped), you will see this log message...
if (logger.isInfoEnabled()) {
logger.info("Channel '" + this.getFullChannelName() + "' has " + counter + " subscriber(s).");
}
(when logging with at least INFO logging).

Why isn't the awaiting thread activated with a signalAll?

I got 2 functions. The first one discoverHosts() sends an request message to other computers. After this it goes to sleep with the await command. A separate threat calls the handleMessage() function when he receives a response. After he handles the response he uses notifyAll() to let discoderHosts() know that he has to check that all responses are received.
DiscoverHosts() does await when he calls the function. However when the separate threat calls handleMessage(), discoverHosts() doesn't awake when handleMessage calls the signalAll(). I checked while debuggig if signalAll() is called and this is the case. I have almost the same bit of code somewhere else in my project where it does work.
Do any of you guys know what I am overlooking?
private final Lock lock = new ReentrantLock();
private final Condition allReceived = lock.newCondition();
private void discoverHosts() throws Exception {
lock.lock();
externalNodes = new HashMap<String, NodeAddress>();
Message msg = new Message(null, "REQUEST_IP");
logger.debug("Broadcasting ip request, waiting responses");
channel.send(msg);
// TODO:Write a time-out
while (channel.getView().size() - 1 != externalNodes.keySet().size()) {
logger.debug("Channel: " + (channel.getView().size() - 1));
logger.debug("Responses: "+ externalNodes.keySet().size());
allReceived.await();
}
logger.debug("All answers received");
lock.unlock();
}
protected void handleMessage(Message msg) {
lock.lock();
if (!((String) msg.getObject()).matches("IP_RESPONSE:[0-9.]*"))
return;
logger.debug("Received answer from " + msg.getObject());
String ip = ((String) msg.getObject()).replaceAll("IP_RESPONSE:", "");
// externalHostIps.add(ip);
NodeAddress currentAddress = new NodeAddress(ip, msg.getSrc());
externalNodes.put(ip, currentAddress);
logger.debug("Signalling all threads");
allReceived.signalAll();
lock.unlock();
logger.debug("Unlocked");
}
Logger output:
4372 [main] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Broadcasting ip request, waiting responses
4372 [main] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Channel: 1
4372 [main] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Responses: 0
4394 [Incoming-1,webhackarena,leendert-K53SV-53745] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Received answer from IP_RESPONSE:192.168.1.106
4396 [Incoming-1,webhackarena,leendert-K53SV-53745] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Signalling all threads
4397 [Incoming-1,webhackarena,leendert-K53SV-53745] DEBUG com.conbit.webhackarena.monitor.monitor.Monitor#3b91eb - Unlocked

I think your problem is this line
if (!((String) msg.getObject()).matches("IP_RESPONSE:[0-9.]*"))
return;
Which means under some condition you acquire the lock and never release it.
Always use try...finally blocks with Lock to avoid this issue.
protected void handleMessage(Message msg) {
lock.lock();
try {
if (!((String) msg.getObject()).matches("IP_RESPONSE:[0-9.]*"))
return;
logger.debug("Received answer from " + msg.getObject());
String ip = ((String) msg.getObject()).replaceAll("IP_RESPONSE:", "");
// externalHostIps.add(ip);
NodeAddress currentAddress = new NodeAddress(ip, msg.getSrc());
externalNodes.put(ip, currentAddress);
logger.debug("Signalling all threads");
allReceived.signalAll();
} finally {
lock.unlock();
}
}

However when the separate threat calls handleMessage(), discoverHosts() doesn't awake when handleMessage calls the signalAll()
I suspect that you have two different instances of the class in question and since the Condition allReceived is private final, they are not actually dealing with the same condition.
If you are trying to debug this (or use System.out.println debugging), be sure that your instance of the wrapping class is the same in both threads.