We Keep Coding

Publish / Subscribe MQTT using SmallRye reactive messaging dynamically

We try to publish and subscribe to MQTT protocol using smallrye reactive messaging. We managed to actually publish a message into a specific topic/channel through the following simple code
import io.smallrye.mutiny.Multi;
import org.eclipse.microprofile.reactive.messaging.Outgoing;
import javax.enterprise.context.ApplicationScoped;
import java.time.Duration;
#ApplicationScoped
public class Publish {
#Outgoing("pao")
public Multi<String> generate() {
return Multi.createFrom().ticks().every(Duration.ofSeconds(1))
.map(x -> "A Message in here");
}
}
What we want to do is to call whenever we want the generate() method somehow with a dynamic topic, where the user will define it. That one was our problem but then we found these classes from that repo in github. Package name io.smallrye.reactive.messaging.mqtt
For example we found that there is a class that says it makes a publish call to a MQTT broker(Mosquitto server up).
Here in that statement SendingMqttMessage<String> message = new SendingMqttMessage<String>("myTopic","A message in here",0,false);
We get the a red underline under the SendingMqttMessage<String> saying 'SendingMqttMessage(java.lang.String, java.lang.String, io.netty.handler.codec.mqtt.MqttQoS, boolean)' is not public in 'io.smallrye.reactive.messaging.mqtt.SendingMqttMessage'. Cannot be accessed from outside package
UPDATE(Publish done)
Finally made a Publish request to the mqtt broker(a mosquitto server) and all this with a dynamic topic configured from user. As we found out the previous Class SendingMqttMessage was not supposed to be used at all. And we found out that we also needed and emitter to actually make a publish request with a dynamic topic.
#Inject
#Channel("panatha")
Emitter<String> emitter;
#POST
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response createUser(Device device) {
System.out.println("New Publish request: message->"+device.getMessage()+" & topic->"+device.getTopic());
emitter.send(MqttMessage.of(device.getTopic(), device.getMessage()));
return Response.ok().status(Response.Status.CREATED).build();
}
Now we need to find out about making a Subscription to a topic dynamically.

first to sett us to the same page:
Reactive messaging does not work with topics, but with channels.
That is important to note, because you can exclusively read or write to a channel. So if you want to provide both, you need to configure two channels pointing at the same topic, one incoming and one outgoing
To answer your question:
You made a pretty good start with Emitters, but you still lack the dynamic nature you'd like.
In your example, you acquired that Emitter thru CDI.
Now that is all we need, to make this dynamic, since we cann dynamically inject Beans at runtime using CDI like this:
Sending Messages
private Emitter<byte[]> dynamicEmitter(String topic){
return CDI.current().select(new TypeLiteral<Emitter<byte[]>>() {}, new ChannelAnnotation(topic)).get();
}
please also note, that i am creating a Emitter of type byte[], as this is the only currently supportet type of the smallrye-mqtt connector (version 3.4.0) according to its documentation.
Receiving Messages
To read messages from a reactive messaging channel, you can use the counterpart of the Emitter, which is the Publisher.
It can be used analog:
private Publisher<byte[]> dynamicReceiver(String topic){
return CDI.current().select(new TypeLiteral<Publisher<byte[]>>() {}, new ChannelAnnotation(topic)).get();
}
You can then process these Date in any way you like.
As demo, it hung it on a simple REST Endpoint
#GET
#Produces(MediaType.SERVER_SENT_EVENTS)
public Multi<String> stream(#QueryParam("topic") String topic) {
return Multi.createFrom().publisher(dynamicReceiver(topic)).onItem().transform(String::new);
}
#GET
#Path("/publish")
public boolean publish(#QueryParam("msg") String msg, #QueryParam("topic") String topic) {
dynamicEmitter(topic).send(msg.getBytes());
return true;
}
One more Thing
When creating this solution I hit a few pitfalls you should know about:
Quarkus removes any CDI-Beans that are "unused". So if you want to inject them dynamically, you need to exclude those, or turne off that feature.
All channels injected that way must be configured. Otherwise the injection will fail.
For some Reason, (even with removal completely disabled) I was unable to inject Emitters dynamically, unless they are ever injected elsewhere.

Why is my handler method not triggered when defined as a lambda?

I am defining an IntegrationFlow to stream from SFTP to S3 with the DSL syntax this way :
return IntegrationFlows.from(Sftp.inboundStreamingAdapter(remoteFileTemplate)
.remoteDirectory("remoteDirectory"),
e -> e.poller(Pollers.fixedDelay(POLL, TimeUnit.SECONDS)))
.transform(new StreamTransformer())
.handle(s3UploadMessageHandler(outputFolderPath, "headers['file_remoteFile']")) // Upload on S3
.get();
private S3MessageHandler s3UploadMessageHandler(String folderPath, String spelFileName) {
S3MessageHandler s3MessageHandler = new S3MessageHandler(amazonS3, s3ConfigProperties.getBuckets().getCardManagementData());
s3MessageHandler.setKeyExpression(new SpelExpressionParser().parseExpression(String.format("'%s/'.concat(%s)", folderPath, spelFileName)));
s3MessageHandler.setCommand(S3MessageHandler.Command.UPLOAD);
return s3MessageHandler;
}
And it works as intended : the file is well uploaded to my S3 bucket. However, I would like to avoid SPEL syntax, and inject headers from the message to the s3uploadMessageHandler method, this way I could use a simple ValueExpression to set the keyExpression in the s3UploadMessageHandler method.
To do this, I changed
handle(s3UploadMessageHandler(outputFolderPath, "headers['file_remoteFile']")) // Upload on S3
to
handle(m -> s3UploadMessageHandler(outputFolderPath, (String) m.getHeaders().get("file_remoteFile"))) // Upload on S3
But now this handler doesn't seem to be triggered anymore. There is no errors in the logs, and I know from the logs that the SFTP polling is still working.
I tried to find the reason behind this, and I saw that when entering the handle method in IntegrationFlowdefinition.java, the messageHandler class type is different : it's an S3MessageHandler when called without lambda, and a MyCallingClass$lambda when calling with a lambda expression.
What did I miss to make my scenario working ?

There are two ways to handle a message. One is via a MessageHandler implementation - this is the most efficient approach and that's done in the framework for channel adapter implementation, like that S3MessageHandler. Another way is a POJO method invocation - this is the most user-friendly approach when you don't need to worry about any framework interfaces.
So, when you use it like this .handle(s3UploadMessageHandler(...)) you refer to a MessageHandler and the framework knows that a bean for that MessageHandler has to be registered since your s3UploadMessageHandler() is not a #Bean.
When you use it as a lambda, the framework treats it as a POJO method invocation and there is a bean registered for the MethodInvokingMessageHandler, but not your S3MessageHandler.
Anyway, even if you change your s3UploadMessageHandler() to be a #Bean method it is not going to work because you don't let the framework to call the S3MessageHandler.handleMessage(). What you do here is just call that private method at runtime to create an S3MessageHandler instance against every request message: the MethodInvokingMessageHandler calls your lambda in its handleMessage() and that's all - nothing is going to happen with S3.
The ValueExpression cannot help you here because you need to evaluate a destination file against every single request message. Therefore you need a runtime expression. There is indeed nothing wrong with the new SpelExpressionParser().parseExpression(). Just because we don't have a choice and have to have only single stateless S3MessageHandler and don't recreate it at runtime on every request like you try to achieve with that suspicious lambda and ValueExpression.

set header on rsocket messages with spring boot

so I've started playing with rsocket and spring boot 2.2 to see if I can use it in my projects, but I'm facing a bit of troubles.
Normally, with spring messaging I define a listener method like the following:
#MessageMapping("addGeolocation")
public Mono<Boolean> addGeolocation(#Header("metadata") MmeMetadata metadata, #Payload String geolocation) { ... }
My understanding is that with rsocket I should be able to use the same logic, but when I'm defining the client I couldn't find an easy way to set message headers.
Currently I'm stuck with this:
boolean outcome = rSocketRequester.route("addGeolocation").metadata(...?).data(geolocationWKT).block();
is the metadata a replacement for headers? that method signature seems a little too generic to be used like headers. If I put an Map in it will spring be able to decode headers out of it?
Thank you,
Fernando

Please see this question: RSocket Metadata - custom object.
I used it as a starting point for my solution.
The term 'header' actually means some custom metadata. So, in order to get the correct value you need to configure metadataExtractorRegistry. For Spring Boot do it this way (code in kotlin):
val CUSTOM_MIMETYPE = MimeType.valueOf("<some custom mime type>")
val CUSTOM_HEADER = "<the name of a header>"
...
#Bean
fun rSocketStrategiesCustomizer(): RSocketStrategiesCustomizer {
return RSocketStrategiesCustomizer { strategies: RSocketStrategies.Builder ->
strategies.metadataExtractorRegistry {
it.metadataToExtract(CUSTOM_MIMETYPE, String::class.java, CUSTOM_HEADER)
}
}
}
The type of the data object can be any, not necessarily a String. There is default String endcoder/decoder, so I didn't provide one in the code. For your own type you can provide one of existing encoders/decoders (Json for example) or create your own:
#Bean
fun rSocketStrategiesCustomizer(): RSocketStrategiesCustomizer {
return RSocketStrategiesCustomizer { strategies: RSocketStrategies.Builder ->
strategies.metadataExtractorRegistry {
it.metadataToExtract(CUSTOM_MIMETYPE, YourType::class.java, CUSTOM_HEADER)
}.decoder(Jackson2JsonDecoder())
.encoder(Jackson2JsonEncoder())
}
}
After you've configured registry as above, use the header name defined in the registry in your controller:
#MessageMapping("addGeolocation")
public Mono<Boolean> addGeolocation(#Header(CUSTOM_HEADER) String metadata, #Payload String geolocation) { ... }
And in order to send that header, use next code:
boolean outcome = rSocketRequester.route("addGeolocation")
.metadata("value", CUSTOM_MIMETYPE)
.data(geolocationWKT)
.block();
Hope this helps

Instead of a bag of name-value pairs (i.e. headers), RSocket uses metadata which can be in any format (i.e. MIME type) and it can be composite metadata with multiple types of metadata each formatted differently. So you can have one section with routing metadata, another with security, yet another with tracing, and so on.
To achieve something similar to headers, you can send name-value pairs as JSON-formatted metadata. Now on the server side, you'll need to provide a hint to Spring for how to extract a Map (of headers) from the metadata of incoming requests. To do that you can configure a MetadataExtractor and that's described in this section of the docs. Once that's configured, the extracted Map becomes the headers of the message and can be accessed from #MessageMapping methods as usual (via MessageHeaders, #Header, etc).

How should my proxy intercept packets

I have two web servers A and C. Because of rules, policies etc A cannot communicate directly with C' (Crequires requests to have special security headers andA` does not support this)
I want to create a "proxy" called B in Java, to facilitate communication between A and B but I having trouble with it conceptually.
When you create a proxy, don't you have to change the hard coded URLs on all your services to now travel through the proxy? And then I suppose you must pass your destination as a query parameter somehow, for example
http://proxy.com/?destination=mydestination.com
Is this how a proxy would work?
(I have found a few similar questions but they do not resolve the fundamental question I am having i.e. How packets reaches Destination throgh proxy servers?)

Don't write your own proxy from scratch that will be a redo, there are a plenty of proxy implementations out there
Spring framework
Spring Boot implementation for (Netflex/Zuul) https://github.com/Netflix/zuul / https://spring.io/guides/gs/routing-and-filtering/
Standard Servlet
https://github.com/mitre/HTTP-Proxy-Servlet
in case if you are intrested in how you can build up your own proxy , here is a simple example using spring framework (Servlet implementation is not that far from spring implementation)
#RestController
public class ProxyController {
/**
* request handler for all the requests on the registered context intended to be proxied
*
* #param request
* #return
*/
#RequestMapping("/**")
public String defaultGateway(HttpServletRequest request) {
// edit the original request headers,body etc
String originalPath = request.getRequestURI();
// then fire another HTTP request using the same URI
return "ProxiedResult"; // return back the result you get from
}
}
the example above is a start point on how to implements HTTP proxy or how it is working there are many things to cover which is droped off like security for instance

What is the correct way to get a template in FTL/spring when not in a web Servlet

We are currently using a sync. web call to send emails. This was done as a quick fix to prove out some basic functionality, but now we need to send the e-mails async. I have the everything pretty much reworked to queue up jobs and then send the emails, but I've run in to one issue. We use FTL for our email templates and before we were passing the servlet context to FTL to get the template folder. Since we are now doing this in a queued job that get's processed by a Spring #Scheduled job, we no longer have access to the web servlet. I've been researching and playing around for awhile now, but I haven't seem to come up with a way that will actually work.
I have a feeling there is some super simple way to get
The code that did the work before looked similar to this:
#PUT
#Produces(MediaType.APPLICATION_JSON)
#Path("someStuffHere")
#Transactional
public function someWebServiceCall(#Context javax.servlet.http.HttpServletRequest req)
{
SomeStuff foo = gotSomeStuff();
sendEmail(req.getServlet(), foo);
}
public sendEmail(Servlet context, SomeStuff foo) //<-- lives in another class somewhere, just showing how FTL uses the servlet
{
Configuration cfg = new Configuration();
cfg.setServletContextForTemplateLoading(context,"communicationTemplates/email");
}
The new code now looks something like this:
public class someClass
{
#Autowired
private SomeRepo someRepo;
#Scheduled(cron = "* */2 * * * ?")
public void sendAnyOutstandingStuffEmails()
{
SomeStuff foo = someRepo.getStuff();
sendEmail(/*how to get context, or a resource so FTL can get the template folder*/, foo)
}

Even though this post is quite old and the author has given another solution a try, it is not necessary to have a servlet context instance in order to load templates. The freemarker documentation states:
Built-in template loaders
You can set up three template loading methods in the Configuration
using the following convenience methods. (Each method will create a
template loader object internally and set up the Configuration
instance to use that.)
void setDirectoryForTemplateLoading(File dir);
or
void setClassForTemplateLoading(Class cl, String prefix);
or
void setServletContextForTemplateLoading(Object servletContext, String path);
http://freemarker.org/docs/pgui_config_templateloading.html
So in this case it should have been possible to configure freemarker to use the ClassLoader (option 2) by naming a class that is on the same level as the templates or to use this class as root node for navigating to the template using relative paths.