I want to subscribe to a PubSub channel, to get all messages payload and processing them.
The thing is, I saw example with IntegrationFlow, but if I understand this correctly, integrationFlow is used to get from a channel, and send to another one.
I don't want to send to another channel, I just want to get the payload, to process it.
How can I do this?
All examples I found use xml files to make the configuration, but the project was not created using this way of doing.
I saw example with IntegrationFlow, but if I understand this correctly, integrationFlow is used to get from a channel, and send to another one.
I think that this is not correct information. You can just handle the messages.
You can do something like this in a configuration class:
#Bean
public IntegrationFlow myFlow() {
return IntegrationFlows.from(...)
.handle((GenericHandler<YourMessage>) (message, headers) -> {...})
}
Related
Long story short: I need to have something like below.
PublishSubscribeChannel firstChannel = new PublishSubscribeSpec(executor).subscribe(subFlow -> ...).get();
Is there a way to create a pubsub channel with subflows which is not (yet) connected to any other flow?
The snippet is not working because of PublishSubscribeSpec(Executor) has protected access in PublishSubscribeSpec.
I will need to register channels like this dynamically without any information about which flow(s) will be using these channels.
has protected access in PublishSubscribeSpec
That was exactly a reason to make it protected - to avoid an unusual configuration problem like your. The subflow cannot be provided like this in the plain PublishSubscribeChannel definition. It is part of Java DSL parser in the framework to determine such a configuration and register respective beans in the application context. With that explicit get() call you just fully eliminate a hook for Java DSL parser to understand your configuration.
without any information about which flow(s) will be using these channels.
That's not true according your .subscribe(subFlow -> intention. Adding a subflow to the PublishSubscribeSpec is indeed "an information which flow will be using these channel".
Perhaps we need to look into your business requirement from another angle. There is no reason to be stuck with subflows approach when we simply can use a PublishSubscribeChannel from any other place where a MessageChannel is needed as an input. I mean if you just create a plain PublishSubscribeChannel and then use it for example for the IntegrationFlows.from(MessageChannel) factory, you'll get the same runtime result as you would expect from those .subscribe(subFlow -> connections.
We try to publish and subscribe to MQTT protocol using smallrye reactive messaging. We managed to actually publish a message into a specific topic/channel through the following simple code
import io.smallrye.mutiny.Multi;
import org.eclipse.microprofile.reactive.messaging.Outgoing;
import javax.enterprise.context.ApplicationScoped;
import java.time.Duration;
#ApplicationScoped
public class Publish {
#Outgoing("pao")
public Multi<String> generate() {
return Multi.createFrom().ticks().every(Duration.ofSeconds(1))
.map(x -> "A Message in here");
}
}
What we want to do is to call whenever we want the generate() method somehow with a dynamic topic, where the user will define it. That one was our problem but then we found these classes from that repo in github. Package name io.smallrye.reactive.messaging.mqtt
For example we found that there is a class that says it makes a publish call to a MQTT broker(Mosquitto server up).
Here in that statement SendingMqttMessage<String> message = new SendingMqttMessage<String>("myTopic","A message in here",0,false);
We get the a red underline under the SendingMqttMessage<String> saying 'SendingMqttMessage(java.lang.String, java.lang.String, io.netty.handler.codec.mqtt.MqttQoS, boolean)' is not public in 'io.smallrye.reactive.messaging.mqtt.SendingMqttMessage'. Cannot be accessed from outside package
UPDATE(Publish done)
Finally made a Publish request to the mqtt broker(a mosquitto server) and all this with a dynamic topic configured from user. As we found out the previous Class SendingMqttMessage was not supposed to be used at all. And we found out that we also needed and emitter to actually make a publish request with a dynamic topic.
#Inject
#Channel("panatha")
Emitter<String> emitter;
#POST
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Response createUser(Device device) {
System.out.println("New Publish request: message->"+device.getMessage()+" & topic->"+device.getTopic());
emitter.send(MqttMessage.of(device.getTopic(), device.getMessage()));
return Response.ok().status(Response.Status.CREATED).build();
}
Now we need to find out about making a Subscription to a topic dynamically.
first to sett us to the same page:
Reactive messaging does not work with topics, but with channels.
That is important to note, because you can exclusively read or write to a channel. So if you want to provide both, you need to configure two channels pointing at the same topic, one incoming and one outgoing
To answer your question:
You made a pretty good start with Emitters, but you still lack the dynamic nature you'd like.
In your example, you acquired that Emitter thru CDI.
Now that is all we need, to make this dynamic, since we cann dynamically inject Beans at runtime using CDI like this:
Sending Messages
private Emitter<byte[]> dynamicEmitter(String topic){
return CDI.current().select(new TypeLiteral<Emitter<byte[]>>() {}, new ChannelAnnotation(topic)).get();
}
please also note, that i am creating a Emitter of type byte[], as this is the only currently supportet type of the smallrye-mqtt connector (version 3.4.0) according to its documentation.
Receiving Messages
To read messages from a reactive messaging channel, you can use the counterpart of the Emitter, which is the Publisher.
It can be used analog:
private Publisher<byte[]> dynamicReceiver(String topic){
return CDI.current().select(new TypeLiteral<Publisher<byte[]>>() {}, new ChannelAnnotation(topic)).get();
}
You can then process these Date in any way you like.
As demo, it hung it on a simple REST Endpoint
#GET
#Produces(MediaType.SERVER_SENT_EVENTS)
public Multi<String> stream(#QueryParam("topic") String topic) {
return Multi.createFrom().publisher(dynamicReceiver(topic)).onItem().transform(String::new);
}
#GET
#Path("/publish")
public boolean publish(#QueryParam("msg") String msg, #QueryParam("topic") String topic) {
dynamicEmitter(topic).send(msg.getBytes());
return true;
}
One more Thing
When creating this solution I hit a few pitfalls you should know about:
Quarkus removes any CDI-Beans that are "unused". So if you want to inject them dynamically, you need to exclude those, or turne off that feature.
All channels injected that way must be configured. Otherwise the injection will fail.
For some Reason, (even with removal completely disabled) I was unable to inject Emitters dynamically, unless they are ever injected elsewhere.
I want to use Spring Integration to expose a simple web service that pushes incoming message into ActiveMQ and responds immediately. My go-to solution was MarshallingWebServiceInboundGateway connected to Jms.outboundAdapter with IntegrationFlow. Below the Gateway and IntegrationFlow snippets. Problem with this is Adapter does not provide response (duh) which Gateway expects. The response I get back from the service is empty 202, with delay of about 1500ms. This is caused by a reply timeout I see in TRACE logs:
"2020-04-14 17:17:50.101 TRACE 26524 --- [nio-8080-exec-6] o.s.integration.core.MessagingTemplate : Failed to receive message from channel 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#518ffd27' within timeout: 1000"
No hard exceptions anywhere. The other problem is I cannot generate the response myself. I can't add anything to IntegrationFlow after the .handle with Adapter.
Any other way I can try to fulfill the scenario?
How, if at all possible, can I generate and return response in situation there is no better approach?
Most likely the proper way would be to use Gateways on both ends, but this is not possible. I cannot wait with response until message in the queue gets consumed and processed.
'''
#Bean
public MarshallingWebServiceInboundGateway greetingWebServiceInboundGateway() {
MarshallingWebServiceInboundGateway inboundGateway = new MarshallingWebServiceInboundGateway(
jaxb2Marshaller()
);
inboundGateway.setRequestChannelName("greetingAsync.input");
inboundGateway.setLoggingEnabled(true);
return inboundGateway;
}
#Bean
public IntegrationFlow greetingAsync() {
return f -> f
.log(LoggingHandler.Level.INFO)
.handle(Jms.outboundAdapter(this.jmsConnectionFactory)
.configureJmsTemplate(c -> {
c.jmsMessageConverter(new MarshallingMessageConverter(jaxb2Marshaller()));
})
.destination(JmsConfig.HELLO_WORLD_QUEUE));
}
'''
The logic and assumptions are fully correct: you can't return after one-way handle() and similar to that Jms.outboundAdapter().
But your problem that you fully miss one of the first-class citizens in Spring Integration - a MessageChannel. It is important to understand that even in the flow like yours there are channels between endpoints (DSL methods) - implicit (DirectChannel), like in your case, or explicit: when you use a channel() in between and can place there any possible implementation: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/dsl.html#java-dsl-channels
One of the crucial channel implementation is a PublishSubscribeChannel (a topic in JMS specification) when you can send the same message to several subscribed endpoints: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/core.html#channel-implementations-publishsubscribechannel
In your case the fists subscriber should be your existing, one-way Jms.outboundAdapter(). And another something what is going to generate response and reply it into a replyChannel header.
For this purpose Java DSL provides a nice hook via sub-flows configuration: https://docs.spring.io/spring-integration/docs/5.3.0.M4/reference/html/dsl.html#java-dsl-subflows
So, some sample of publish-subscriber could be like this:
.publishSubscribeChannel(c -> c
.subscribe(sf -> sf
.handle(Jms.outboundAdapter(this.jmsConnectionFactory))))
.handle([PRODUCE_RESPONSE])
I'm trying to create a Spring Cloud Stream Source Bean inside a Spring Boot Application that simply sends the results of a method to a stream (underlying Kafka topic is bound to the stream).
Most of the Stream samples I've seen use #InboundChannelAdapter annotation to send data to the stream using a poller. But I don't want to use a poller. I've tried setting the poller to an empty array but the other problem is that when using #InboundChannelAdapter you are unable to have any method parameters.
The overall concept of what I am trying to do is read from an inbound stream. Do some async processing, then post the result to an outbound stream. So using a processor doesn't seem to be an option either. I am using #StreamListener with a Sink channel to read the inbound stream and that works.
Here is some code i've been trying but this doesn't work at all. I was hoping it would be this simple because my Sink was but maybe it isn't. Looking for someone to point me to an example of a source that isn't a Processor (i.e. doesn't require listening on an inbound channel) and doesn't use #InboundChannelAdapter or to give me some design tips to accomplish what I need to do in a different way. Thanks!
#EnableBinding(Source.class)
public class JobForwarder {
#ServiceActivator(outputChannel = Source.OUTPUT)
#SendTo(Source.OUTPUT)
public String forwardJob(String message) {
log.info(String.format("Forwarding a job message [%s] to queue [%s]", message, Source.OUTPUT));
return message;
}
}
Your orginal requirement can be achieved through the below steps.
Create your custom Bound Interface (you can use the default #EnableBinding(Source.class) as well)
public interface CustomSource {
String OUTPUT = "customoutput";
#Output(CustomSource.OUTPUT)
MessageChannel output();
}
Inject your bound channel
#Component
#EnableBinding(CustomSource.class)
public class CustomOutputEventSource {
#Autowired
private CustomSource customSource;
public void sendMessage(String message) {
customSource.output().send(MessageBuilder.withPayload(message).build());
}
}
Test it
#RunWith(SpringRunner.class)
#SpringBootTest
public class CustomOutputEventSourceTest {
#Autowired
CustomOutputEventSource output;
#Test
public void sendMessage() {
output.sendMessage("Test message from JUnit test");
}
}
So if you don't want to use a Poller, what causes the forwardJob() method to be called?
You can't just call the method and expect the result to go to the output channel.
With your current configuration, you need an inputChannel on the service containing your inbound message (and something to send a message to that channel). It doesn't have to be bound to a transport; it can be a simple MessageChannel #Bean.
Or, you could use a #Publisher to publish the result of the method invocation (as well as being returned to the caller) - docs here.
#Publisher(channel = Source.OUTPUT)
Thanks for the input. It took me a while to get back to the problem. I did try reading the documentation for #Publisher. It looked to be exactly what I needed but I just couldn't get the proper beans initialized to get it wired properly.
To answer your question the forwardJob() method is called after some async processing of the input.
Eventually I just implemented using spring-kafka library directly and that was much more explicit and felt easier to get going. I think we are going to stick to kafka as the only channel binding so I think we'll stick with that library.
However, we did eventually get the spring-cloud-stream library working quite simply. Here was the code for a single source without a poller.
#Component
#EnableBinding(Source.class)
public class JobForwarder {
private Source source;
#Autowired
public ScheduledJobForwarder(Source source) {
this.source = source;
}
public void forwardScheduledJob(String message) {
log.info(String.format("Forwarding a job message [%s] to queue [%s]", message, Source.OUTPUT));
source.output().send(MessageBuilder.withPayload(message).build());
}
}
I am using Spring Integration DSL with Java 1.7 and AMQP, and I am trying to log my events with LoggingHandler through logback, this is a snippet of my configuration code.
#Bean
public IntegrationFlow inboundFlow() {
return IntegrationFlows.from(Amqp.inboundGateway(simpleMessageListenerContainer())
.mappedReplyHeaders(AMQPConstants.AMQP_CUSTOM_HEADER_FIELD_NAME_MATCH_PATTERN)
.mappedRequestHeaders(AMQPConstants.AMQP_CUSTOM_HEADER_FIELD_NAME_MATCH_PATTERN)
.errorChannel(gatewayErrorChannel())
.requestChannel(gatewayRequestChannel())
.replyChannel(gatewayResponseChannel())
)
.transform(getCustomFromJsonTransformer())
.route(new HeaderValueRouter(AMQPConstants.OPERATION_ROUTING_KEY))
.get();
}
#Bean
public MessageChannel gatewayRequestChannel() {
return MessageChannels.publishSubscribe().get();
}
#Bean
public MessageChannel gatewayResponseChannel() {
return MessageChannels.publishSubscribe().get();
}
#Bean
public LoggingHandler getLoggingHandler(){
LoggingHandler loggingHandler = new LoggingHandler(LoggingHandler.Level.INFO.name());
loggingHandler.setLoggerName("analytics");
loggingHandler.setExpression("headers");
return loggingHandler;
}
private IntegrationFlow fileLoggerOutboundFlowTemplate(MessageChannel fromMessageChannel) {
return IntegrationFlows.from(fromMessageChannel)
.handle(getLoggingHandler())
.get();
}
#Bean
public IntegrationFlow requestFileLoggerOutboundFlow() {
return fileLoggerOutboundFlowTemplate(gatewayRequestChannel());
}
#Bean
public IntegrationFlow responseFileLoggerOutboundFlow() {
return fileLoggerOutboundFlowTemplate(gatewayResponseChannel());
}
Now, I have two main problems here:
This loggingHandler is logging headers from request and response to logback, an example for a response log entry with headers I currently have, look like this :
{errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#ff940b1, replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#ff940b1, contentType=JSON, amqp_consumerQueue=clientAdapter, userId=12345678, monitoringId=34841e03-a7f8-4f0b-8860-2f94c94e2c7a, timestamp=1454460846221, id=799de4c1-6167-6f45-de4d-1964d357d2dc, adapterRoutingKey=clientRoutingKey, messageType=RESPONSE, operationRoutingKey=login, amqp_consumerTag=amq.ctag-YMP8rrbTxLAXdYsFJ7HVjQ, result=OK, metaData={"clientVersion":"1.2.1","clientType":"WEB","ipAddress":"127.0.0.1"}}
So, the question here is, is there a way to format this output to kind of CSV format using an expression? This make me think that maybe this could be solved with a logback layout.
This log events happen in two different times, with request and response, is there a way to log elapsed time between request and response? I was using Splunk to process both log entries and calculate elapsed time but I would like to have this in one log line and avoid Splunk because license cost.
Thanks for your help.
UPDATE:
At the end, I used what Artem Bilan proposed with enrichHeaders(), I added timestamp in the request and when response is going to send back calculate elapsed time.
Also I have to say, that this requirement about logging in CSV format is more than just logging, so decided to handle this in a different way using AMQP to track every transaction and have it in the desired format.
Not sure what you mean about "CSV format using an expression". But you really can use any bean from that expression and build any format from there. According your CSV requirements I can assume that you finally will aggregate all logs into one file. So, maybe something like headers.values(), would be enough for the start. But you should bear in mind that not all map instances will be sorted the same way.
The LoggingHandler isn't intended to do that. Its responsibility just log the message coming to is channel. That is a main message principle - loose coupling. There are a couple of techniques which can help you determine the proper solution.
2.1 Since you are going to to elapse the time between request and response (and looks like you already can do that with Splunk), so, you can aggregate them the similar way using .aggregate() from Spring Integration. And only after that send the single message with the desired time field to the logger.
2.2 Another way (and I found it as better one). Just calculate the request message before sending it to the process channel. E.g. .enrichHeaders() can help you with that. And use that header when the response is ready to return to calculate the elapsed time. And only from here send the single message to the logger.