I have this very basic setup of IntegrationFlow with Spring Integration Java DSL:
#IntegrationComponentScan
#EnableIntegration
#Configuration
public class DummyConfig {
#MessagingGateway
public interface DummyGateway {
#Gateway(requestChannel = "dummyInChannel")
void echo(String payload);
}
#Bean(name = "dummyInChannel")
public MessageChannel dummyInChannel() {
return MessageChannels.direct().get();
}
#Bean
public IntegrationFlow dummyFlow() {
return IntegrationFlows.from(dummyInChannel())
.handle(String.class, (payload, headers) -> {
System.out.println(payload);
return "";
})
.get();
}
}
When I try to post a message to my gateway
dummyGateway.echo("test");
I'm getting and exception:
Caused by: org.springframework.messaging.MessageDeliveryException:
Dispatcher has no subscribers for channel 'application.dummyInChannel'.; nested exception
is org.springframework.integration.MessageDispatchingException: Dispatcher
has no subscribers, failedMessage=GenericMessage [payload=test,
headers={replyChannel=nullChannel, id=6e4302e4-95f0-bf5a-c1a3-e8cd587c23fb, timestamp=1643269549272}]
I thought, that doing .handle() in my flow is exactly subscribing to a channel. Then, why am I getting this exception? How to properly subscribe to my channel in this scenario?
No, the ctor is too early. The beans are created at this point, but they have not started their heavy lifting. You cannot do low-level resources interaction (technically any actions) from the bean initialization phase. You need to wait until application context is fully started. Please, learn a lifecycle of Spring container: https://docs.spring.io/spring-framework/docs/current/reference/html/core.html#beans-factory-lifecycle-processor .
You may implement a SmartLifecycle, or listener for the ContextStartedEvent. But bean initialization phase is really too early to start emitting messages.
The QueueChannel works because it has its own internal buffer to keep messages until they are consumed. And they are consumed when the endpoint is started. In case of DirectChannel there is no buffer and consumer is called immediately, when we send a message. There is just no subscriber yet on that channel within bean initialization phase.
public static void main(String[] args) {
SpringApplication application = new SpringApplication(DummyConfig.class);
ConfigurableApplicationContext ctx = application.run(args);
ctx.getBean(DummyGateway.class).echo("MyAwesomeString");
ctx.close();
}
Related
In my testing and review of the Artemis LastValueQueue code, it looks like the scheduling delay for a message takes precedence over its evaluation of the "last-value-key". In other words, if you schedule a message, it is only evaluated for replacing the last-value in the queue at the time it is prepared for delivery.
My question is whether I have correctly understood the code, and if so, if there's a workaround or a feature of ActiveMQ / Artemis that might help meet our requirements.
Our requirements are as follows:
Generate a message, and delay processing of that message to a point in the future (usually 30 seconds out).
If an updated version of the message is generated due to a new external event, replace any existing scheduled message with the new version of the message - the scheduled delivery time should also be updated, in addition to the message payload.
Some other notes:
My current prototype is using Artemis embedded server
Spring-jms JmsTemplate is being used to produce messages
Spring-jms JmsListenerContainerFactory is being used to consume messages
We don't currently use SpringBoot, so you'll see some bean setup below.
ArtemisConfig.java:
#Configuration
#EnableJms
public class ArtemisConfig {
#Bean
public org.apache.activemq.artemis.core.config.Configuration configuration() throws Exception {
org.apache.activemq.artemis.core.config.Configuration config = new ConfigurationImpl();
config.addAcceptorConfiguration("in-vm", "vm://0");
config.setPersistenceEnabled(true);
config.setSecurityEnabled(false);
config.setJournalType(JournalType.ASYNCIO);
config.setCreateJournalDir(true);
config.setJournalDirectory("/var/mq/journal");
config.setBindingsDirectory("/var/mq/bindings");
config.setLargeMessagesDirectory("/var/mq/large-messages");
config.setJMXManagementEnabled(true);
QueueConfiguration queueConfiguration = new QueueConfiguration("MYLASTVALUEQUEUE");
queueConfiguration.setAddress("MYLASTVALUEQUEUE");
queueConfiguration.setLastValueKey("uniqueJobId");
queueConfiguration.setDurable(true);
queueConfiguration.setEnabled(true);
queueConfiguration.setRoutingType(RoutingType.ANYCAST);
CoreAddressConfiguration coreAddressConfiguration = new CoreAddressConfiguration();
coreAddressConfiguration.addQueueConfiguration(queueConfiguration);
config.addAddressConfiguration(coreAddressConfiguration);
return config;
}
#Bean
public EmbeddedActiveMQ artemisServer() throws Exception {
EmbeddedActiveMQ server = new EmbeddedActiveMQ();
server.setConfiguration(configuration());
server.start();
return server;
}
#PreDestroy
public void preDestroy() throws Exception {
artemisServer().stop();
}
#Bean
public ConnectionFactory activeMqConnectionFactory() throws Exception {
return ActiveMQJMSClient.createConnectionFactory("vm://0", "artemis-client");
}
#Bean
public JmsListenerContainerFactory<?> jmsListenerContainerFactory() throws Exception {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(activeMqConnectionFactory());
factory.setSessionTransacted(true);
factory.setConcurrency("8");
factory.setMessageConverter(jacksonJmsMessageConverter());
return factory;
}
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
#Bean
public JmsTemplate jmsTemplate() throws Exception {
JmsTemplate jmsTemplate = new JmsTemplate(activeMqConnectionFactory());
jmsTemplate.setMessageConverter(jacksonJmsMessageConverter());
jmsTemplate.setDeliveryPersistent(true);
return jmsTemplate;
}
#Bean
QueueMessageService queueMessageService() {
return new QueueMessageService();
}
}
QueueMessageService.java
public class QueueMessageService {
#Resource
private JmsTemplate jmsTemplate;
public void queueJobRequest(
final String queue,
final int priority,
final long deliveryDelayInSeconds,
final MyMessage message) {
jmsTemplate.convertAndSend(queue, jobRequest, message -> {
message.setJMSPriority(priority);
if (deliveryDelayInSeconds > 0 && deliveryDelayInSeconds <= 86400) {
message.setLongProperty(
Message.HDR_SCHEDULED_DELIVERY_TIME.toString(),
Instant.now().plus(deliveryDelayInSeconds, ChronoUnit.SECONDS).toEpochMilli()
);
}
message.setStringProperty(Message.HDR_LAST_VALUE_NAME.toString(), "uniqueJobId");
message.setStringProperty("uniqueJobId", jobRequest.getUniqueJobId().toString());
return message;
});
}
}
Your understanding about the semantics of scheduled messages with a last-value queue is correct. When a message is scheduled it is not technically on the queue yet. It is not put onto the queue until the scheduled time arrives at which point last-value queue semantics are enforced.
Short of implementing a new feature I don't see how you can implement your desired behavior in any kind of automatic way. My recommendation at this point would be to use the management API (i.e. QueueControl) to manually remove the "old" scheduled message before you send the "new" scheduled message. You can use one of the removeMessage methods for this as they will work on scheduled messages and non-scheduled messages alike.
I have spring integration flow that gets triggered once a every day, that pulls all parties from database and sends each party to an executorChannel.
The next flow would pull data for each party and then process them parallelly by sending in to a different executor channel.
Challenge i'm facing is how do i know when this entire process ends. Any ideas on how to acheve this .
Here's my pseudo code of executor channels and integration flows.
#Bean
public IntegrationFlow fileListener() {
return IntegrationFlows.from(Files.inboundAdapter(new
File("pathtofile"))).channel("mychannel").get();
}
#Bean
public IntegrationFlow flowOne() throws ParserConfigurationException {
return IntegrationFlows.from("mychannel").handle("serviceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowOne() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelOne").handle("parallelServiceHandlerOne",
"handle").nullChannel();
}
#Bean
public IntegrationFlow parallelFlowTwo() throws ParserConfigurationException {
return IntegrationFlows.from("executorChannelTwo").handle("parallelServiceHandlerTwo",
"handle").nullChannel();
}
#Bean
public MessageChannel executorChannelOne() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Bean
public MessageChannel executorChannelTwo;() {
return new ExecutorChannel(
Executors.newFixedThreadPool(10));
}
#Component
#Scope("prototype")
public class ServiceHandlerOne{
#Autowired
MessageChannel executorChannelOne;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("parties");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelOne.send(message);
});
return message;
}
}
#Component
#Scope("prototype")
public class ParallelServiceHandlerOne{
#Autowired
MessageChannel executorChannelTwo;;
#ServiceActivator
public Message<?> handle(Message<?> message) {
List<?> rowDatas = repository.findAll("party");
rowDatas.stream().forEach(data -> {
Message<?> message = MessageBuilder.withPayload(data).build();
executorChannelTwo;.send(message);
});
return message;
}
}
First of all no reason to make your services as #Scope("prototype"): I don't see any state holding in your services, so they are stateless, therefore can simply be as singleton. Second: since you make your flows ending with the nullChannel(), therefore point in returning anything from your service methods. Therefore just void and flow is going to end over there naturally.
Another observation: you use executorChannelOne.send(message) directly in the code of your service method. The same would be simply achieved if you just return that new message from your service method and have that executorChannelOne as the next .channel() in your flow definition after that handle("parallelServiceHandlerOne", "handle").
Since it looks like you do that in the loop, you might consider to add a .split() in between: the handler return your List<?> rowDatas and splitter will take care for iterating over that data and replies each item to that executorChannelOne.
Now about your original question.
There is really no easy to say that your executors are not busy any more. They might not be at the moment of request just because the message for task has not reached an executor channel yet.
Typically we recommend to use some async synchronizer for your data. The aggregator is a good way to correlate several messages in-the-flight. This way the aggregator collects a group and does not emit reply until that group is completed.
The splitter I've mentioned above adds a sequence details headers by default, so subsequent aggregator can track a message group easily.
Since you have layers in your flow, it looks like you would need a several aggregators: two for your executor channels after splitting, and one top level for the file. Those two would reply to the top-level for the final, per-file grouping.
You also may think about making those parties and party calls in parallel using a PublishSubscribeChannel, which also can be configured with a applySequence=true. This info then will be used by the top-level aggregator for info per file.
See more in docs:
https://docs.spring.io/spring-integration/docs/current/reference/html/core.html#channel-implementations-publishsubscribechannel
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#splitter
https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#aggregator
Hi I was trying to build a Spring boot application , with spring integration
Application 1 : Publisher
Jms Message -> Broker ->queue1
Application 2: Subscriber & Publisher
Broker->queue1->Transform->HTTP CALL->HTTP Response->JMS Message->Broker->queue2
Publisher Flow
#Configuration
public class EchoFlowOutBound {
#Autowired
private ConnectionFactory connectionFactory;
#Bean
public IntegrationFlow toOutboundQueueFlow() {
return IntegrationFlows.from("requestChannel")
.handle(Jms.outboundGateway(connectionFactory)
.requestDestination("amq.outbound1")).get();
}
}
//Gateway
#MessagingGateway
public interface EchoGateway {
#Gateway(requestChannel = "requestChannel")
String echo(String message);
}
Subscriber & Publisher Flow
#Configuration
public class MainOrchestrationFlow {
#Autowired
private ConnectionFactory connectionFactory;
#Autowired
private QueueChannel jmsOutChannel;
#Bean
public IntegrationFlow orchestrationFlow() {
return IntegrationFlows.from(
Jms.messageDrivenChannelAdapter(connectionFactory)
.destination("amq.outbound1")
.outputChannel(jmsOutChannel))
.<String, String>transform(s -> {
return s.toLowerCase();
})
// HTTP part goes here
.<String, HttpEntity>transform(HttpEntity::new)
.handle(
Http.outboundChannelAdapter("http://localhost:8080/uppercase")
.httpMethod(HttpMethod.POST)
.extractPayload(true)
.expectedResponseType(String.class)
)
// and here HTTP part ends
.handle(
Jms.outboundAdapter(connectionFactory)
.destination("amq.outbound2")
)
.get();
}
}
When i run the application, I'm getting error
Caused by: org.springframework.integration.MessageTimeoutException:
failed to receive JMS response within timeout of: 5000ms at
org.springframework.integration.jms.JmsOutboundGateway.handleRequestMessage(JmsOutboundGateway.java:762)
~[spring-integration-jms-5.0.6.RELEASE.jar:5.0.6.RELEASE] at
org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
~[spring-integration-core-5.0.6.RELEASE.jar:5.0.6.RELEASE] at
org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:158)
~[spring-integration-core-5.0.6.RELEASE.jar:5.0.6.RELEASE] at
org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
~[spring-integration-core-5.0.6.RELEASE.jar:5.0.6.RELEASE]
Can someone tell me what I'm doing wrong,
Your problem that your consumer is not a request-reply. You receive message from the amq.outbound1 and send to the amq.outbound2. That's all: nothing more happens. You have there a one-way flow.
At the same time your producer is a request-reply - handle(Jms.outboundGateway(connectionFactory). That Outbound Gateway really expects a reply in the ReplyTo header according default options for JMS request-reply scenarios.
So, you have to determine for yourself: or you need send reply back to the producer or you just need send-and-forget from that producer. See Jms.outboundAdapter() if that.
In case of request-reply you don't need a Jms.outboundAdapter() on the consumer side: you must use a Jms.inboundGateway() instead of the Jms.messageDrivenChannelAdapter().
Environment
Spring Boot: 1.5.13.RELEASE
Cloud: Edgware.SR3
Cloud AWS: 1.2.2.RELEASE
Java 8
OSX 10.13.4
Problem
I am trying to write an integration test for SQS.
I have a local running localstack docker container with SQS running on TCP/4576
In my test code I define an SQS client with the endpoint set to local 4576 and can successfully connect and create a queue, send a message and delete a queue. I can also use the SQS client to receive messages and pick up the message that I sent.
My problem is that if I remove the code that is manually receiving the message in order to allow another component to get the message nothing seems to be happening. I have a spring component annotated as follows:
Listener
#Component
public class MyListener {
#SqsListener(value = "my_queue", deletionPolicy = ON_SUCCESS)
public void receive(final MyMsg msg) {
System.out.println("GOT THE MESSAGE: "+ msg.toString());
}
}
Test
#RunWith(SpringRunner.class)
#SpringBootTest(properties = "spring.profiles.active=test")
public class MyTest {
#Autowired
private AmazonSQSAsync amazonSQS;
#Autowired
private SimpleMessageListenerContainer container;
private String queueUrl;
#Before
public void setUp() {
queueUrl = amazonSQS.createQueue("my_queue").getQueueUrl();
}
#After
public void tearDown() {
amazonSQS.deleteQueue(queueUrl);
}
#Test
public void name() throws InterruptedException {
amazonSQS.sendMessage(new SendMessageRequest(queueUrl, "hello"));
System.out.println("isRunning:" + container.isRunning());
System.out.println("isActive:" + container.isActive());
System.out.println("isRunningOnQueue:" + container.isRunning("my_queue"));
Thread.sleep(30_000);
System.out.println("GOT MESSAGE: " + amazonSQS.receiveMessage(queueUrl).getMessages().size());
}
#TestConfiguration
#EnableSqs
public static class SQSConfiguration {
#Primary
#Bean(destroyMethod = "shutdown")
public AmazonSQSAsync amazonSQS() {
final AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration("http://127.0.0.1:4576", "eu-west-1");
return new AmazonSQSBufferedAsyncClient(AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("key", "secret")))
.withEndpointConfiguration(endpoint)
.build());
}
}
}
In the test logs I see:
o.s.c.a.m.listener.QueueMessageHandler : 1 message handler methods found on class MyListener: {public void MyListener.receive(MyMsg)=org.springframework.cloud.aws.messaging.listener.QueueMessageHandler$MappingInformation#1cd4082a}
2018-05-31 22:50:39.582 INFO 16329 ---
o.s.c.a.m.listener.QueueMessageHandler : Mapped "org.springframework.cloud.aws.messaging.listener.QueueMessageHandler$MappingInformation#1cd4082a" onto public void MyListener.receive(MyMsg)
Followed by:
isRunning:true
isActive:true
isRunningOnQueue:false
GOT MESSAGE: 1
This demonstrates that in the 30 second pause between sending the message the container didn't pick it up and when I manually poll for the message it is there on the queue and I can consume it.
My question is, why isn't the listener being invoked and why is the isRunningOnQueue:false line suggesting that it's not auto started for that queue?
Note that I also tried setting my own SimpleMessageListenerContainer bean with autostart set to true explicitly (the default anyway) and observed no change in behaviour. I thought that the org.springframework.cloud.aws.messaging.config.annotation.SqsConfiguration#simpleMessageListenerContainer that is set up by #EnableSqs ought to configure an auto started SimpleMessageListenerContainer that should be polling for me message.
I have also set
logging.level.org.apache.http=DEBUG
logging.level.org.springframework.cloud=DEBUG
in my test properties and can see the HTTP calls create the queue, send a message and delete etc but no HTTP calls to receive (apart from my manual one at the end of the test).
I figured this out after some tinkering.
Even if the simple message container factory is set to not auto start, it seems to do its initialisation anyway, which involves determining whether the queue exists.
In this case, the queue is created in my test in the setup method - but sadly this is after the spring context is set up which means that an exception occurs.
I fixed this by simply moving the queue creation to the context creation of the SQS client (which happens before the message container is created). i.e.:
#Bean(destroyMethod = "shutdown")
public AmazonSQSAsync amazonSQS() {
final AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration("http://localhost:4576", "eu-west-1");
final AmazonSQSBufferedAsyncClient client = new AmazonSQSBufferedAsyncClient(AmazonSQSAsyncClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("dummyKey", "dummySecret")))
.withEndpointConfiguration(endpoint)
.build());
client.createQueue("test-queue");
return client;
}
Here is a small Spring program that is expected to insert a message into a rabbitmq queue:
public class Main {
public static void main(String [] args) throws IOException {
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(QueueConfiguration.class);
AmqpTemplate template = context.getBean(AmqpTemplate.class);
template.convertAndSend("asdflk ...");
context.destroy();
}
}
The ApplicationContext is as follows:
#Configuration
public class QueueConfiguration {
#Bean
public ConnectionFactory connectionFactory() {
return new CachingConnectionFactory("192.168.1.39");
}
#Bean
public RabbitTemplate rabbitTemplate() {
return new RabbitTemplate(connectionFactory());
}
}
When I check the contents of the queues on the server, nothing gets inserted. I also tried to set the name of the exchange or the name of the queue on the RabbitTemplate, but still nothing shows up on the server.
The log of the application does not show any errors, but logs this:
17:28:02.441 [main] DEBUG o.s.amqp.rabbit.core.RabbitTemplate - Executing callback on RabbitMQ Channel: Cached Rabbit Channel: AMQChannel(amqp://guest#192.168.1.39:5672/,1)
17:28:02.441 [main] DEBUG o.s.amqp.rabbit.core.RabbitTemplate - Publishing message on exchange [], routingKey = []
Any ideas what's wrong?
I had to give the queue as a parameter in the call to convertAndSend():
template.convertAndSend("hello2", "asdflk ...");
Still wondering why spring-amqp would not throw an exception. Anybody knows where the messages are delivered when no queue is given?
I think I will keep the Routing Key and Queue name in the bean rabbitTemplate() as per spring-amqp example. Since I am working with multiple queues currently I have different class for each queue in which I have the rabbitTemplate like this:
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
//The routing key = name of the queue in default exchange.
template.setRoutingKey("MyQueue");
// Queue name
template.setQueue("MyQueue");
return template;
}
Are you using tomcat to deploy this? If yes then these can be loaded at startup which will initialize all the connection/channel/queues etc as well.