Can we use Spring Integration to configure directory polling for files such that -
With 2 servers configured, polling occurs on 1 server and corresponding processing get distributed b/w both the servers.
Also, can we switch the polling on either of the servers on runtime ?
Edit -
Tried configuring JBDC MetaStore and run the two instances separately, able to poll and process but getting intermittently DeadLockLoserDataAccessException
Configuration below
#Bean
public MessageChannel fileInputChannel(){
return new DirectChannel();
}
#Bean(PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller(){
PollerMetadata pollermetadata = new PollerMetadata();
pollermetadata.setMaxMessagesPerPoll(-1);
pollermetadata.setTrigger(new PeriodicTrigger(1000));
return pollermetadata;
}
#Bean
#InBoundChannelAdapter(value = "fileInputChannel"){
FileReadingMessageSource source = new FileReadingMessageSource();
source.setDirectory("Mylocalpath");
FileSystemPersistentAcceptOnceFileListFilter acceptOnce = new FileSystemPersistentAcceptOnceFileListFilter();
ChainFileListFilter<File> chainFilter = ChainFileListFilter(".*\\.txt"));
chainFilter.addFilter(acceptOnce);
source.setFilter(chainFilter);
source.setUseWatchService(true);
source.setWatchEvents(FileReadingMessageSource.WatchEventType.CREATE,FileReadingMessageSource.WatchEventType.MODIFY);
return source;
}
#Bean
public IntegrationFlow processFileFlow(){
return IntegrationFlows.from("fileInputChannel")
.handle(service).get();
}
It is really one of the features of Spring Integration to easy implement a distributed solution. You just need add a messaging middle ware into your cluster infrastructure and have all the nodes connected to some destination for sending and receiving. A good example could be a SubscribableJmsChannel which you just simple can declare in your application context and all the nodes of your cluster are going to subscribe to this channel for round-robin consumption from JMS queue. It already wouldn't matter which node produces to this channel.
See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/jms.html#jms-channel.
Another sample of similar distributed channels are: AMQP, Kafka, Redis, ZeroMQ.
You also can have a shared message store and use it in the QueueChannel definition: https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#message-store
It is not clear what you mean about "poller on runtime", so I would suggest you to start a new SO thread with much more info.
See rules as a guidance: https://stackoverflow.com/help/how-to-ask
Related
I've got an issue connecting to an ActiveMQ Artemis cluster (AMQ from Red Hat in fact) through Spring's DefaultJmsListenerContainerFactory.
DefaultMessageListenerContainer makes use of only one connection, regardless of the number of consumers you specify through the concurrency parameter. The problem is that, in the cluster, there are 3 brokers configured at the moment (and, as a dev, I shouldn't care about the topology of the cluster). Since here is only one connection consumers are only listening to one broker.
To solve the issue I disabled the cache (i.e. setCacheLevel(CACHE_NONE) in the factory).
It "solved" the problem because now I can see the connections distributing on all the nodes of the cluster but it's not a good solution because connections are perpetually dropped and recreated and that makes a lot of overhead at the broker side (it makes me think of a Christmas Tree :D).
Can you guys tell me what's the correct approach to handle this?
I trie using a JmsPoolConnectionFactory, but I didn't get any good results till now. I still have only one connection.
I'm using Spring Boot 2.7.4 with Artemis Starter.
You can find below a code snippet of the actual config.
(Side note, I don't use Spring autoconfig because i need to be able to switch between ActiveMQ Artemis and the old ActiveMQ "Classic" implementation).
#Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setDestinationResolver(destinationResolver());
factory.setSessionTransacted(true);
factory.setConcurrency(config.getConcurrency());
//Set this to allow load balancing of connections to all members of the cluster
factory.setCacheLevel(DefaultMessageListenerContainer.CACHE_NONE);
final ExponentialBackOff backOff = new ExponentialBackOff(
config.getRetry().getInitialInterval(), config.getRetry().getMultiplier());
backOff.setMaxInterval(config.getRetry().getMaxDuration());
factory.setBackOff(backOff);
return factory;
}
ConnectionFactory connectionFactory() {
return new ActiveMQJMSConnectionFactory(
config.getUrl(), config.getUser(), config.getPassword());
}
DestinationResolver destinationResolver() {
final ActiveMQQueue activeMQQueue = new ActiveMQQueue(config.getQueue());
return (session, destinationName, pubSubDomain) -> activeMQQueue;
}
#JmsListener(destination = "${slp.amq.queue}")
public void processLog(String log) {
final SecurityLog securityLog = SecurityLog.parse(log);
fileWriter.write(securityLog);
logsCountByApplicationId.increment(securityLog.getApplicationId());
if (elasticClient != null) {
elasticClient.write(securityLog);
}
}
The connection URL is:
(tcp://broker1:port,tcp://broker2:port,tcp://broker3:port)?useTopologyForLoadBalancing=true
The cluster can be configured so that any consumer on any node can consume messages sent to any node. Therefore, you shouldn't strictly need to "target all nodes" of the cluster with your consumer. Message redistribution and re-routing in the cluster should be transparent to your application. As you said, as a developer you shouldn't care about the topology of the cluster.
That said, the goal of clustering is to increase overall message throughput (i.e. performance) via horizontal scaling. Furthermore, every node in the cluster should ideally have sufficient producers and consumers so that messages aren't being redistributed or re-routed between cluster nodes as that's not optimal for performance. If you're in a situation where you have just a few consumers connected to your cluster then it's likely you don't actually need a cluster in the first place. A single ActiveMQ Artemis broker can handle millions of messages per second in certain use-cases.
I'm trying to write a basic ActiveMQ client to listen to a topic. I'm using Spring Boot ActiveMQ. I have an implementation built off of various tutorials that uses DefaultJmsListenerContainerFactory, but I am having some issues getting it working properly.
#Configuration
#EnableJms
public class JmsConfig {
#Bean
public DefaultJmsListenerContainerFactory jmsContainerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
factory.setConcurrency("3-10");
factory.setConnectionFactory(connectionFactory);
configurer.configure(factory, connectionFactory);
factory.setSubscriptionDurable(true);
factory.setClientId("someUniqueClientId");
return factory;
}
}
#JmsListener(destination="someTopic", containerFactory="jmsContainerFactory", subscription="someUniqueSubscription")
public void onMessage(String msg) {
...
}
Everything works fine, until I try to get a durable subscription going. When I do that, I'm finding that with the client id set on the container factory, I get an error about how the client id cannot be set on a shared connection.
Cause: setClientID call not supported on proxy for shared Connection. Set the 'clientId' property on the SingleConnectionFactory instead.
When I change the code to set the client id on the connection factory instead (it's a CachingConnectionFactory wrapping an ActiveMQConnectionFactory), the service starts up successfully, reads a couple messages and then starts consistently outputting this error:
Setup of JMS message listener invoker failed for destination 'someTopic' - trying to recover. Cause: Durable consumer is in use for client: someUniqueClientId and subscriptionName: someUniqueSubscription
I continue to receive messages, but also this error inter-mingled in the logs. This seems like it is probably a problem, but I'm really not clear on how to fix it.
I do have a naive implementation of this going without any spring code, using ActiveMQConnectionFactory directly and it seems happy to use a durable consumer (but it has its own different issues). In any case, I don't think it's a lack of support for durable connections on the other side.
I'm hoping someone with more experience in this area can help me figure out if this error is something I can ignore, or alternatively what I need to do to address it.
Thanks!
JMS 1.1 (which is what you're using since you're using ActiveMQ 5.x) doesn't support shared durable subscriptions. Therefore, when you use setConcurrency("3-10") and Spring tries to create > 1 subscription you receive an error. I see two main ways to solve this problem:
Use setConcurrency("1") which will limit the number of subscribers/consumers to 1. Depending on your requirements this could have a severe negative performance impact.
Switch to ActiveMQ Artemis which does support JMS 2.0 and invoke setSubscriptionShared(true).
Our application consumes data from several queues that are provided by RabbitMQ. To increase throughput we start several threads per queue that do blocking takes from those queues.
For a new service we want to use Spring Boot and again have several threads per queue that take data from those queues. Here is the canonical Spring Boot code for processing data that arrived from some queue:
#StreamListener(target = Processor.INPUT)
#SendTo(Processor.OUTPUT)
public Message<SomeData> process(Message<SomeData> message) {
SomeData result = service.process(message.getPayload());
return MessageBuilder
.withPayload(result)
.copyHeaders(message.getHeaders())
.build();
}
Question is now how to make Spring Boot spawn several threads to serve one queue instead of a single thread. Throughput is very critical for our application, hence the need for this.
Check the available properties, search for rabbitmq.
spring.rabbitmq.listener.simple.concurrency= # Minimum number of listener invoker threads
That looks promising
You can set the concurrent consumers for the queue when you configurate it.
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory
(MessageConverter contentTypeConverter,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
// the number of consumers is set as 5
factory.setConcurrentConsumers(5);
configurer.configure(factory, connectionFactory);
factory.setMessageConverter(contentTypeConverter);
return factory;
}
I have the following configurations for Spring and RabbitMQ:
Spring Boot : 1.2.7
RabbitMQ : 3.5.4
I am using the following Spring beans to create Stomp endpoint (My config class extends AbstractWebSocketMessageBrokerConfigurer):
#Bean
public TopicExchange streamingExchange(#Qualifier("admin") final RabbitAdmin rabbitAdmin) {
TopicExchange topicExchange = new TopicExchange(exchangeName, true, false);
topicExchange.setAdminsThatShouldDeclare(rabbitAdmin);
return topicExchange;
}
#Override
public void configureMessageBroker(final MessageBrokerRegistry config) {
config.enableStompBrokerRelay("/my_stream", "/test").setRelayHost(host)
.setSystemLogin(username).setSystemPasscode(password).setClientLogin(username)
.setClientPasscode(password);
}
#Override
public void registerStompEndpoints(final StompEndpointRegistry registry) {
registry.addEndpoint("/test").setAllowedOrigins("*").withSockJS();
}
Now, when a client connects to this end point, a temporary queue gets created and response data is streamed through the queue. If clients get disconnected, queue gets deleted and messages are lost.
To prevent this, I want to create durable queues (as these queues are have durable set to false and auto-delete set to true) if not, I want to have some expiration set on these queues (e.g. 1 hour or something). From RabbitMQ documentation, it seems we can pass these values in headers, however, that is only applicable for versions 3.6.0 onwards, as we have 3.5.4, it's not an option.
Is there any other way by which we can configure this? (Another approach would be to add some kind of Listener for connect request and configure queue parameters programmatically? I am not sure whether this is feasible as I don't know much about spring rabbitmq stomp plugin)
Wondering if you have tried declaring the queue as durable using the rabbitmqadmin tool ?
rabbitmqadmin declare queue name=your-queue durable=true
Admin tool can be downloaded from here https://www.rabbitmq.com/management-cli.html
I have a Java based web app hosted on AWS. It is read-mostly so it makes a lot of sense to cache objects retrieved from the database for performance.
When I do update an object, I would like to be able to broadcast to all the servers that the object was saved and it should be invalidated from all local caches.
The does not need to be real time. Stale objects are annoying and need to be flushed within about 20 seconds. Users notice if they stick around for minutes. Cache invalidation does not have to happen the millisecond that objects get saved.
What I've thought about
I've looked into broadcast technologies just as jGroups, but jGroups isn't supported on AWS.
I don't think that Amazon's SQS messaging service can be made into a broadcast service.
I'm considering using the database for this purpose: I'd write events to a database table and have each server poll this table every few seconds to get a new list items.
Two options come to mind. The first is to use Amazon SNS, which can use SQS as a delivery backend. This might be overkill, though, since it's designed as a frontend to lots of delivery types, including e-mail and SMS.
The approach I'd try is something along the lines of Comet-style push notifications. Have each machine with a cache open a long-lived TCP connection to the server who's responsible for handling updates, and send a compact "invalidate" message from that server to everyone who's listening. As a special-purpose protocol, this could be done with minimal overhead, perhaps just by sending the object ID (and class if necessary).
Redis is handy solution for broadcasting a message to all subscribers on a topic. It is convenient because it can be used as a docker container for rapid prototyping, but is also offered by AWS as a managed service for multi-node clusters.
Setting up a ReactiveRedisOperations bean:
#Bean
public ReactiveRedisOperations<String, Notification> notificationTemplate(LettuceConnectionFactory lettuceConnectionFactory){
RedisSerializer<Notification> valueSerializer = new Jackson2JsonRedisSerializer<>(Notification.class);
RedisSerializationContext<String, Notification> serializationContext = RedisSerializationContext.<String, Notification>newSerializationContext(RedisSerializer.string())
.value(valueSerializer)
.build();
return new ReactiveRedisTemplate<>(lettuceConnectionFactory, serializationContext);
}
Subscribing on a topic:
#Autowired
private ReactiveRedisOperations<String, Notification> reactiveRedisTemplate;
#Value("${example.topic}")
private String topic;
#PostConstruct
private void init() {
this.reactiveRedisTemplate
.listenTo(ChannelTopic.of(topic))
.map(ReactiveSubscription.Message::getMessage)
.subscribe(this::processNotification);
}
Publishing a message on a topic:
#Autowired
private ReactiveRedisOperations<String, Notification> redisTemplate;
#Value("${example.topic}")
private String topic;
public void publish(Notification notification) {
this.redisTemplate.convertAndSend(topic, notification).subscribe();
}
RedisInsight is a GUI that can be used for interacting with redis.
Here is a complete sample implementation using spring-data-redis.