I am planning to write code that's similar to producer and consumer using ExecutorService with a fixed threadpool and IBM MQ messaging.
Suppose as consumer I created 10 fixed threads. How it will handle it if I place 10 messages in consumer queue? How 10 consumer worker threads will cover below scenarios?
each worker thread take single message synchronously and process the message?
each consumer worker threads take all these 10 message like 1 worker thread per one message?
After reading this message as second scenario above ,How each thread call executor service.Is it done concurrenly or synchronously.
if there are 20 messages in queue,how consumer worker thread takes these message,each thread takes 2 messages? If it takes one message per one thread what will happen to
other 10 messages?
While processing the above scenarios there is webservice call and internal api method calls but those are synchronous methods. So is there any use if i implement this class to process the code concurrently?
If you're running in an application server, like WebSphere for example, then you can simply deploy a Message Driven Bean (MDB) onto the JMS queue, and it will do pretty much exactly what you're describing.
If you're just building a Java application, then using the ExecutorService would work. Start by placing a MessageListener onto the Session, and have the onMessage() of that listener submit() a processor (e.g. a Runnable) of the message to the ExecutorService. Once that processor has done its work, it should acknowledge() the message.
I don't think it's a good idea to implement producer/consumer scheme when you have good tools like JMS already implemented, debugged, support many features and supported by frameworks (like Spring for instance).
Don't invent the wheel!
Related
I am trying to use Spring Integration for a batch process. There are certain steps that are time consuming and hence would benefit with a QueueChannel with multiple consumers each running on a separate thread.
The problem with this approach is that there is no clean way to shut-down the application after all the messages have been consumed. I have tried using a control bus and shutting down the task executor but that only works if you can guess by what time all messages would have been consumed and none are in flight which is impossible.
Is there a clean way to do this for a batch process or is this just a wrong use case to use Spring Integration in ?
EDIT:
Essentially it would be nice if there was a way for me to send a special message which signifies one of the lifecycle events like start or stop which is automatically carried through all the spring integration components. This way the stop message is guaranteed to reach last and if there is a way to trigger shutdown() on the lifecycle aware beans when stop message reaches them.
shut-down the application after all the messages have been consumed
If you know the number of messages in the beginning, you can have an AtomicInteger bean and increment it every time a message is processed. Separately you have an inbound channel adapter to poll the current state of that AtomicInteger and make a decision to shutdown your app.
Alternatively you can use an aggregator to gather results for all the messages and shutdown the app in the downstream flow after that aggregator.
I am using ActiveMQ classic as a queue manager. My message consumer (#JmsListener using Spring) writes to MongoDB. If MongoDB is unavailable, then it sends the message to a different queue, lets call it a redelivery queue.
So, imagine after mongoDB been down for many hours, its finally up. What is the best way to now read the message from this redelivery queue?
I am thinking if there is a possibility to do this by creating a batch job that runs once a day? If so, what are the options that can be used to create a job like that or if there are any other better options available.
There is no "batch" mode for JMS. A JMS consumer can only receive one message at a time. Typically the best way boost message throughput to deal with lots of messages is by increasing the number of consumers. This should be fairly simple to do with a Spring JmsListener using the concurrency setting.
You can, of course, use something like cron to schedule a job to deal with these messages or you use something like the Quartz Job Scheduler instead.
It's really impossible to give you the "best" way to deal with your situation on Stack Overflow. There are simply too many unknown variables.
I have an application which consumes a message from RabbitMQ using Spring AMQP.
I have to implement Threads in consumer to handle the request. If my threads pool are available , i am consuming the message using threads i will process the message.
I have a question where what happens when all the threads are busy. I
dont have a threads to process it. Will the message be consumed from
RabbitMQ ? Will it wait till my thread pool becomes available. How do
handle this using spring amqp?
Is there any thread logic to be implemented from Spring AMQP side as well?
Please suggest.
You should not add your own threading in your listener, it will cause messages to be ack'd early and potentially lost. Instead, use the container's concurrentConsumers property to determine how many threads to use.
I suggest you read about all the configuration options before asking questions like this here.
Using Spring-AMQP abstraction API, how do you implement work or task queue? I have a single producer and multiple worker consumers. Consumers pickup next item from the queue and processes it. This is supported when you use RabbitMQ client API directly but I don't see any support in Spring-AMQP. Am I missing something here? Is there another way to implement worker queue?
Have you read the reference documentation?
When configuring multiple listener containers to listen to the same queue, each will compete for messages from that queue - the "work" will be distributed to them by the server.
Also, increasing the concurrency (concurrentConsumers) within a container (from the default 1) will create multiple consumers within each container (with each running on a separate thread).
I have a simple DropWizard service and I'd like a REST API to start a long running processing task - both CPU and I/O bound. REST call will not wait for task completion, notification will happen by polling/long polling/web socket.
For now, I'd prefer if I can do this in Dropwizard and keep everything in single deployable JAR. What are my options?
UPDATE: I am interested in what my options are regarding running long running tasks in Dropwizard, deployed as single jar without external dependencies. Just spawn a new thread? Assuming there are just few such requests it would probably work but there should be better options.
You probably want to use a managed resource:
https://dropwizard.io/en/stable/manual/core.html#managed-objects
to setup a thread pool. Then, your initial request could push a message onto a queue. Your thread pool can pull messages off the queue and process them asynchronously.
You could maybe provide an additional endpoint so that clients can obtain the current state of the asynchronous process.