I need to setup RabbitMQ in an attempt to redesign our architecture using asynchronous messaging.
Existing Application Flow:
JEE web application (via browser) creates a new thread.
This thread creates a new OS process to invoke a Perl script to do some processing.
Perl script writes its output in a file and the control comes back to the thread.
The thread then reads the output file and loads the results to the database.
The control passes to the servlet which displays the result to the UI.
All these are synchronous and time consuming and we need to convert this to asynchronous messaging communication.
Now, I am planning to break this down to the following different components but not sure if this would work with RabbitMQ:
Application Breakdown:
JEE Web Application which is the Producer for RabbitMQ
Separate the Perl Script in to its own application that supports RabbitMQ communication. This Perl client will consume the message, process it and places a new message in RabbitMQ for the next step
Separate the output file to database loader into its own Java application that suppors RabbitMQ communication. This would consume the message from the queue corresponding to the Perl client's message from the previous step.
This way, the output would be available in the database and the asynchronous flow would be completed.
Is is possible to separate the applications this way compatible to RabbitMQ?
Are there any better ways to do this?
Please suggest some framework components for RabbitMQ and Perl
Appreciate your inputs with this.
Yes, you can do it that way. If it's not a hard work, I'll include the database load on the Perl step. This probably avoids to handle an intermediate file, but I don't know if it's a viable task on your project.
In order to use RabbitMQ, I'll recommend you the AnyEvent::RabbitMQ CPAN module. As the documentation establishes, You can use AnyEvent::RabbitMQ to:
Declare and delete exchanges
Declare, delete, bind and unbind queues
Set QoS and confirm mode
Publish, consume, get, ack, recover and reject messages
Select, commit and rollback transactions
Related
We are trying to migrate our legacy system to Micro service
With Paas environment, we have scheduler jobs to trigger and put messages in MQ one by one and we have MQ listener in our Microservice to get message and create request and send request to external party.
Here the problem comes our micro service is capable doing Asynchronous call to external service, but our external service is not able to handle Asynchronous call so it is returning wrong data.
For example, we are hitting external service with 40 to 60 request per minute and external service is capable to handle only 6 request per minute.
So how can I make the MQ listener to process slowly.
I have tried reducing setMaxConcurrenceConsumer to 1 and
Used observable.toblocking.single() to make the process to run in only one thread.
We use RxJava in our micro service.
It sounds like either your micro service or the external service is not following the use case for Request-Reply messaging.
(1) Is the external service setting the Reply's message Correlation ID with the Request message's Message ID?
(2) Is your micro service performing an MQGET with the matching option of getting by Correlation ID.
You can blame the external service for the error but if your micro service is actually picking up the wrong message then it is your application's fault. i.e. Does your micro service simply get the "next" message on the queue?
Read this answer: How to match MQ Server reply messages to the correct request
Here's a explanation (looks like from the 90's but has good information): https://www.enterpriseintegrationpatterns.com/patterns/messaging/RequestReplyJmsExample.html
In long term approach we are planning to migrate the External service to as well.
In short time i have fixed it using the observable.toblocking.single() ,thread.sleep(), and setMaxConcurrenceConsumer() to 1 so only one thread will run at a time. which will avoid the Asynchronous call to external service.The sleep time will set dynamically with some analysis done on the external service.
We have configured storm cluster with one nimbus server and three supervisors. Published three topologies which does different calculations as follows
Topology1 : Reads raw data from MongoDB, do some calculations and store back the result
Topology2 : Reads the result of topology1 and do some calculations and publish results to a queue
Topology3 : Consumes output of topology2 from the queue, calls a REST Service, get reply from REST service, update result in MongoDB collection, finally send an email.
As new bee to storm, looking for an expert advice on the following questions
Is there a way to externalize all configurations, for example a config.json, that can be referred by all topologies?
Currently configuration to connect MongoDB, MySql, Mq, REST urls are hard-coded in java file. It is not good practice to customize source files for each customer.
Wanted to log at each stage [Spouts and Bolts], Where to post/store log4j.xml that can be used by cluster?
Is it right to execute blocking call like REST call from a bolt?
Any help would be much appreciated.
Since each topology is just a Java program, simply pass the configuration into the Java Jar, or pass a path to a file. The topology can read the file at startup, and pass any configuration to components as it instantiates them.
Storm uses slf4j out of the box, and it should be easy to use within your topology as such. If you use the default configuration, you should be able to see logs either through the UI, or dumped to disk. If you can't find them, there are a number of guides to help, e.g. http://www.saurabhsaxena.net/how-to-find-storm-worker-log-directory/.
With storm, you have the flexibility to push concurrency out to the component level, and get multiple executors by instantiating multiple bolts. This is likely the simplest approach, and I'd advise you start there, and later introduce the complexity of an executor inside of your topology for asynchronously making HTTP calls.
See http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html for the canonical overview of parallelism in storm. Start simple, and then tune as necessary, as with anything.
We have a requirement, where we have to run many async background processes which accesses DBs, Kafka queues, etc. As of now, we are using Spring Batch with Tomcat (exploded WAR) for the same. However, we are facing certain issues which I'm unable to solve using Spring Batch. I was thinking of other frameworks to use, but couldn't find any that solves all my problems.
It would be great to know if there exists a framework which solves the following problems:
Since Spring Batch runs inside one Tomcat container (1 java process), any small update in any job/step will result in restarting the Tomcat server. This results in hard-stopping of all running jobs, resulting in incomplete/stale data.
WHAT I WANT: Bundle all the jars and run each job as a separate process. The framework should store the PID and should be able to manage (stop/force-kill) the job on demand. This way, when we want to update a JAR, the existing process won't be hindered (however, we should be able to stop the existing process from UI), and no other job (running or not) will also be touched.
I have looked at hot-update of JARs in Tomcat, but I'm skeptical whether to use such a mechanism in production.
Sub-question: Will OSGI integrate with Spring Batch? If so, is it possible to run each job as a separate container with all JARs embedded in it?
Spring batch doesn't have a master-slave architecture.
WHAT I WANT: There should be a master, where the list of jobs are specified. There should be slave machines (workers), which are specified to master in a configuration file. There should exist a scheduler in the master, which when needed to start a job, should assign a slave a job (possibly load-balanced, but not necessary) and the slave should update the DB. The master should be able to send and receive data from the slaves (start/stop/kill any job, give me update of running jobs, etc.) so that it can be displayed on a UI.
This way, in case I have a high load, I should be able to just add machines into the cluster and modify the master configuration file and the load should get balanced right away.
Spring batch doesn't have an in-built alerting mechanism in case of job stall/failure.
WHAT I WANT: I should be able to set up alerts for jobs in case of failure. If necessary, a job should have a timeout where it should able to notify the user (via email probably) or should force stop the job when the job crosses a specified threshold.
Maybe vertx can do the trick.
Since Spring Batch runs inside one Tomcat container (1 java process), any small update in any job/step will result in restarting the Tomcat server. This results in hard-stopping of all running jobs, resulting in incomplete/stale data.
Vertx allows you to build microservices. Each vertx instance is able to communicate with other instances. If you stop one, the others can still work (if there are not dependant, eg if you stop master, slaves will fail)
Vert.x is not an application server.
There's no monolithic Vert.x instance into which you deploy applications.
You just run your apps wherever you want to.
Spring batch doesn't have a master-slave architecture
Since vertx is even driven, you can easily create a master slave architecture. For example handle the http request in an vertx instance and dispatch them between severals other instances depending on the nature of the request.
Spring batch doesn't have an in-built alerting mechanism in case of job stall/failure.
In vertx, you can set a timeout for each message and handle failure.
Sending with timeouts
When sending a message with a reply handler you can specify a timeout in the DeliveryOptions.
If a reply is not received within that time, the reply handler will be called with a failure.
The default timeout is 30 seconds.
Send Failures
Message sends can fail for other reasons, including:
There are no handlers available to send the message to
The recipient has explicitly failed the message using fail
In all cases the reply handler will be called with the specific failure.
EDIT There are other frameworks to do microservices in java. Dropwizard is one of them, but I can't talk much more about it.
The premise is this: For asynchronous job processing I have a homemade framework that:
Stores jobs in database
Has a simple java api to create more jobs and processors for them
Processing can be embedded in a web application or can run by itself in on different machines for scaling out
Web UI for monitoring the queue and canceling queue items
I would like to replace this with some ready made library because I would expect more robustness from those and I don't want to maintain this. I've been researching the issue and figured you could use JMS for something similar. But I would still have to build a simple java API, figure out a runtime where I would put the processing when I want to scale out and build a monitoring UI. I feel like the only thing I would benefit from JMS is that I would not have to do is the database stuff.
Is there something similar to this that is ready made?
UPDATE
Basically this is the setup I would want to do:
Web application runs in a Servlet container or Application Server
Web application uses a client api to create jobs
X amount of machines process those jobs
Monitor and manage jobs from an UI
You can use Quartz:
http://www.quartz-scheduler.org/
Check out Spring Batch.
Link to sprint batch website: http://projects.spring.io/spring-batch/
I am currently evaluating JMS and I don't get what I could use it for.
Currently, I believe this would be a Usecase: I want to create a SalesInvoice PDF and print it when an SalesOrder leaves the Warehouse, so during the Delivery transaction I could send a transactional print request which just begins when the SalesOrder transaction completes successfully.
Now I found out most JMS products are standalone server.
Why would a need a Standalone Server for Message Processing, vs. e.g. some simple inproc processing with Quartz scheduler?
How does it interact with my application?
Isn't it much too slow?
What are Usecases you already implemented successfully?
JMS is an amazingly useful system, but not for every purpose.
It's essentially a high-level framework for sending messages between nodes, with options for discovery, robustness, etc.
One useful use case is when you want a client and a server to talk to one another, but without the client actually having the server's address (E.g., you may have more than one server). The client only needs to know the broker and the queue/topic name, and the server can connect as well.
JMS also adds robustness. For instance, you can configure it so that if the server dies while the client sends messages or the other way around, you can still send messages from the client or poll messages from the server. If you ever tried implementing this directly with sockets - it's a nightmare.
The scenario you describe sounds like a classic J2EE problem, why are you not using a J2EE framework? JMS is often used inside J2EE for communications, but you got all the other benefits.
What ist Java Message Service (JMS) for
JMS is a messaging standard that allows Java EE applications to create, send, receive, and consume messages in a loosely coupled, reliable, and asynchronous way. I'd suggest to read the Java Message Service API Overview for more details.
Why would a need a Standalone Server for Message Processing, vs. e.g. some simple inproc processing with Quartz scheduler?
Sure, in your case, Quartz is an option. But what if the invoice system is a remote system? What if you don't want to wait for the answer? What if the remote system is down when you want to communicate with it? What if the network is not always available? This is where JMS comes in. JMS allows to send a message guaranteed to be delivered and to consume it in a transactional way (sending or consuming a message can be part of a global transaction).
How does it interact with my application?
JMS supports two communication modes: point-to-point and publish/subscribe (if this answers the question).
Isn't it much too slow?
The MOMs I've been working with were blazing fast.
What are Usecases you already implemented successfully?
Used in system such as a reservation application, a banking back-office (processing market data), or more simply to send emails.
See also
EJB Message-Driven Beans
Why would a need a Standalone Server
for Message Processing, vs. e.g. some
simple inproc processing with Quartz
scheduler?
The strength of JMS lies in the fact that you can have multiple producers and multiple consumers for the same queue, and the JMS broker manages the load.
If you have multiple producers but a single consumer, you can use other approaches as well, such as a quartz scheduler and a database table. But as soon as you have multiple consumer, the locking scheme become very hard to design; better go for already approved messaging solution. See these other answers from me for a few more details: Why choosing JMS for asynchronous solution ? and Producer/consumer system using database
The other points are just too vague to be answered.
I've used it on a number of projects. It can help with scalability, decoupling of services, high availability. Here's a description of how I used it on a project several years ago:
http://coders-log.blogspot.com/2008/12/favorite-projects-series-installment-2.html
The description explains what JMS brought to the table for this particular project, but other projects will use messaging systems for a variety of reasons.
Messaging is usually used to interconnect different systems and send requests/commands asynchronously. A common example is a bank client application requesting an approval for a transaction. The server is located in another bank's system. Both systems are connected in an Enterprise Service Bus. The request goes into the messaging bus, which instantly acknowledges the reception of the message. The client can go on with processing. Whenever the server system becomes available, the bus forwards the message to it. Of course there needs to be a second path, for the server to inform the client that the transaction executed successfully or failed. This again can be implemented with JMS.
Please note that the two systems need not to implement JMS. One can use JMS and the other one MSMQ. The bus will take care of the interconnection.
JMS is a message-oriented middleware.
Why would a need a Standalone Server for Message Processing, vs. e.g. some simple inproc processing with Quartz scheduler?
It depends on what other components you may have. I guess. But I don't know anything about Quartz
How does it interact with my application?
You send messages to the broker.
Isn't it much too slow?
Compare to what ?
What are Usecases you already implemented successfully?
I've used JMS to implement a SIP application server, to communicate between the various components.
From the Javadoc:
The Java Message Service (JMS) API provides a common way for Java programs to create, send, receive and read an enterprise messaging system's messages.
In other words, and contrary to every other answer here, JMS is nothing more than an API, which wraps access to third-party Message Brokers, via 'JMS Providers' implemented by the vendor. Those Message Brokers, such as IBM MQ and dozens of others, have the features of reliability, asynchronicity, etc. that have been mentioned in other answers. JMS itself provides exactly none of them. It is to Message Brokers what JDBC is to SQL databases, or JNDI is to LDAP servers (among other things).
I have found a very good explanation of JMS with an example.
That is a simple chat application with JMS queues are used to communicate messages between users and messages stay in the queue if the receiver is offline.
In this example implementation they have used
XSD to generate domain classes.
Eclipse EE as IDE.
JBoss as web/application server.
HTML/JavaScript/JQuery for UI.
Servlet as controller.
MySQL as DB.
The JBoss configuration step for queue is explained nicely
Its available at http://coder2design.com/messaging-service/
Even the downloadable code is also available there.