We are developing a webapp that needs to send out emails written in Java/Groovy. Currently, we are persisting each email to a database before we call the Java Mail APIs to send the mail to our SMTP server.
I want to send out email asynchronously. I want to persist the email and then have another process pick up the email and send it (and send it only once). Ideally, this process is running outside of my webapp.
Are there any tools that do that?
Update: This solution needs to prevent duplicate emails and it needs to handle spikes in email. I was hoping someone already wrote an offline email processor. (I'd rather not implement this myself.)
The suggestions to use a cron job to read the database are quite workable.
Another good approach here is to use a Java Message Service (JMS) message queue. These are persistent (backed up by a database) and reliable. You can have one or more producer programs enqueue messages with the relevant data in them, and then one or more consumers process the messages and dequeue them. All of this is set up for very high reliability, and you gain the flexibility of asynchronously decoupling the operations, which means during email spikes the message queue can grow larger until the consumers catch up with the spike. Another benefit is that the email goes out as soon as a consumer gets to it instead of on a timer. Plus, if you require high availability, you can have multiple consumers in case one goes down.
Check out Apache's ActiveMQ for a good open source implementation of JMS.
If you're using Linux/Unix you could create a cron job to run every few minutes which calls a program to grab the email from the database and send it out. You could also have a field in the database to indicate whether the message has been sent. The downside of this approach is that there may be a delay of a few minutes from when your webapp persists the email and when the cron job is run.
Setup a cron job and use scripts to query the db and send out emails via sendmail.
On the off chance it's an Oracle DB, you can use the UTL_MAIL package to write PL/SQL to send the mail through your SMTP server. Then create a scheduled job to execute on your desired schedule.
http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14258/u_mail.htm
Since you are already using groovy, this might be an interesting tool to solve your problem
http://gaq.sourceforge.net/
You could use Quartz, a scheduling library (similar to cron), to schedule a recurring task that reads the DB and sends the emails. If you're using Grails, there's a Quartz plugin that makes working with Quartz a bit more Groovy.
Related
Many people suggest that the good way for organizing IPC (ImicroservicesC) is asynchronous communication via queries like Kafka and JMS.
But what if I need to pass large data files between services?
Suppose I have a Video Microservice and a Publisher Microservice. The first one receives videos from the user, verifies and sends them to Publisher for converting and publishing. It's oblivious video can be a very large file and it can overload messaging system (Kafka is not suitable for big messages at all). Of course, I can share one database for them and send video_id via Kafka, but it couples these services and its not a real microservices architecture anymore.
Do you have similar situations in practice? How do you handle it?
Thanks
There is an Enterprise Integration Pattern from the book by Hohpe/Wolfe called the Claim Check Pattern that addresses these concerns.
Essentially the big blob is removed from the message and stored somewhere that both sender and receiver can access, whether that be a common file share, FTP server, an Amazon S3 blob, whatever. It leaves a "claim check" behind: some sort of address that describes how to find the blob back.
The tiny message can then be transmitted over Kafka/JMS, or some other message queue system, most of which are fairly bad at dealing with large data blobs.
Of course, a very simple implementation is to leave the files on a file share and only refer to them by file path.
It's more complex when it's preferable to have the blob integrated with the rest of the message, requiring a true Claim Check implementation. This can be handled at an infrastructure level so the message sender and receiver don't need to know any of the details behind how the data is transmitted.
I know that you're in the Java landscape, but in NServiceBus (I work for Particular Software, the makers of NServiceBus) this pattern is implemented with the Data Bus feature in a message pipeline step. All the developer needs to do is identify what type of message properties apply to the data bus, and (in the default file share implementation) configure the location where files are stored. Developers are also free to provide their own data bus implementation.
One thing to keep in mind is that with the blobs disconnected from the messages, you have to provide for cleanup. If the messages are one-way, you could clean them up as soon as the message is successfully processed. With Kafka (not terribly familiar) there's a possibility to process messages from a stream multiple times, correct? If so you'd want to wait until it was no longer possible to process that message. Or, if the Publish/Subscribe pattern is use, you would not want to clean up the files until you were sure all subscribers had a chance to be processed. In order to accomplish that, you'd need to set an SLA (a timespan that each message must be processed within) on the message and clean up the blob storage after that timespan had elapsed.
In any case, lots of things to consider, which make it much more useful to implement at an infrastructure level rather than try to roll your own in each instance.
My setup consists of a Weblogic server that runs various REST services that accepts an HTTP request from a client(an android app), reads data from a MySql database and sends this data as a response to the client.
My requirement is that the client should be notified as soon as the database is updated. I have implemented a "polling" mechanism. The client constantly sends requests and checks for any changes made in the database. However this is not a very efficient mechanism.
Can you please suggest any efficient ways of achieving this?
Thanks in advance!
One of the way to use writing a CRON job for observation of Database change.
also, SO has already discussed this topic. refer below link,
Oracle database to send notification / Invoke a java method on reaching Time specified in a column
We have a Java (Spring) web application with Tomcat servlet container.
We have a something like blog.
But the blog must load its posts dynamically with Ajax.
The client's ajax script checks for new posts every second.
I.e. Ajax must ask the server for new posts every second and it will be very heavy for database.
But what if we have hundreds of thousands connects simultaneously?
I think that we must retrieve all posts with cron every second and after that save it somewhere. But where? The main idea is to unload the database.
Any ideas about architecture?
Thanks in advance!
There is other architecture for polling that could be more optimal, depending on the case:
Long polling
Long polling is a variation of the
traditional polling technique and
allows emulation of an information
push from a server to a client. With
long polling, the client requests
information from the server in a
similar way to a normal poll. However,
if the server does not have any
information available for the client,
instead of sending an empty response,
the server holds the request and waits
for some information to be available.
Once the information becomes available
(or after a suitable timeout), a
complete response is sent to the
client. The client will normally then
immediately re-request information
from the server, so that the server
will almost always have an available
waiting request that it can use to
deliver data in response to an event.
In a web/AJAX context, long polling is
also known as Comet programming.
Long Polling
Example of Implementations of this technology:
Push Server
You could also use the observer pattern to register the requests, and notify them when an update is done.
Hundreds of thousands of concurrent users all polling our site every second makes for a huge amount of traffic. If you truly expect this load you are going to have to design your platform accordingly, probably by clustering multiple web, application and DB servers.
Remember that with a database connection pool you don't need a DB connection for every user.
I'm not as familiar with Tomcat, but in WebSphere we can set up connection pools to prepare a certain number of connections.
Also, are you mainly worried about reads or the same number of writes?
Plus, you may also want to have the database "split" depending on region etc. This way there is no single heavy load across the entire database, but it can then be split and even load balanced.
There is also the "NoSQL" databases to look into as well. Maybe something to consider. Just ideas to help out.
How can we keep track of every message that gets into our Java Message Queue? We need to save the message for later reference. We already log it into an application log (log4j) but we need to query them later.
You can store them
in memory - in a collection or in an in-memory database
in a standalone database
You could create a database logging table for the messages, storing the message as is in a BLOB column, the timestamp that it was created / posted to the MQ and a simple counter as primary key. You can also add fields like message type etc if you want to create statistical reports on messages sent.
Cleanup of the tabe can be done simply by deleting all message older than the retention period by using the timestamp column.
I implemented such a solution in the past, we chose to store messages with all their characteristics in a database and developed a search, replay and cancel application on top of it. This is the Message Store pattern:
(source: eaipatterns.com)
We also used this application for the Dead Letter Channel.
(source: eaipatterns.com)
If you don't want to build a custom solution, have a look at the ReplayService for JMS from CodeStreet.
The best way to do this is to use whatever tracing facility your middleware provider offers. Or possibly, you could set up an intermediate listener whose only job was to log messages and forward on to your existing application.
In most cases, you will find that the middleware provider already has the ability to do this for you with no changes or awareness by your application.
I would change the queue to a topic, and then keep the original consumer that processes the messages, and add another consumer for auditing the messages to a database.
Some JMS providers cater for topic-to-queue-bridge definitions, the consumers then receive from their own dedicated queues, and don't have to read past messages that are left on the queue due to other consumers being inactive.
Alternatively, you could write a log4j appender, which writes your logged messages to a database.
I have a constantly-running Java program that needs to send an email whenever it encounters a problem. However it is possible that the mail server it uses could be down at the time it tries to send the email.
What is the best way to ensure that the email will be delivered when the mail server comes back up?
Queue up the requests. Have a separate thread which merely waits for something to enter the queue, then tries to email it. If it fails, it waits a few hours and tries again. Once it sends a message, it goes back to the queue to get the next message.
Put the email object into a stack or list when it fails to send, when the email server comes back up, pop each email out until it is empty.
You may want to save the email in a file, perhaps an xml file, so that should the application crash you won't lose this information.
This file is loaded when the application starts, and it keeps everything in memory, so that while there are pending emails then it keeps checking every 5 minutes or so, then, as it sends each email it will resave the xml file, so that should it crash after sending 3 emails out of 10 it won't resend those three when it starts up.
But, how you handle that is really going to depend on the specification for how to handle error conditions.
If you go from "forward everything to this SMTP server which is always there" to a situation where you need to handle all kinds of conditions normally handled by a full SMTP-server like retry later, retransmit if connection closed, use MX-hosts in their stated order and similar, you may want to consider simply having a SMTP-server inside your client (but one that does not accept incoming connections) since this moves all the dirty logic away from your applications.
I believe that the James email server - http://james.apache.org/ - is easily embeddable, but I have not actually tried.
The suggestion of using James is a good one but I've had some issues in the past of James being a bit flaky and needing to be restarted.
You could use something like Quartz to have a scheduler check for messages that need to be sent. If the message can't be sent (eg. smtp server isn't available), then that message is rescheduled to be sent at a later time. You could either have a task per message or have a persistent task that checks for messages and available mail server then sends the messages. The persistent task would give you email batching.
If you are in a Unix/Linux world, then consider the alternative of sending your alerts using syslog, and dealing with the generation of emails on that side. For example, nsyslogd has a module called ommail for generating emails natively.
IIRC, there are adapters for log4j and the like that can bridge between the Java and syslog worlds with a minimum of (zero ?) coding.
Apache James - http://james.apache.org/ will let you run your own mailserver as a proxy, not only that but is written in 100% java, so you can figure out what its doing,
and as an extra bonus James uses databases to queue the mail, so you can even inject mail directly into the queues by inserting into a database, then leave whole business of sending the mail up to James.