I have an application that is used to archive emails using imap. Also in this application are many imap accounts that need to be archived.
In this moment from time to time the application connects to imap accounts and download only new emails. My issue is that every time when it connects to an imap account it verifies all emails from all folders and downloads only emails that aren't downloaded yet (I store Message-ID for all emails and download only emails that have an Message-ID that is not stored).
So I want to know if there is an alternative for this, because it takes some time to verify all emails (for 10-20K it takes 2-5 minutes).
I use JavaMail API to connect to imap accounts.
The javadoc helps:
IMAPFolder provides the methods:
getMessagesByUID(long start, long end) and
getUID(Message message)
With getUID() you can get the UID of the last message you already have downloaded. With getMessagesByUID you can define this last message you have downloaded as start-range and look with the method getUIDNext() to find the last message which would be the end of the range.
check only the headers and when you reach a known (the last known), bail out:
for instance (i feel extra nice today) and that's an except from real production code (some parts were cut, so it might not compile, state.processed is some set preferrably LinkedHashMap surrogate [keySet()] (and w/ some max boundary boolean removeEldestEntry())
try {
store = mailSession.getStore("imap");
try {
store.connect();
Folder folder = store.getFolder("INBOX");
folder.open(Folder.READ_ONLY);
int count = folder.getMessageCount();
for(int localProc=0, chunk=49;localProc<10 && count>0; count -=chunk+1){
Message messages[] = folder.getMessages(Math.max(count-chunk, 1), count);
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add("Message-ID");
//add more headers, if need be
folder.fetch(messages,fp);
for (int i=messages.length;--i>=0;) {
//can check abort request here
Message message = messages[i];
String msgId = getHeader(message,"Message-ID");
if (msgId!=null && !state.processed.add(msgId)){
if (++localProc>=10){
break;
}
continue;
}
///process here, catch exception, etc..
}
}
folder.close(false);
} catch (MessagingException e) {
logger.log(Level.SEVERE, "Mail messaging exception", e);
}
} catch (NoSuchProviderException e) {
logger.log(Level.SEVERE, "No mail provider", e);
}
if(store != null) {
try {
store.close();
} catch (MessagingException e) {}
}
Filter on the SEEN flag. This flag is intended for finding new messages. The one Caveat is that if your user is using multiple readers, then it may have been seen using another reader.
message-Id which comes as part of the header is always unique even if u set it manually .i have tested it with gamil and racksoace.
Related
I have developed an application that monitors a mailbox and processes certain messages, which are identified by containing a specific keyword in the message subject. When such a message is received, it is deleted from the mailbox and triggers local processing. The code roughly looks like this:
public void process() {
// Create properties
Properties properties = new Properties();
properties.put("mail.imap.host", host);
properties.put("mail.imap.port", port);
properties.put("mail.imap.socketFactory.port", port);
properties.put("mail.imap.partialfetch", "false");
properties.put("mail.imap.fetchsize", 36700160);
// Open
Session session = Session.getInstance(properties);
Store store = session.getStore("imap");
store.connect(emailAddress, password);
Folder folder = store.getFolder("INBOX");
folder.open(Folder.READ_WRITE);
// Count relevant messages
int count = 0;
try {
// List messages
for (Message message : folder.getMessages()) {
String subject = message.getSubject();
if (subject.contains(keyword)) {
count++;
message.setFlag(Flag.DELETED, true);
// Do some local processing
}
}
} catch (MessagingException e) {
// Log and ignore
} finally {
// Expunge and close if a message was marked deleted
try {
if (count > 0) {
folder.close(true);
}
store.close();
} catch (MessagingException e) {
// Log and ignore
}
}
}
The problem is that some messages that should be processed are never processed. When activating the log on Jakarta mail (log com.sun.mail) I can see that the delete flag is being set for some messages and they are removed before they can be read by the method sketched above (which I see in the application log). As no other client is connected to the mailbox, I believe that my code might be the problem.
The problem only occurs irregularly when the application runs for several hours, so I am not able to provide a minimal example that can be used to reproduce the issue. My question is more of a general nature: has anyone also observed this behaviour when processing messages in a mailbox via IMAP with Jakarte Mail or with any other Email-API or protocol? Do you have any hints what the underlying issue might be? Do you see any issue with the pseudocode shown above?
Any help is very appreciated!
I found a solution which solved the problem:
I did not close the folder and the store object in all cases after a receiving cycle:
If messages were actually received the folder was always closed with folder.close(true).
Additionally I ensured now that in all cases (also exceptions) a method close() is called which contains:
if (folder != null && folder.isOpen()) {
folder.close(false);
}
if (store != null && store.isConnected()) {
store.close();
}
So far the issue did not occur again
This is my code:
if (Recipients_To != null) {
for (int i = 0; i < Recipients_To.length; i++) {
message.setRecipients(Message.RecipientType.TO, Recipients_To[i].toString());
Transport.send(message);
}
}
I have more than 500 Recipients list to send mail and this code will send personal mail to each recipients. But if i got exception in between this for loop i want to continue loop for remaining recipients. How can i do?
You want to use try catch blocks to do this, like so
for (int i = 0; i < Recipients_To.length; i++)
{
try {
message.setRecipients(Message.RecipientType.TO,Recipients_To[i].toString());
Transport.send(message);
}
catch (YourException e){
//Do some thing to handle the exception
}
}
This catches the potential issue and will still continue with the for loop once the exception has been handled.
You can catch the exception e.g.
try {
message.setRecipients(Message.RecipientType.TO, Recipients_To[i].toString());
Transport.send(message);
} catch (Exception e) {
// handle it or leave it be
}
Technically, it's simply a matter of catching the exception (see Murat K's answer). I would recommend, however, that since you're sending e-mail, that you do cease sending the rest when the first exception occurs, unless you are certain that it is an error you can safely ignore. A few examples of things that can go wrong:
Invalid credentials: this means that if you continue attempting to send, every subsequent attempt will also fail. Best case: no e-mail sent. Worst case: SMTP server blocks your access due to excessive login failures.
Malformed recipient address: no issue to continue trying the other addresses, but you need to do something with this error (remove recipient from list for future mailings)
Misconfigured mail server address: each iteration of your loop will try to connect to the mailserver, and fail. This will slow down the method tremendously (server timeouts) or spam your log (assuming you did something with the exception)
So please consider your course of action carefully when handling e-mail.
You can try something like this
if (Recipients_To != null) {
for (int i = 0; i < Recipients_To.length; i++) {
try {
message.setRecipients(Message.RecipientType.TO, Recipients_To[i].toString());
Transport.send(message);
} catch (Exception exp) {
//Log your exception here or do whatever you want to happen in case of exception
}
}
}
JavaMail Folder.search() returns multiple IMAP messages even if a MessageIDTerm is passed to the method. It seems to happen when the messages are subsequent forwards of an orginal message. JavaMail version is 1.4.4. The mail server is MS Exchange 2013. Users send emails with MS Outlook.
Here is the code:
MessageIDTerm messageIDTerm = new MessageIDTerm(uniqueMessageID);
Message[] messages = folder.search(messageIDTerm);
If the uniqueMessageID is a message ID of an email the has been forwarded, the messages array will contain the message with uniqueMessageID and all the subsequent forwarded messages.
Is this behaviour correct? Is there any way to get only the message with the messagedID passed to the search method?
Most likely it's a bug in Exchange. Turn on JavaMail Session debugging and it should provide enough information for you to report the bug to Microsoft.
Are the forwarded messages being sent as attachments to a new message? If so, Exchange may be searching for the header in the attachments as well as the main message, which would be wrong.
BTW, you might want to upgrade to the current 1.5.3 version of JavaMail.
To get only the Messages try this
public void loadFolder() {
if (store != null && store.isConnected()) {
try {
final Folder inbox = store.getFolder("foldername");
inbox.open(Folder.READ_ONLY);
final FlagTerm ft = new FlagTerm(new Flags(Flags.Flag.USER), false);
final Message[] messages = inbox.search(ft);
for (Message message : messages) {
try {
if (message.getContentType().contains("text")) {
final String text = (String) message.getContent();
System.out.println(text);
}
} catch (Exception e) {
LOG.error("", e);
}
}
}
With the Flag you can specify your search
I'm trying to download only subject of emails, because it should take less time (downloading ~10 emails with photos take about 10 min :/).
Code that I'm now using is:
try {
Store store = session.getStore("imaps");
store.connect(...);
Folder folder = store.getFolder(folderName);
folder.open(Folder.READ_ONLY);
message = folder.getMessages();
for (Message m : message) {
System.out.println(m.getSubject());
}
folder.close(false);
store.close();
} catch (MessagingException e) {
...
} catch (IOException e) {
...
}
it seems to me that you should look into prefetching the messages with:
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add("Subject");
folder.fetch(message, fp);
What you're doing will download the entire message envelope (but not the entire message), which includes the subject and the recipients. That's usually pretty cheap to download. If you really want only the subject because you're never going to look at the other information, you need to deal with the raw header using something like this:
String rawvalue = msg.getHeader("Subject", null);
if (rawvalue == null)
return null;
try {
return MimeUtility.decodeText(MimeUtility.unfold(rawvalue));
} catch (UnsupportedEncodingException ex) {
return rawvalue;
}
The folder.fetch call described in the other answer will allow you to prefetch all the Subject headers in one operation, instead of fetching each one as you process that message. You can also prefetch the entire envelope if you decide that's what you want; see the javadocs for details.
I am using the apache commons net library to get a file from FTP server.
I don't need to download the whole file but just to read the headers to determine the file size. The library I am using to do this is metadata extractor
The problem is that when I call client.completePendingCommand() it always returns false - however the date variable is printed correctly. I have asked the developer of metadata extractor and he doesn't know why its returning false. Anyone have an explanation? I'm not sure if it is ok just to ignore the false?
FTPClient client = new FTPHTTPClient(proxy settings);
InputStream stream = null;
try {
client.connect(FTPProperties.getInstance().getProperty("ftp.server"));
client.login(FTPProperties.getInstance().getProperty("ftp.username"), FTPProperties.getInstance().getProperty("ftp.password"));
client.enterLocalPassiveMode();
for (String path : paths) { //paths are the jpeg files to download
try {
stream = client.retrieveFileStream(p);
Metadata metadata = ImageMetadataReader.readMetadata(stream);
Directory directory = metadata.getDirectory(ExifSubIFDDirectory.class);
Date date = directory.getDate(ExifSubIFDDirectory.TAG_DATETIME_ORIGINAL);
System.out.println("DATE " + date);
} catch (IOException ex) {
Logger.getLogger(UploadImage.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if(stream != null) {
stream.close();
}
if (in != null) {
in.close();
}
if (!client.completePendingCommand()) {
Logger.getLogger("Error");
}
}
}
} catch (Exception ex) {
Logger.getLogger(UploadImage.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (client != null && client.isConnected()) {
client.disconnect();
}
}
I don't think you are doing anything wrong and I don't think there is anything wrong with metadata extractor. You may be better off checking the stream you are retrieving can be correctly processed rather than using completePendingCommand() as an indication of success. Metadata extractor may already do this for you by throwing an exception if there is a problem.
Explanation:
completePendingCommand() verifies the success of the entire transaction and success or failure relies on the FTPClients reply code being within the range of 200 <= replyCode < 300 (http://commons.apache.org/proper/commons-net/apidocs/src-html/org/apache/commons/net/ftp/FTPReply.html#line.133).
I had a similar problem and found that my FTPClient object had a reply code of 150 which indicates that according to the FTP server, the transaction has not completed. Reply code 150 is a positive preliminary reply but is not classified as a positive completion reply (https://www.rfc-editor.org/rfc/rfc959 page 37). My point is that the response was still positive and although I thought I have completed the transaction, the FTP server still thinks I need to do something. This may be an issue with org.apache.commons.net.ftp.FTPClient or the FTP server it is interacting with.