I have developed an application that monitors a mailbox and processes certain messages, which are identified by containing a specific keyword in the message subject. When such a message is received, it is deleted from the mailbox and triggers local processing. The code roughly looks like this:
public void process() {
// Create properties
Properties properties = new Properties();
properties.put("mail.imap.host", host);
properties.put("mail.imap.port", port);
properties.put("mail.imap.socketFactory.port", port);
properties.put("mail.imap.partialfetch", "false");
properties.put("mail.imap.fetchsize", 36700160);
// Open
Session session = Session.getInstance(properties);
Store store = session.getStore("imap");
store.connect(emailAddress, password);
Folder folder = store.getFolder("INBOX");
folder.open(Folder.READ_WRITE);
// Count relevant messages
int count = 0;
try {
// List messages
for (Message message : folder.getMessages()) {
String subject = message.getSubject();
if (subject.contains(keyword)) {
count++;
message.setFlag(Flag.DELETED, true);
// Do some local processing
}
}
} catch (MessagingException e) {
// Log and ignore
} finally {
// Expunge and close if a message was marked deleted
try {
if (count > 0) {
folder.close(true);
}
store.close();
} catch (MessagingException e) {
// Log and ignore
}
}
}
The problem is that some messages that should be processed are never processed. When activating the log on Jakarta mail (log com.sun.mail) I can see that the delete flag is being set for some messages and they are removed before they can be read by the method sketched above (which I see in the application log). As no other client is connected to the mailbox, I believe that my code might be the problem.
The problem only occurs irregularly when the application runs for several hours, so I am not able to provide a minimal example that can be used to reproduce the issue. My question is more of a general nature: has anyone also observed this behaviour when processing messages in a mailbox via IMAP with Jakarte Mail or with any other Email-API or protocol? Do you have any hints what the underlying issue might be? Do you see any issue with the pseudocode shown above?
Any help is very appreciated!
I found a solution which solved the problem:
I did not close the folder and the store object in all cases after a receiving cycle:
If messages were actually received the folder was always closed with folder.close(true).
Additionally I ensured now that in all cases (also exceptions) a method close() is called which contains:
if (folder != null && folder.isOpen()) {
folder.close(false);
}
if (store != null && store.isConnected()) {
store.close();
}
So far the issue did not occur again
Related
JavaMail Folder.search() returns multiple IMAP messages even if a MessageIDTerm is passed to the method. It seems to happen when the messages are subsequent forwards of an orginal message. JavaMail version is 1.4.4. The mail server is MS Exchange 2013. Users send emails with MS Outlook.
Here is the code:
MessageIDTerm messageIDTerm = new MessageIDTerm(uniqueMessageID);
Message[] messages = folder.search(messageIDTerm);
If the uniqueMessageID is a message ID of an email the has been forwarded, the messages array will contain the message with uniqueMessageID and all the subsequent forwarded messages.
Is this behaviour correct? Is there any way to get only the message with the messagedID passed to the search method?
Most likely it's a bug in Exchange. Turn on JavaMail Session debugging and it should provide enough information for you to report the bug to Microsoft.
Are the forwarded messages being sent as attachments to a new message? If so, Exchange may be searching for the header in the attachments as well as the main message, which would be wrong.
BTW, you might want to upgrade to the current 1.5.3 version of JavaMail.
To get only the Messages try this
public void loadFolder() {
if (store != null && store.isConnected()) {
try {
final Folder inbox = store.getFolder("foldername");
inbox.open(Folder.READ_ONLY);
final FlagTerm ft = new FlagTerm(new Flags(Flags.Flag.USER), false);
final Message[] messages = inbox.search(ft);
for (Message message : messages) {
try {
if (message.getContentType().contains("text")) {
final String text = (String) message.getContent();
System.out.println(text);
}
} catch (Exception e) {
LOG.error("", e);
}
}
}
With the Flag you can specify your search
I'm trying to download only subject of emails, because it should take less time (downloading ~10 emails with photos take about 10 min :/).
Code that I'm now using is:
try {
Store store = session.getStore("imaps");
store.connect(...);
Folder folder = store.getFolder(folderName);
folder.open(Folder.READ_ONLY);
message = folder.getMessages();
for (Message m : message) {
System.out.println(m.getSubject());
}
folder.close(false);
store.close();
} catch (MessagingException e) {
...
} catch (IOException e) {
...
}
it seems to me that you should look into prefetching the messages with:
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add("Subject");
folder.fetch(message, fp);
What you're doing will download the entire message envelope (but not the entire message), which includes the subject and the recipients. That's usually pretty cheap to download. If you really want only the subject because you're never going to look at the other information, you need to deal with the raw header using something like this:
String rawvalue = msg.getHeader("Subject", null);
if (rawvalue == null)
return null;
try {
return MimeUtility.decodeText(MimeUtility.unfold(rawvalue));
} catch (UnsupportedEncodingException ex) {
return rawvalue;
}
The folder.fetch call described in the other answer will allow you to prefetch all the Subject headers in one operation, instead of fetching each one as you process that message. You can also prefetch the entire envelope if you decide that's what you want; see the javadocs for details.
I am using the apache commons net library to get a file from FTP server.
I don't need to download the whole file but just to read the headers to determine the file size. The library I am using to do this is metadata extractor
The problem is that when I call client.completePendingCommand() it always returns false - however the date variable is printed correctly. I have asked the developer of metadata extractor and he doesn't know why its returning false. Anyone have an explanation? I'm not sure if it is ok just to ignore the false?
FTPClient client = new FTPHTTPClient(proxy settings);
InputStream stream = null;
try {
client.connect(FTPProperties.getInstance().getProperty("ftp.server"));
client.login(FTPProperties.getInstance().getProperty("ftp.username"), FTPProperties.getInstance().getProperty("ftp.password"));
client.enterLocalPassiveMode();
for (String path : paths) { //paths are the jpeg files to download
try {
stream = client.retrieveFileStream(p);
Metadata metadata = ImageMetadataReader.readMetadata(stream);
Directory directory = metadata.getDirectory(ExifSubIFDDirectory.class);
Date date = directory.getDate(ExifSubIFDDirectory.TAG_DATETIME_ORIGINAL);
System.out.println("DATE " + date);
} catch (IOException ex) {
Logger.getLogger(UploadImage.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if(stream != null) {
stream.close();
}
if (in != null) {
in.close();
}
if (!client.completePendingCommand()) {
Logger.getLogger("Error");
}
}
}
} catch (Exception ex) {
Logger.getLogger(UploadImage.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (client != null && client.isConnected()) {
client.disconnect();
}
}
I don't think you are doing anything wrong and I don't think there is anything wrong with metadata extractor. You may be better off checking the stream you are retrieving can be correctly processed rather than using completePendingCommand() as an indication of success. Metadata extractor may already do this for you by throwing an exception if there is a problem.
Explanation:
completePendingCommand() verifies the success of the entire transaction and success or failure relies on the FTPClients reply code being within the range of 200 <= replyCode < 300 (http://commons.apache.org/proper/commons-net/apidocs/src-html/org/apache/commons/net/ftp/FTPReply.html#line.133).
I had a similar problem and found that my FTPClient object had a reply code of 150 which indicates that according to the FTP server, the transaction has not completed. Reply code 150 is a positive preliminary reply but is not classified as a positive completion reply (https://www.rfc-editor.org/rfc/rfc959 page 37). My point is that the response was still positive and although I thought I have completed the transaction, the FTP server still thinks I need to do something. This may be an issue with org.apache.commons.net.ftp.FTPClient or the FTP server it is interacting with.
I'm trying to debug a web app hostde on weblogic 10r3 server. The App is receiving input from foreign IBM JMS Queue (classname: com.ibm.mq.jms.MQQueue) via Java Message Driven Beans.
I wrote a small Test App to connect to that queue and sending test messages. The problem is for now test message generates exceptions and somehow it gets put back on queue and is looping again and again. This generate lots of exceptions rendering log unreadable.
First I tried remove the poison message by constructing a consumer on my Test App, but the code blocks indefinitely on consumer.receive().
Then I tried to set JMSexpiration to some number instead of default 0, but in the end the message still used 0 as expiration.
All Ideas Welcome, Many Thanks
Code outline the JMS PRODUCER:
static String rawTradeUpload = "some long chunk of data"
Hashtable ht = new Hashtable();
ht.put(Context.INITIAL_CONTEXT_FACTORY,
weblogic.jndi.WLInitialContextFactory.class.getName());
//ht.put(Context.PROVIDER_URL, "t3://gprimeap1d.eur.nsroot.net:12016");
ht.put(Context.PROVIDER_URL, "t3://gprimeap1d.eur.nsroot.net:12001");
ht.put(Context.SECURITY_PRINCIPAL, "weblogic");
ht.put(Context.SECURITY_CREDENTIALS, "welcome5");
Connection con = null;
Session s = null;
try {
if(ctx == null)
ctx = new InitialContext(ht);
ConnectionFactory myConnFactory = null;
Queue myQueue = null;
myConnFactory = (ConnectionFactory) ctx
.lookup("SwiftConnectionFactory");
con = myConnFactory.createConnection();
s = con.createSession(false, Session.AUTO_ACKNOWLEDGE);
myQueue = (Queue) ctx
.lookup("IncomingSwiftFxQueue");
MessageProducer producer = s.createProducer(myQueue);
Message msg = s.createTextMessage(rawTradeUpload);
producer.send(msg);
s.close();
con.close();
} catch (NamingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
It is very simple to configure the backout queue and the redelivery settings on the IBM WebSphere MQ queue manager. Setting BOThreshold and BO Queue Name along with a local queue. Perhaps you might be able to convince them to make this setup, after all - you both benefit from this solution to work as good as possbile, right?
Otherwise, why don't you try to catch all exceptions and, if they occur - re-queue the failed message to an error queue on WebLogic. Seems to be a decent choice.
If you need to do receive, call it with a timeout instead.
Message msg = consumer.receive(1000L); // wait for a message for 1 sec, then continue.
Have you tried setting a re-delivery limit and defining an error queue destination so that after x attempts, the container takes care of moving the message to the error queue? This allows you to keep the primary queue free of the poison messages and a dedicated error queue to browse/debug issues.
I have an application that is used to archive emails using imap. Also in this application are many imap accounts that need to be archived.
In this moment from time to time the application connects to imap accounts and download only new emails. My issue is that every time when it connects to an imap account it verifies all emails from all folders and downloads only emails that aren't downloaded yet (I store Message-ID for all emails and download only emails that have an Message-ID that is not stored).
So I want to know if there is an alternative for this, because it takes some time to verify all emails (for 10-20K it takes 2-5 minutes).
I use JavaMail API to connect to imap accounts.
The javadoc helps:
IMAPFolder provides the methods:
getMessagesByUID(long start, long end) and
getUID(Message message)
With getUID() you can get the UID of the last message you already have downloaded. With getMessagesByUID you can define this last message you have downloaded as start-range and look with the method getUIDNext() to find the last message which would be the end of the range.
check only the headers and when you reach a known (the last known), bail out:
for instance (i feel extra nice today) and that's an except from real production code (some parts were cut, so it might not compile, state.processed is some set preferrably LinkedHashMap surrogate [keySet()] (and w/ some max boundary boolean removeEldestEntry())
try {
store = mailSession.getStore("imap");
try {
store.connect();
Folder folder = store.getFolder("INBOX");
folder.open(Folder.READ_ONLY);
int count = folder.getMessageCount();
for(int localProc=0, chunk=49;localProc<10 && count>0; count -=chunk+1){
Message messages[] = folder.getMessages(Math.max(count-chunk, 1), count);
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add("Message-ID");
//add more headers, if need be
folder.fetch(messages,fp);
for (int i=messages.length;--i>=0;) {
//can check abort request here
Message message = messages[i];
String msgId = getHeader(message,"Message-ID");
if (msgId!=null && !state.processed.add(msgId)){
if (++localProc>=10){
break;
}
continue;
}
///process here, catch exception, etc..
}
}
folder.close(false);
} catch (MessagingException e) {
logger.log(Level.SEVERE, "Mail messaging exception", e);
}
} catch (NoSuchProviderException e) {
logger.log(Level.SEVERE, "No mail provider", e);
}
if(store != null) {
try {
store.close();
} catch (MessagingException e) {}
}
Filter on the SEEN flag. This flag is intended for finding new messages. The one Caveat is that if your user is using multiple readers, then it may have been seen using another reader.
message-Id which comes as part of the header is always unique even if u set it manually .i have tested it with gamil and racksoace.