Using javamail i am reading all the messages from inbox folder and use to pass the message object to executor one by one,. Executor use to read email body using get content and use to categorize it and after categorizing it I use to move it to specific folder and use to delete it from inbox folder.
But in this process I am facing issue In get content method, it is returning me body of different mail. Seems like state sharing issue in multithreading environment in Java API.
Can someone suggest me what can be the possible issue.
Related
I have the most basic problem ever. The user wants to export some data which is around 20-70k records and can take from 20-40 seconds to execute and the file can be around 5-15MB.
Currently my code is as such:
User clicks a button which makes an API call to a Java Lambda
AWS Lambda Handler calls a method to get the data from DB and generate excel file using Apache POI
Set Response Headers and send the file as XLSX in the response body
I am now faced with two bottlenecks:
API Gateway times out after 29 seconds; if file takes longer to
generate it will not work and user get 504 in the browser
Response from lambda can only be 6MB, if file is bigger the user will
get 413/502 in the browser
What should be my approach to just download A GENERATED RUNTIME file (not pre-built in s3) using AWS?
If you want to keep it simple (no additional queues or async processing) this is what I'd recommend to overcome the two limitations you describe:
Use the new AWS Lambda Endpoints. Since that option doesn't use the AWS API Gateway, you shouldn't be restricted to the 29-sec timeout (not 100% sure about this).
Write the file to S3, then get a temporary presigned URL to the file and return a redirect (HTTP 302) to the client. This way you won't be restricted to the 6MB response size.
Here are the possible options for you.
Use Javascript skills to rescue. Accept the request from browser/client and immediately respond from server that your file preparation is in progress. Meanwhile continue preparing the file in the background (sperate job). Using java script, keep polling the status of file using separate request. Once the file is ready return it back.
Smarter front-end clients use web-sockets to solve such problems.
In case DB query is the culprit, cache the data on server side, if possible, for you.
When your script takes more than 30s to run on your server then you implement queues, you can get help from this tutorial on how to implement queues using SQS or any other service.
https://mikecroft.io/2018/04/09/use-aws-lambda-to-send-to-sqs.html
Once you implement queues your timeout issue will be solved because now you are fetching your big data records in the background thread on your server.
Once the excel file is ready in the background then you have to save it in your s3 bucket or hard disk on your server and create a downloadable link for your user.
Once the download link is created you will send that to your user via email. In this case, you should have your user email.
So the summary is Apply queue -> send a mail with the downloadable file.
Instead of some sophisticated solution (though that would be interesting).
Inventory. You will split the Excel in portions of say 10 k rows. Calculate the number of docs.
For every Excel generation called you have a reduced work load.
Whether e-mail, page with links, using a queue you decide.
The advantage is staying below e-mail limits, response time-outs, denial of service.
(In Excel one could also create a master document, but I have no experience.)
I am trying to use rest APIs provided by mule management console to retrieve server log files.
http://www.mulesoft.org/documentation/display/current/Servers
My intention is to use this List File API
http://localhost:8080/mmc-console-3.4.0/api/servers/{serverId}/files/{relativePathToFile}[?metadata=true]
provided and display the logs in UI.
1) What should be the return type of the method I make as the above API call is returning a file? Would it be 'File' ?
2) Since the size of the mule_ee.log file could be large, I want to send the entire file for first call and from next call I just want to send the few lines appended at the last so that UI will do the appending and show it in console. Is this feasible ? Is there a better approach to do this ?
As per documentation you will get the file itself. However there is no incremental mechanism.
For this matters you should probably use rsync or an advanced log distribution system.
I am developing a Java app that reads all the emails from GMAIL and save them run time on objects of self made Class that holds some of the attributes of the email. Now I want to delete an email object and when I do that I want it to be removed from the internet as well. In order to do so I will need some information to query the GMAIL account to delete that particular email I am referring to. Is there a possible way to do that? Maybe by saving the header and then searching for that? I couldn't find a good way to do it.
I assume you are accessing GMAIL with IMAP. Then you could store the Message, call Message.setFlag(Flag.DELETED) then expunge the Folder, or search the Folder yourself. Or maybe you can search the Google Gmail api and find a better way. For example you can use the unique message ID
UPDATE this is a google project that can help you: java-gmail-imap. There is also a companion project for OAuth authentication (needed for run the sample)
I'd recommend you to take a look on James.
The Apache James Project delivers a rich set of open source modules and libraries, written in Java, related to Internet mail communication which build into an advanced enterprise mail server.
It depends on the time span between when you access the message and when you want to delete it. If it's short, and the folder is still open, you can just set the DELETED flag and close the folder to expunge the message.
But I'm assuming that you need to locate the message some time later. In that case, you should learn about IMAP UIDs. See the UIDFolder interface in JavaMail and the IMAP spec for details. In essence, you need to keep track of the folder's UIDVALIDITY value and the message's UID value. When you come back to the folder, you can make sure it's the same folder by checking the UIDVALIDITY value and then look up the message based on its UID.
I have written a nice program in Java that connects to a gmail account and download atachments sent to it. Once an attachment has been downloaded, it is marked as read and is not downloaded ever again. This program will have to run in multiple instances with each program downloading unique attachments so that a single attachment is never downloaded twice. The problem is that at the moment if the attachment is of a decent size, one program is still downloading it, when another instance connects and also starts to download the attachment before it has been marked as read.
I have tried checking and setting various flags and checking whether the folder is open, nothing seems to work. Any solutions?
Update: Thank you for the quick answers, sadly IMAP is not an option due to other reasons.
Consider using IMAP instead - it is designed for client-server interaction.
From RFC1939 (Post Office Protocol - Version 3):
POP3 is not intended to provide
extensive manipulation operations of
mail on the server; normally, mail is
downloaded and then deleted. A more advanced (and complex) protocol, IMAP4, is discussed in RFC1730.
I don't think POP3 is made for multiple simultaneous access.
Ask yourself this: do i really need multiple processes accessing the same mailbox?
If you do, you'll have to find a way to have these processes communicate to each other.
Use a common database or server process to coordinate actions.
IMAP does have more options, but i'm not sure if you can "lock" a single mail to mark it as being processed.
As the others have mentioned, POP3 isn't really intended for this kind of scenario.
If you absolutely have to use POP3, I'd suggest downloading all the e-mail to an intermediate server which sorts the messages and makes them available for each of the other clients.
It sounds like you're just trying to distribute the processing of the e-mails. If that's the case, you can just have each client connect to your intermediate server to retrieve the next available message.
I'm not sure what your constraints are, but you may even want to consider receiving the attachments some other way besides e-mail. If people are uploading files, you could set up a web form that automatically sends each file to the next available instance of your application for processing.
If you need to stay with a POP3 connection, you could keep a local database of previously downloaded message ids. Then new instances could check against that before downloading again. The best solution is just to use IMAP, though, as IMAP is able to set the read/unread flags before downloading.
You could mark the mail as read before starting the download, and then start downloading it.
Just got a request from my boss for an application I'm working on. Basically we're getting an email address setup for an external client to submit excel files to.
What I need is a way to automatically pick up any email sent to this address, so I can take the attachment, process it and save it to a folder.
Any information of even where to start would be helpful.\
Note: We're using a lotus notes server to do this, but a generic way would be more helpful (If possible).
Email -> mailserver ->[something] -> file-on-disk.
File on disk is pretty easy to parse, use JavaMail.
The [something] could be:
listener for smtp connections (overkill)!
Pop3/imap client
Maildir/Mailbox
Edit: since I first wrote this answer, Wiser has moved and now claims to only be a unit testing tool, so take the answer below with a pinch of salt...
Svrist's answer is good, but if you want to avoid his middle step (the mailserver that writes the mail to disk for later pickup by the Java system) you can use Wiser.
Wiser lets you start an in-Java mailserver:
Wiser wiser = new Wiser();
wiser.setPort(2500);
wiser.start();
Then you can just poll it periodically for mail:
for (WiserMessage message : wiser.getMessages())
{
String envelopeSender = message.getEnvelopeSender();
String envelopeReceiver = message.getEnvelopeReceiver();
MimeMessage mess = message.getMimeMessage();
// mail processing goes here
}
Use a mail in database (your Domino administrator can set that up for you but it's in the help file as well).
In that database, you can create an agent that runs periodically to process all new documents. That agent will use the EmbeddedObjects property of the NotesRichTextItem class and the ExtractFile method of the NotesEmbeddedObject class to get a handle on the file attachment and extract it to the location you specify.
For example, this script goes through all the file attachments, object links, and embedded objects in the Body item of a document. Each time it finds a file attachment, it detaches the file to the SAMPLES directory on the C drive and removes the attachment from the document
Dim doc As NotesDocument
Dim rtitem As Variant
'...set value of doc...
Set rtitem = doc.GetFirstItem( "Body" )
If ( rtitem.Type = RICHTEXT ) Then
Forall o In rtitem.EmbeddedObjects
If ( o.Type = EMBED_ATTACHMENT ) Then
Call o.ExtractFile( "c:\samples\" & o.Source )
Call o.Remove
Call doc.Save( False, True )
End If
End Forall
End If
I've done quite a bit lately with Java agents on Domino servers. The Domino 8.5 server supports Java 6 and its embedded so it won't take someone with a bit of Domino development experience long to put together an agent that runs when new mail arrives. In LotusScript its even easier but that needs more specialised skills which you'd probably need to get a contractor in to provide.
The limitation your likely to encounter concerns the extracted file, you can easily place it on the Domino server's file structure but you may be limited by the OS security from placing it on a different server.
Lotus Notes/Domino stores mail in a Notes database. There are APIs available for getting documents (emails), reading field values (From, Subject), and detaching files.
APIs include
-LotusScript (VB variant, available within the Notes database)
-Java (from within or external to the database)
-C API (external)
-Same API available through COM server
You can create a "scheduled agent" within the database (using LotusScript or Java) that can locate documents created since it last ran, locate the attachments, and extract them. The agent will need to be signed with an ID that has the appropriate permissions on the server, including those required to write to the file system and initiate any other processes.
External to the database, you can use any API except LotusScript to log-in to the server/mail database, and follow a similar process, e.g. extracting the files locally on a client or separate server. C API and COM require a notes client install, but Java applications can be set up to run via CORBA/DIIOP without a full install.
Consult the Domino Designer help (or IBM's website for C API) for more information.
As to a "generic way" to do this, if you are accessing data in Notes and needing to extract attachments, I believe these APIs are your best option. If you envision porting the application to another mail system, consider decoupling the API routines via an "interface" so you only need to add a new implementation of that interface to support a new mail system.
You can access Notes Documents relatively easily using DIIOP, would be a lot easier than going down the C Api road...
Try POP3Client in the Net Commons package; it'll let your Java program check for new mail for a particular account at whatever interval you want (every few minutes? hourly?), and get/delete messages as desired.
SMTP/POP3 can be enabled on the Domino server. Worked with this before and gotten Squirrel Mail running with it. SMTP is a bit resource intensive, but well worth the effort because then you don't have to descend into LotusLand to get things working. Just write a small Java CLI program that will check a specific email box (POP3 or SMTP), and parse through the messages, pulling the attachments and placing them where needed.
Plenty of documentation and examples here:
http://java.sun.com/products/javamail/
The techniques that you develop taking this approach will be more widely applicable in your future career than anything Lotus/Domino specific.
No matter what you do, you'll need an understanding of the Lotus Notes data structures. The good news is that a fully automated solution can be built in Notes very easily.
Your best bet is to have it built within Notes, and it can be set up to run automatically whenever new mail is received. Gary's answer is dead on, but without any experience, it would probably be hard to figure out how to implement it yourself. On the other hand, it really shouldn't take any competent Notes programmer more than an hour or two to set it up.