Downloading sent mails from Yahoo, Gmail and Hotmail - java

I am doing an Android application and I want to get the last 25 sent mails from a certain email account.
For Gmail I might use http://g4j.sourceforge.net/ and there is Mail Web Service API for Yahoo at http://developer.yahoo.com/mail/.
But I couldn't find something to do it with Hotmail.
Do you know if it is possible?
Also I am worried of having so many dependencies. I don't know if I should do something like https://sourceforge.net/projects/mrpostman/ and do web scraping.

You can download (or maybe upload) emails in various ways when using different email providers. My experience with the following providers is:
Yahoo:
POP3: Only available for Plus users (paid service). No get new messages, no folder access, no sent mail; just fetch all inbox (or all email UIDs). UPDATE: Yahoo provides free POP access and forwarding for Yahoo Asia users.
Mail Web Service API: Only listing email headers for Free users but complete access, including fetch mail from sent folder, for Plus users (paid service again). Of course, you are paid a commission by Yahoo if you can encourage (force) users to buy the Plus service if you are not sued before that by Yahoo lawyers because it is stated in the Web Service documentation that: "You may not use the Yahoo! Mail Web Service API to display the user's Yahoo! account information in a third party email client".
Web Scraping: It seems the only available solution for Free users of Yahoo but be aware of the compatibility problems that may arise when Yahoo changes its web pages. Also make sure to delay link accesses because Yahoo has web scraping detection mechanism on its servers.
GMail:
IMAPv4: Available for all users. Make sure to use this protocol for accessing almost everything in GMail. It is implemented completely; you can access all incoming and sent mails and even send an email by saving it in the sent folder. You can use JavaMail or any other IMAP client library in Java such as Ristretto API to do this. Make sure to know the JavaMail limitations before starting to use it for any protocol. It has many limitations (and minor bugs) in all protocols (SMTP/POP3/IMAP).
POP3: Available for all users of GMail but not recommended because of POP3 inherent limitations (no folder, no get new messages).
Hotmail:
POP3: Available for all users but again POP3 inherent limitations (no folder, no sent mail, no get new messages) in addition to Hotmail limitation called 15-minutes-delay-necessary for POP3 access.
Web Scraping: It seems the only solution for accessing sent mail but again be aware of the compatibility problems that may arise when Microsoft changes Hotmail web pages and web scraping detection software which may exist.
General IMAPv4 Provider:
In general, not all IMAP providers support sent folder because it is not a standard IMAP folder but most of them do this. Take a look at the provider's Help or FAQ for this option.
Genral POP3 Provider:
Do not expect POP3 to do this because POP3 does its best not to crash both the client and server when fetching 2 new emails from inbox ;-)
Meanwhile, do not forget that Web Scraping has legal issues and is forbidden in most web sites.

In our company's webapp, we use JavaMail to send mail through gmail account (very easy to use and powerful API). On JavaMail third-party product page I found project JDAVMail. It provides access method for WebDAV based protocol services. Maybe it'll be useful for you.
And, for Android: javamail-android

Why don't you just use either the built-in email facilities or standard access methods over IMAP? I guess using a separate lib for every provider won't work well in practice.

Google has its Mail application as open source.
https://android.googlesource.com/platform/packages/apps/Email
You might be able to utilize some of the source code yourself.

Related

Rest API security for mobile apps

I'm in the process of designing a mobile application that will need to connect to a server-side process for its business logic and data transactions. I'm writing my server-side code in Java using Spring Boot and I intend to create a Rest API in order for the mobile app to connect to the server.
I'm just doing some research at the moment for the best way to secure the connection between mobile app and server. What I'd like to do is allow the user on the mobile app to log in once they open the app and for them to use the app for as long as they like and for their access to time out after a period of inactivity.
Can anyone recommend any very simple reading on this? I've looked at OAuth2 but that appears to be for logging into web services using another account (like Google, GitHub, etc).
Would it be acceptable to login over https (SSL) by passing username and password to a rest endpoint and returning some sort of token (a GUID?). Then have the client (mobile app) pass that GUID with each subsequent call so the server can verify the call? Is it better to just do everything over SSL in this scenario?
I have done a fair bit of research but I don't seem to be able to find anything that quite matches what I'm trying to do.
Hope someone can help
Thanks
OAUTH2 IS NOT ONLY FOR WEB
Can anyone recommend any very simple reading on this?
I've looked at OAuth2 but that appears to be for logging into web services using another account (like Google, GitHub, etc).
No OAUTH2 is not only for web apps is also for mobile apps and you read this article for a more in depth explanation, but I will leave you with the article introduction:
Like single-page apps, mobile apps also cannot maintain the confidentiality of a client secret. Because of this, mobile apps must also use an OAuth flow that does not require a client secret. The current best practice is to use the Authorization Flow along with launching an external browser, in order to ensure the native app cannot modify the browser window or inspect the contents. If the service supports PKCE, then that adds a layer of security to the mobile and native app flow.
The linked article is very brief, you will need to follow the next chapters to get the full picture.
DO NOT ROLL YOUR OWN AUTHENTICATION / AUTHORIZATION SOLUTION
Would it be acceptable to login over https (SSL) by passing username and password to a rest endpoint and returning some sort of token (a GUID?). Then have the client (mobile app) pass that GUID with each subsequent call so the server can verify the call?
While you can do it I strongly advise you to use an already established OAUTH2 or OPENID connect solution, because they are developed and maintained by experts in the field and battle tested by millions of web and mobile apps using them. This enables to identify ans fix security issues much more quickly that anyone could do in their own in-house solution.
OAuth2
OAuth 2.0 is the industry-standard protocol for authorization. OAuth 2.0 supersedes the work done on the original OAuth protocol created in 2006. OAuth 2.0 focuses on client developer simplicity while providing specific authorization flows for web applications, desktop applications, mobile phones, and living room devices. This specification and its extensions are being developed within the IETF OAuth Working Group.
OpenID Connect
OpenID Connect 1.0 is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.
OpenID Connect performs many of the same tasks as OpenID 2.0, but does so in a way that is API-friendly, and usable by native and mobile applications. OpenID Connect defines optional mechanisms for robust signing and encryption. Whereas integration of OAuth 1.0a and OpenID 2.0 required an extension, in OpenID Connect, OAuth 2.0 capabilities are integrated with the protocol itself.
So for your authentication/authorization needs I would recommend you to go with an OpenID connect solution, that leverages OAuth2 under the hood.
SSL IS ALWAYS NECESSARY?
Is it better to just do everything over SSL in this scenario?
SSL must be always used for everything, http MUST not be used at all in any situation, because once you allow an http request you are vulnerable to a man in the middle attack and I strongly recommend you to read this article from a well know security researcher, Troy Hunt, to see how even a static website must use https and he goes to a great extent to explain why and names very important attack vectors that can harm an application not using https, like WiFi hot-spots hijacking, DNS Hijeacking, Router Exploits, China great cannon, and others.
IMPROVE SSL WITH CERTIFICATE PINNING
Communicating using https is the way to go for any kind of application but developers must be aware that an attacker in control of device where the application is installed can spy https traffic by doing a man in the middle attack with a custom certificate installed in the device the mobiel app is installed, enabling this way for him to understand how the mobile app communicates with the API server in order to mount automated attacks to abuse from same API.
Certificate Pinning
Pinning is the process of associating a host with their expected X509 certificate or public key. Once a certificate or public key is known or seen for a host, the certificate or public key is associated or 'pinned' to the host. If more than one certificate or public key is acceptable, then the program holds a pinset (taking from Jon Larimer and Kenny Root Google I/O talk). In this case, the advertised identity must match one of the elements in the pinset.
You can read this article, with code sample to see how easy is to implement certificate pinning, how it can be difficult to maintain in the operational side, and with a video to see how an attacker can bypass pass certificate pinning in the client side by using Xposed framework.
xPosed
Xposed is a framework for modules that can change the behavior of the system and apps without touching any APKs. That's great because it means that modules can work for different versions and even ROMs without any changes (as long as the original code was not changed too much). It's also easy to undo.
EDIT:
Nowadays, you can use the Mobile Certificate Pinning Generator to help you with implementing certificate pinning in your mobile app:
That will give you a ready to use pinning configuration for Android and iOS:
RESEARCHING FOR A SOLUTION
Before I point you out to a possible solution I would like to make clear the distinction between 2 concepts that developers frequently are not aware off or take as being the same thing...
The Difference Between WHO and WHAT is Accessing the API Server
The WHO is the user of the mobile app that you can authenticate,authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
Now you need a way to identify WHAT is calling your API server and here things become more tricky than most developers may think. The WHAT is the thing making the request to the API server, is it really your genuine mobile app or is a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
Well to identify the WHAT developers tend to resort to an API key that usually they hard-code in the code of their mobile app and some go the extra mile and compute it at run-time in the mobile app, thus becomes a dynamic secret in opposition to the former approach that is a static secret embedded in the code.
Some Mobile API Security Techniques
I have done a fair bit of research but I don't seem to be able to find anything that quite matches what I'm trying to do.
You can start by read this series of articles about Mobile API Security techniques to understand how Https, Certificate Pinning, APi Keys, HMAC, OAuth2 and other techniques can be used to protect the communication channel between your mobile app and the API serve, and how they can be bypassed.
To solve the problem of WHAT is accessing your mobile app you need to use one or all the solutions mentioned in the series of articles about Mobile API Security Techniques that I mentioned above and accepted that they can only make unauthorized access to your API server harder to bypass but not impossible.
A better solution can be employed by using a Mobile App Attestation solution that will enable the API server to know is receiving only requests from a genuine mobile app.
A POSSIBLE BETTER SOLUTION
The use of a Mobile App Attestation solution will enable the API server to know WHAT is sending the requests, thus allowing to respond only to requests from a genuine mobile app while rejecting all other requests from unsafe sources.
The role of a Mobile App Attestation service is to guarantee at run-time that your mobile app was not tampered or is not running in a rooted device by running a SDK in the background that will communicate with a service running in the cloud to attest the integrity of the mobile app and device is running on.
On successful attestation of the mobile app integrity a short time lived JWT token is issued and signed with a secret that only the API server and the Mobile App Attestation service in the cloud are aware. In the case of failure on the mobile app attestation the JWT token is signed with a secret that the API server does not know.
Now the App must sent with every API call the JWT token in the headers of the request. This will allow the API server to only serve requests when it can verify the signature and expiration time in the JWT token and refuse them when it fails the verification.
Once the secret used by the Mobile App Attestation service is not known by the mobile app, is not possible to reverse engineer it at run-time even when the App is tampered, running in a rooted device or communicating over a connection that is being the target of a Man in the Middle Attack.
This is a positive model where false positives do not occur, thus the API server is able to deny requests with the confidence of not blocking legit users of your mobile app.
The Mobile App Attestation service already exists as a SAAS solution at Approov(I work here) that provides SDKs for several platforms, including iOS, Android, React Native and others. The integration will also need a small check in the API server code to verify the JWT token issued by the cloud service. This check is necessary for the API server to be able to decide what requests to serve and what ones to deny.
CONCLUSION
Properly securing a mobile app and the API server is a task composed of several layers of defense that you must put together in order to protect it.
How many layers to use will depend on the data your are protecting, the value it have for the business, the damage it can cause when leaked in a data breach and how much you may be penalized by law enforcement, like GDPR in Europe.
What I usually do is crafting a JSON web token (https://jwt.io/), and handle the sessions on my own.
JWT is really nice, since you only need to define a secret key on the server side. As long as your clients are able to pass the string you crafted (inside the headers for example), and as long as nobody gets to retrieve your secret key, you are sure that every data you push when creating the token was generated by you. (Don't hesitate to use the strongest encryption algorithm)
For a secure connection use HTTPS at level TLS 1.2 level. Then pin the server certificate in the app, that will prevent MITM attacks.
It is safe to pass the user name and password. You can return a time-limited token for further authentication is needed/desired.
With HTTPS everything but the address portion of the URL is encrypted. But be careful with the query string, it may end up in the server logs.
Thanks again for these replies. I've been implementing my service to run under https by using server.ssl.key-store parameters and it looks like it's working okay. I have used keytool.exe to create a trust store and I run my SpringBoot app (with Tomcat embedded) using that trust store. I can open a browser to my REST endpoint (using https this time, not http), the browser asks for authentication and when I enter my user details, it matches them against my db user and allows me to see the response from the server.
One question though, what's the point of having a trust store on the server side (Java) if I can just access the REST endpoint using any old browser and just enter my user name and password? Eventually, this REST endpoint won't be accessed via a browser, it'll be accessed using a mobile app programmatically, so I assume I'll be logging on using that with username and password over https. I thought I'd need to have a certificate of some sort on the client side in order to communicate or does it not work like this?
Thanks again

Send mail (Java vs Php)

I have developed an app with a login. To help users I would like to send a mail reminding their credentials.. 2 solutions but 2 problems:
1) If I send the mail from the mobile phone (using my gmail account and mail.jar library) Google always reports a possible violation of my account or a suspect access, blocking it. I read several forum page about remove this feature but some people say that Gmail is built to be used by a single person
2) If I send a mail from a php page (using the function mail($mailto,$subject,$message,$header) several mail server put automatically this email into the trash folder.
Any suggestion? How to solve this issue?
Thank you in advance
You could send mail in java (mail.jar or mailServices of Spring Framework). I suggest you to use a mail provider service to send mail. It's better because there many things to care with when you send email ( email does not exist, delay of receiving the email, the current status of email sent, received or read...)

Be able to send an email to an email address, and have that appear on a web page

I want to be able to send an email to an email address, and then have it appear on a web page. Is this possible?
My guess is you'd have to write your own email server, something which I am not capable of doing. So I am assuming this won't be possible for me.
But if there is some way it can be done, that would be great. I generally program in Java and use Tomcat as my app server.
No, you wouldn't have to run your own SMTP server. You'd simply need to be able to retrieve mail from a POP3 or IMAP server, using something like the mail client API found in javax.mail.
It would be up to you to decide how much control you'd give to users. For example, who specifies the IMAP settings? Who decides which messages to fetch and display? Maybe that's all pre-configured. Maybe you write full-featured, web-based email client that can send messages as well as retrieve. This is all determined through the design of your web application.
You don't need to write your own mail server. You can use an ordinary (external) mail server and poll its inbox via POP3 or IMAP from your software. This introduces a short delay up to the full poll interval, but that might or might not be acceptable for you.
I can't give you a good tip for a email client lib to use for that, though.
Here is a simple example of sending email trough Google's SMTP server.

Java GroupWise Mail Client

Is anyone aware of a way to retrieve and send mails by talking directly to the GroupWise server.
I want to be able to retrieve mail using Java if possible.
I wrote a Outlook Plug-In that retrieved mail out of GroupWise via the Groupwise Outloop Plug-in (If that makes sense).
Then dropped the message in a directory where I retrieved it with a Java App.
The problem is that I cannot add more than one GroupWise account in Outlook and need to.
I had a look at this question, but would like to know if there is a Java API
that will allow me to retrieve/send mail from the GroupWise server in a JavaMail like manner.
Thanks.
GroupWise allows mails to be retrieved using IMAP and POP3 which are standard supported by JavaMail. (And outlook too for that matter)
Sending mail should also be possible using smtp.
If it is for plain email I prefer to use these basic proptocols like smtp, pop3 and imap because they work almost everywhere, anytime and on any platform. They are less feature rich than the proprietary protocols, but that point is often not very relevant since many of these features are only meaningful for a subset of the mailclients out there.

How can I send a confirmation email to users of my application?

I have designed a chat application where different users can create the account, but I do not know how to send a confirmation email to the users. Since I collect their e-mail address in the registration form, how can I send mail to those addresses in Java?
First of all, you need a SMTP server. It's required to be able to send emails. You can make use of the SMTP server associated with your own existing email account, such as the one from your ISP or public mailboxes like Gmail, Yahoo, etc. You can find SMTP connection details at their documentation. You usually just need to know the hostname and the port number. The login details are just the same as from your email account.
You're however restricted to using your own address in the From field of the email and usually also in the amount of emails you're allowed to send at certain intervals. If you'd like to get around this, then you need to install your own SMTP server, for example Apache James, which is Java based, or Microsoft Exchange and so on.
Then, to send an email using Java code, you would need the JavaMail API or the more convenienced Apache Commons Email.
This looks like a good site for you: http://www.javacommerce.com/displaypage.jsp?name=javamail.sql&id=18274
Google 'send mail java'
The easiest way of doing this really depends on the environment that your JVM is running in.
If you're running in a standard Linux/UNIX environment and don't want to faff about with extra libraries, then one way is just to "manually" call sendmail (e.g. via ProcessBuilder). As with executing commands generally, you just need to be slightly careful that you don't just pass user input as parameters without screening them.

Categories