I'm currently trying to write a simple Spring email verification microservice. I'm using Apache SMTPClient API to connect to an SMTP server and verify the email address existence (connect -> login -> setSender -> setRecipient -> logout -> disconnect). When connecting to most big email providers (gmail, yahoo, etc.) there are no issues, but for others there's a bizarre 5-20s delay when connecting. For example, with protonmail I sometimes wait for what looks like a fixed 5s delay, sometimes I connect normally after ~150ms. With some other, smaller services it's a fixed delay every time I try to connect.
Since I'm doing my best to reduce the time required for such verification, this is a pretty massive issue, given it generally takes <2s when no such issues are encountered.
I was trying to find a reason for this delay, and I encountered 2 potential explanations:
tarpitting - while the delay it causes fits the issue, I am sending a singular connection request, while the mechanism is supposed to prevent spam/bulk emails,
being blacklisted - that also could be the cause, but in general that information is put into a (negative) reply to one of the SMTP commands I send.
I would appreciate any suggestions on what could be causing the delay and how to avoid it (if it's even possible).
I made a couple of SMTP connections to mail.protonmail.ch, and it is definitely doing some things that are unusual. I think they're implementing a variation of tarpitting.
For instance, sometimes after the connecting, mail.protonmail.ch immediately responds with an SMTP banner (that looks very much like a 220 response):
220-mailin008.protonmail.ch ESMTP Postfix
then, after a seemingly random delay, an actual 220 response:
220 mailin008.protonmail.ch ESMTP Postfix
Other times, after a seemingly random delay, it responds with a 220 response without sending a banner first:
220 mailin025.protonmail.ch ESMTP Postfix
Many client SMTP programs that are used to send spam do not implement SMTP properly. A shoddy program might 'get fooled' by the unusual way that mail.protonmail.ch responds, and send EHLO before it should. Then, mail.protonmail.ch can drop the connection, on the assumption that the client is not a 'real' SMTP MTA. A real MTA, however, would wait for the 'real' 220 response, before sending EHLO.
In this case I was able to solve the problem by simply using sockets - if all you need to do is verify the existnace of an email by sending RCPT TO command, this is the best way to do it. I personally could not find any Java APIs that both avoided the problem I had and allowed for easy sending of SMTP commands.
Related
Well I am new to this and I don't know how to do it, so my senior fellows please help!!!!
There is a situation described below:
An HTTP client is sending a request (Request can be of any type, not concerned regarding the request type) that directly hits a Loadbalancer. The Loadbalancer then redirects the traffic, based on the load of the traffic, towards a "Gateway" system running in two V440 Server, GW logic is written in Java, that actually logically routs this request towards another two server nodes which actually process the request.
Now the scene is something like that: there are several parallel connections are established with this Gateway from several HTTP clients. One connection per client. It has been observed that, while making connections to this GW, in case of some clients the CPU utilization is going 98-99%.
Client is creating one connection with the GW on particular port. Opens a socket connection:
ServerSocket _ss = new ServerSocket(_port);
Socket s = _ss.accept();
and then GW waits for the input to come from the client.
Now my question is:
Why this kind of situation is happening, as it seems all fine
for rest of the clients and there connections.
Only few clients who are creating connections with the GW is making the situation?
Is there anyway we can track this client's IP so that we can understand if this
has been occurred by same clients every time?
Is there any resolution for this?
Since it is not happening for all the clients, we are certainly not going to find an immediate answer for this. However, this is what my limited research on the question yields
Firstly, Question 2
Configure your F5 to capture the client's IP. Since it is HTTP, there are multiple ways of tracking the requests. One is to
sniff the header X-FORWARDED-FOR which will give the client's IP
Address
Or try adding this rule in your logging engine
when CLIENT_ACCEPTED { log local0. "clientIP:[IP::client_addr] accessed" }
If you also need other data such as resources you can use one of the
other events such as HTTP_REQUEST:
when HTTP_REQUEST {
log local0. "clientIP:[IP::client_addr] accessed [HTTP::host][HTTP::uri]" }
Refer link for above here
Secondly, Question 1
For this you need to look at your available traffic statistic mechanisms. I read this, this and this. Enable the statistics, monitor them live, test, mock requests and analyze the output. I do not know of any other options other than this, right now.
Another option, if you can modify your Java program is to include some sort of performance logging mechanism for each request. But this means there is a lot of development and that I do not recommend at all.
Thirdly, Question 3
This is primarily opinion based. As far as I think if you figure out the problem, you can resolve it.
today i'm researching about the jmeter.
connect the gmail
login
send email
I passed 2 step and got error "java.net.UnknownHostException: mail.google.com" on step 3
Login :
Send Email :
I connected the link "mail.google.com" on the website.
Can somebody teach me what am i wrong ?
I would recommend using SMTP Sampler instead of simulating sending an email using web interface.
The relevant configuration would be:
Server: smtp.googlemail.com
Port: 587
Address from: your Gmail address
Address to: recipient address(es)
Check "Use Auth" and provide your full GMail username and password
Tick "Use StarTLS" radiobutton
Other fields are pretty much self-explanatory
References:
Set up Gmail with Outlook, Apple Mail, or other mail clients
Load Testing Your Email Server: How to Send and Receive E-mails with JMeter
I know this is not an answer you will like, but you should not be pointing any performance testing tool at a server you do not own, control or manage (or have permission from the people who do). In addition, Google's own end user agreement prohibits this behavior, supporting automation only with their published API set. License-wise and ethically, this is a problematic set of behavior for performance testing.
If you wish to experiment, then consider setting up a LAMP stack box inside of your own domain/control with an SMTP relay, pointing to mailbox destinations on that server. If you must test routing then setup two email servers with destinations on the opposite host (easy with two virtual machines).
Note, SMTP supports direct addressing, such as [name]#[host|IP address] so you can control directly any email routing to outside mailboxes accidentally and SPAMming individuals or organizations. This also avoids the problem of clogging up an SMTP relay with thousands of messages undeliverable due to lack of a valid address - this last condition will slow all email delivery for an organization and if not addressed can grind delivery to a halt before the first undeliverable messages start failing out after the delivery timeout is reached which is typically around 72 hours.
I'm creating a small utility which receives a lot of HTTP requests. It is written in java and uses embedded-jetty to handle requests via https.
I have a load-testing tool for it, but when it is being run for some time it starts to throw exceptions:
java.net.BindException: Address already in use: connect
(note, this is on sender's side, not in my project)
As I understand this means no more free sockets were found in system when another connect was called. Throughput is about 1000 requests per second, and failures start to appear somewhere after 20000 to 50000 requests.
However when I use the same load testing tool with another program (a kind of simple consumer, written in scala using netty by some colleague - it simply receives all requests and returns empty ok response) - there is no so problem with sockets (though typical speed is 1.5-2 times slower).
I wonder if this could be fixed by telling Jetty somehow to close connections immediately after response was sent. Anyway each new request is sent via new connection. I tried to play with Connector#setIdleTimeout - it seems to be 30000 by default but have not succeeded.
What can I do to fix this - or at least to research the matter deeper to find its cause (if I am wrong in my suggestions)?
UPD Thanks for suggestions, I think I am not allowed to post the source, but I get the idea that I should study client's code (this will make me busy for some time since it is written in scala).
I found that really there was a problem with client - it sends requests with Connection: Keep-Alive in header, though creates new HttpURLConnection for each request and calls disconnect() method after it.
To solve this trouble on the server-side it was sufficient to send Connection: close in response header, since I have no allowance to change testing utility.
I'm developing a small app for Android, which monitors the availability of servers and their services on my network.
So far I have implemented ping and HTTP(S) checks. I'd also like to monitor some mail servers. I know about libs like javamail-android, but I don't need to really send a mail. I'm looking for a simple solution (ideally without an additional lib) to connect to my MTAs and to check whether they respond correctly.
What is the best way to check the availability of mail servers on Android?
Open a socket to port 25 (smtp, no ssl) and see if you get a 220 prompt like this conversation (on a command shell):
$ telnet lilly 25
Trying lilly...
Connected to lilly.
Escape character is '^]'.
220 lilly ESMTP Postfix
noop
250 2.0.0 Ok
quit
221 2.0.0 Bye
Connection closed by foreign host.
You may also check how long the individual steps (connect until 220 , noop , quit) are taking and react on slow response times.
I have an app which is programmatically generating and sending out emails. The recipients list can get upwards of 1000. I had been looping through and sending out individual emails, but that was taking too long at about .5 sec each. The approach I'm looking into now is to remove customization in the message body and send out a single email with all addresses in the BCC. (Maybe other solutions are possible and I welcome them, but I'm mainly interested in delving into the complexities of this BCC solution.)
Is there a limit to the number of recipients allowed on a single email? Is this wholly dependent on my email client and/or SMTP server's configuration? Are there other limits outside the control of my domain? Furthermore, how is BCC handled? I assume that the BCC distribution needs to be broken down into separate mail messages at some point. Is the mail client (in my case javax.mail) responsible for this, or does the mail server do this?
I'm also interested in suggestions of how can I test my new email blaster program?
I don't think it will be a valid test by creating 1000 accounts at google or wherever (nor do I want to). I've heard there are some mail server optimizations geared toward multiple recipients at the same host. In my case, most will be distinct hosts.
Another way is to involve all the recipients to see if they recieved the email. I can do this, but I want to avoid spamming them, assuming I may need to test multiple times if things don't go right the first time.
Or do I just assume some limitation and send out batches of emails with some arbitrary number of recipients each, say 50 or 100?
BCC works inside your SMTP server; no recipients ever know other BCC'ed email addresses, so this is a limitation that depends entirely on your SMTP server.
You should check with your server administrator.
even more definitive, the RFC that covers SMTP (2821) makes no mention of recipient limitations outside of mail-server specific ones:
"If an SMTP server has an implementation limit on the number of RCPT
commands and this limit is exhausted, it MUST use a response code of
452 (but the client SHOULD also be prepared for a 552, as noted
above). If the server has a configured site-policy limitation on the
number of RCPT commands, it MAY instead use a 5XX response code.
This would be most appropriate if the policy limitation was intended
to apply if the total recipient count for a particular message body
were enforced even if that message body was sent in multiple mail
transactions."
http://www.ietf.org/rfc/rfc2821.txt
Thanks for your comments. As I understand it now, the outgoing SMTP server will be responsible for breaking up each of the messages. In construction of the new messages, the outgoing SMTP server will only send applicable RCPT TO commands for each BCC recipient. So in the case where all recipients are BCC, there will be only one RCPT TO command for each message.
That being the case, it seems like I really only need be concerned about our outgoing SMTP server configuration. No need to worry about the destination SMTP servers.
I got a suggestion that seems like a good way to test this. I could send my message to a number of recipients, each with a unique bogus child domain of our valid parent domain. When no MX record is found for the child, the parent will be used. The outgoing SMTP server will not be aware that the bogus domains don't exist, so this should avoid any SMTP optimatization for multiple recipients with common domains. We can probably also configure that these messages will all get routed to the same inbox.