We Keep Coding

Different results between IMAP request

My team and I coded a mail component to be used by other teams in our application. This component is designed to allow them to download emails using custom search conditions. The one that is being used at the moment is a mail search between a range of dates from Gmail accounts using IMAP protocol. It was working properly until April 4th 2016 6:00AM GMT+1, only downloading email within the range. However, since 8:00AM GMT+1, the response from the server suddenly changed and it started returning the whole INBOX folder!
Repeating the initial tests, we discovered that we were receiving the whole INBOX folder instead of those within the range. After some research down to the IMAP request, it seemed that Gmail changed the amount of emails returned under certain conditions:
When the "start date" SearchTerm had a ComparisonTerm.GT term, or
When the "end date" SearchTerm had a ComparisonTerm.LE term
In both cases, Gmail's IMAP server returns all the emails in the folder. On the contrary, as an example, Yahoo's IMAP server just returns those emails which match the given condition.
We opened a ticket to them, asking if there was any change in their response, but they answered that they did not change anything. Of course, there is no way to prove it (we still don't have the time machine).
As we had a great impact with this unexpected behaviour, we decided to investigate further, and we created a simple mail client to download mail using different condition combinations from different IMAP servers.
Starting from this link's program, we added the functionality to check with other, but changing the library to meet our javax.mail version 1.5.5. Therefore, we avoid different implementation behaviours.
The code is attached below:
package gmail;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;
import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.Session;
import javax.mail.Store;
import javax.mail.search.AndTerm;
import javax.mail.search.ComparisonTerm;
import javax.mail.search.ReceivedDateTerm;
import javax.mail.search.SearchTerm;
public class JavaMailTest {
public static void main(String[] args) {
JavaMailTest testClient=new JavaMailTest();
//Yahoo
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GE,ComparisonTerm.LT);
//Gmail
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GE,ComparisonTerm.LT);
//Yahoo
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GE,ComparisonTerm.LE);
//Gmail
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GE,ComparisonTerm.LE);
//Yahoo
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GT,ComparisonTerm.LT);
//Gmail
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GT,ComparisonTerm.LT);
//Yahoo
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getYahooImapProps(),ComparisonTerm.GT,ComparisonTerm.LE);
//Gmail
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
testClient.performTest(testClient.getGmailImapProps(),ComparisonTerm.GT,ComparisonTerm.LE);
}
public int performTest(Properties props, int startTermCondition, int endTermCondition)
{
try {
Session session = Session.getDefaultInstance(props, null);
Message[] totalMessages = null;
Store store = session.getStore(props.getProperty("protocol"));
store.connect(props.getProperty("mail.imap.host"), props.getProperty("account"), props.getProperty("pass"));
Folder inbox = store.getFolder("inbox");
inbox.open(Folder.READ_ONLY);
SimpleDateFormat ft = new SimpleDateFormat ("yyyy-MM-dd");
Date startDate = ft.parse("2016-04-03");
Date endDate = ft.parse("2016-04-07");
System.out.println("Testing email account: <"+props.getProperty("account")+"> with startDate: <" + startDate + "> (ComparisonTerm."+
getConditionText(startTermCondition)+") and endDate: <" + endDate+"> (ComparisonTerm."+getConditionText(endTermCondition)+")");
Message[] messages = inbox.getMessages();
if (messages != null && messages.length > 0) {
SearchTerm startDateTearm =
new ReceivedDateTerm(startTermCondition, startDate);
SearchTerm endDateTerm =
new ReceivedDateTerm(endTermCondition, endDate);
SearchTerm andTerm = new AndTerm(startDateTearm, endDateTerm);
totalMessages = inbox.search(andTerm);
if(totalMessages.length > 0){
System.out.println("Found "+totalMessages.length+" matching messages (Total messages in folder: "+messages.length+")");
}
/*for (int i = 0; i < totalMessages.length; i++) {
System.out.println(totalMessages[i].getReceivedDate() + " --> Mail Subject:- " + totalMessages[i].getSubject());
}*/
//First message
int currentMessage=0;
System.out.println("First Message: "+totalMessages[currentMessage].getReceivedDate() + " --> Mail Subject: " + totalMessages[currentMessage].getSubject().substring(0, 5));
//Last message
currentMessage=totalMessages.length-1;
System.out.println("Last Message: "+totalMessages[currentMessage].getReceivedDate() + " --> Mail Subject: " + totalMessages[currentMessage].getSubject().substring(0, 5));
}
inbox.close(true);
store.close();
return totalMessages.length;
} catch (Exception e) {
e.printStackTrace();
return -1;
}
}
public static String getConditionText(int condition)
{
String returnValue="";
switch (condition) {
case ComparisonTerm.EQ:
returnValue="EQ";
break;
case ComparisonTerm.GE:
returnValue="GE";
break;
case ComparisonTerm.GT:
returnValue="GT";
break;
case ComparisonTerm.LE:
returnValue="LE";
break;
case ComparisonTerm.LT:
returnValue="LT";
break;
case ComparisonTerm.NE:
returnValue="NE";
break;
default:
returnValue="Error";
break;
}
return returnValue;
}
public Properties getYahooImapProps()
{
Properties props = new Properties();
//Yahoo
String imapServer="imap.mail.yahoo.com";
String imapPort="993";
String account="********#yahoo.es";
String pass="*********";
props.setProperty("mail.imap.host", imapServer);
props.setProperty("mail.imap.port", imapPort);
props.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
props.setProperty("mail.imap.auth", "true");
props.setProperty("account", account);
props.setProperty("pass", pass);
props.setProperty("protocol", "imaps");
return props;
}
public Properties getGmailImapProps()
{
Properties props = new Properties();
//Gmail
String imapServer="imap.gmail.com";
String imapPort="143";
String account="******#gmail.com";
String pass="********";
props.setProperty("mail.imap.host", imapServer);
props.setProperty("mail.imap.port", imapPort);
/*props.setProperty("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
props.setProperty("mail.imap.auth", "true");*/
props.setProperty("account", account);
props.setProperty("pass", pass);
props.setProperty("protocol", "imap");
return props;
}
}
To make the program output clearer, I added the request right before the first method output:
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********#yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********#gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 92 matching messages (Total messages in folder: 20138)
First Message: Sun Apr 03 04:08:47 CEST 2016 --> Mail Subject: Tu es
Last Message: Wed Apr 06 22:12:45 CEST 2016 --> Mail Subject: ¿Quie
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********#yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********#gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GE) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********#yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 BEFORE 7-Apr-2016 ALL
Testing email account: <*********#gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LT)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********#yahoo.es> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 5 matching messages (Total messages in folder: 78)
First Message: Mon Apr 04 23:14:22 CEST 2016 --> Mail Subject: Aviso
Last Message: Tue Apr 05 00:48:00 CEST 2016 --> Mail Subject: Alter
//SEARCH NOT ON 3-Apr-2016 SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
Testing email account: <*********#gmail.com> with startDate: <Sun Apr 03 00:00:00 CEST 2016> (ComparisonTerm.GT) and endDate: <Thu Apr 07 00:00:00 CEST 2016> (ComparisonTerm.LE)
Found 20138 matching messages (Total messages in folder: 20138)
First Message: Sat Sep 04 13:23:35 CEST 2004 --> Mail Subject: Gmail
Last Message: Tue Apr 19 10:09:06 CEST 2016 --> Mail Subject: El re
From this result we can say the following:
Yahoo returns the proper amount of emails, considering the given Java conditions.
Javamail's request is probably too ambiguous.
#Gmail: given the bunch of conditions in some requests... why you just decide to return the whole folder?
Why is there a different response for the same request?
Who is returning mails right?
Aren't protocols designed to standarise things instead of making it even worse?
We also tried with an Outlook account, with the same result as Gmail.

It sure looks like a bug in Gmail to me. I was able to reproduce it myself. I'm really surprised if Outlook.com has the same bug.
Why do you think the request is ambiguous? The IMAP spec is pretty clear.
A few notes...
Are you sure you're using JavaMail 1.5.5? The SEARCH commands you're showing reflect a JavaMail bug from before 1.5.3.
You should change Session.getDefaultInstance to Session.getInstance.
You don't need the socket factory properties; just set "mail.imap.ssl" to "true" instead.

#BillShannon
I think #popilla20k sees the request ambiguous because when he makes a request with a "GT" AND "LE" searchTerm (andTerm), it is decomposed, for example, as:
//SEARCH SINCE 3-Apr-2016 OR BEFORE 7-Apr-2016 ON 7-Apr-2016 ALL
I must say I've seen a change in Google's behaviour on April 6. Days before, I retrieved all mails that satisfied the condition with "GT" AND "LE" searchTerm. After April 6, I see the same behaviour showed by #popilla20k (using javamail 1.5.5)
Even so, I think the clause OR BEFORE invalids the first one. Does it should be AND clause, isn't? Anyway, days before April 6 the same requests were answered differently than nowadays.
Thanks.

FROM address incorrect

The FROM: address in my emails sent from java don't appear correctly. This is not running in an application server but from a main() call:
// Get system properties
final Properties properties = System.getProperties();
// Setup mail server
properties.setProperty("mail.smtp.host", "localhost");
// Get the default Session object.
final Session session = Session.getDefaultInstance(properties);
// Create a default MimeMessage object.
final MimeMessage message = new MimeMessage(session);
// Set From: header field of the header.
message.setFrom(new InternetAddress("support#mydomain.com"));
// Set To: header field of the header.
message.setRecipients(javax.mail.Message.RecipientType.TO, "you#you.com");
// Set Subject: header field
message.setSubject("Your pants are on fire!");
message.setSentDate(new Date());
// Now set the actual message
message.setText("Take me to the bridge. ow!");
// Send message
Transport.send(message);
Here's the raw received email. Two things to note: the linux username under which the java process runs is mungo. The linux servername/hostname is servername. The domain is santamaria.com. These have been changed from their original to protect the guilty.
My question is: why isn't Return-Path support#mydomain.com and how can I make it so? Secondarily, how can I add a friendly name? e.g. Roger Earl <support#mydomain.com>
Delivered-To: you#you.com
Received: by 10.70.125.201 with SMTP id ms9csp51721pdb;
Sat, 22 Nov 2014 07:38:22 -0800 (PST)
X-Received: by 10.140.84.71 with SMTP id k65mr15089869qgd.76.1416670702208;
Sat, 22 Nov 2014 07:38:22 -0800 (PST)
Return-Path: <mungo#servername.santamaria.com>
Received: from servername.santamaria.com (servername.santamaria.com. [201.74.27.72])
by mx.google.com with ESMTPS id z90si10522625qgd.46.2014.11.22.07.38.21
for <you#you.com>
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Sat, 22 Nov 2014 07:38:22 -0800 (PST)
Received-SPF: none (google.com: mungo#servername.santamaria.com does not designate permitted sender hosts) client-ip=201.74.27.72;
Authentication-Results: mx.google.com;
spf=none (google.com: mungo#servername.santamaria.com does not designate permitted sender hosts) smtp.mail=mungo#servername.santamaria.com
Received: from servername.santamaria.com (localhost [127.0.0.1])
by servername.santamaria.com (8.14.4/8.14.4) with ESMTP id sAMFcLkq012340
for <you#you.com>; Sat, 22 Nov 2014 15:38:21 GMT
From: RogerEarl <mungo#servername.santamaria.com>
Date: Sat, 22 Nov 2014 15:38:21 +0000 (UTC)
To: you#you.com
Message-ID: <1542856295.1.1416670701712.JavaMail.mungo#servername.santamaria.com>
Subject: Your pants are on fire!
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Take me to the bridge. ow!
UPDATE
The working code looks like this:
properties.setProperty("mail.smtp.from", "support#mydomain.com");
message.setFrom(new InternetAddress("support#mydomain.com", "My Friendly Support Name");

The InternetAddress class includes a constructor that allows you to set a "personal name" along with the email address; read the javadocs.
The Return-Path header is set by the receiving mail server, probably based on the "envelope from" address. Set the mail.smtp.from property to the address you want to use.

Google app engine mail sends without error, but cannot be found in inbox

I am using google app engine's mail to send email. Bellow is my code
Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
Message msg = new MimeMessage(session);
try {
msg.setFrom(new InternetAddress(from));
msg.addRecipient(Message.RecipientType.TO, new InternetAddress(to));
msg.setSubject(subject);
msg.setText(message);
Transport.send(msg);
} catch (AddressException e) {
throw new InvalidMailInputException("Invalid email.");
} catch (MessagingException e) {
throw new InvalidMailInputException("Unable to send eamil at this time.");
}
Log:
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: MailService.send
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: From: myaddress#gmail.com
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: To: myaddress#gmail.com
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: Reply-to: myaddress#gmail.com
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: Subject: Appoint has been cancelled successfully (5)
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: Body:
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: Content-type: text/plain
Apr 13, 2013 5:26:29 PM com.google.appengine.api.mail.dev.LocalMailService log
INFO: Data length: 147
In the log message I have not recived any errors. However, I am not receiving email as well.
Appriceate your help.

It looks like you're using the local development server. In such case no mail is actually sent.
From:
https://developers.google.com/appengine/docs/java/mail/overview#Development_Server
When an application running in the development server calls the Mail service to send an email message, the message is printed to the log. The Java development server does not send the email message.

Uploading large files with user metadata to Amazon S3 using java sdk

I'm trying to use the Java API to upload large files, but I need to also add user metadata. If I just use
TransferManager tm = new TransferManager(new BasicAWSCredentials(accessKey, secretKey));
Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, file);
upload.waitForCompletion();
Then everything works fine, but if I use:
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());
metadata.addUserMetadata("My key", "My value");
FileInputStream input = new FileInputStream(file);
Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, input, metadata);
Then it doesn't work, and I get the following output in the console:
Jul 5, 2011 4:33:15 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: POST https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploads: null, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, x-amz-meta-length: 312612077, )
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 2A5B3538795CE730
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 1, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 5E5AF291FBDBDD36
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 2, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: I/O exception (java.io.IOException) caught when processing request: Read error
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: Retrying request
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
WARNING: Unable to execute HTTP request: null
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: DELETE https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:01 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 204, AWS Request ID: 0EFC3F8D0FA6097E
Any help greatly appreciated!

You may also need to set content type and, possibly, checksum as it can't be determined from input stream. This has some information for similar situation http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#putObject(com.amazonaws.services.s3.model.PutObjectRequest)

If you are trying to upload more than 5 GB file with updating metadata then amazon not allowing to do so. Because amazon s3 not providing copy operation for more then 5Gb file.
Amazon S3 recently started to support copy operation wtih multipart operation so for that i can not say you something yet.
https://forums.aws.amazon.com/thread.jspa?messageID=256605&#256605
Thanks

You can use following code for your file upload:
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());
metadata.addUserMetadata("My key", "My value");
FileInputStream input = new FileInputStream(file);
PutObjectRequest putObjectRequest = new PutObjectRequest(AmazonS3Manager.bucketName, imageKey, input, metadata);
s3client.putObject(putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead));
Thanks

HTTP Builder/Groovy - lost 302 (redirect) handling?

I am reading here
http://groovy.codehaus.org/modules/http-builder/doc/handlers.html
"In cases where a response sends a redirect status code, this is handled internally by Apache HttpClient, which by default will simply follow the redirect by re-sending the request to the new URL. You do not need to do anything special in order to follow 302 responses."
This seems to work fine when I simply use the get() or post() methods without a closure.
However, when I use a closure, I seem to lose 302 handling. Is there some way I can handle this myself? Thank you
p.s. Here is my log output showing it is a 302 response
[java] FINER: resp.statusLine: "HTTP/1.1 302 Found"
Here is the relevant code:
// Copyright (C) 2010 Misha Koshelev. All Rights Reserved.
package com.mksoft.fbbday.main
import groovyx.net.http.ContentType
import java.util.logging.Level
import java.util.logging.Logger
class HTTPBuilder {
def dataDirectory
HTTPBuilder(dataDirectory) {
this.dataDirectory=dataDirectory
}
// Main logic
def logger=Logger.getLogger(this.class.name)
def closure={resp,reader->
logger.finer("resp.statusLine: \"${resp.statusLine}\"")
if (logger.isLoggable(Level.FINEST)) {
def respHeadersString='Headers:';
resp.headers.each() { header->respHeadersString+="\n\t${header.name}=\"${header.value}\"" }
logger.finest(respHeadersString)
}
def text=reader.text
def lastHtml=new File("${dataDirectory}${File.separator}last.html")
if (lastHtml.exists()) {
lastHtml.delete()
}
lastHtml<<text
new XmlSlurper(new org.cyberneko.html.parsers.SAXParser()).parseText(text)
}
def processArgs(args) {
if (logger.isLoggable(Level.FINER)) {
def argsString='Args:';
args.each() { arg->argsString+="\n\t${arg.key}=\"${arg.value}\"" }
logger.finer(argsString)
}
args.contentType=groovyx.net.http.ContentType.TEXT
args
}
// HTTPBuilder methods
def httpBuilder=new groovyx.net.http.HTTPBuilder ()
def get(args) {
httpBuilder.get(processArgs(args),closure)
}
def post(args) {
args.contentType=groovyx.net.http.ContentType.TEXT
httpBuilder.post(processArgs(args),closure)
}
}
Here is a specific tester:
#!/usr/bin/env groovy
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.Method
import static groovyx.net.http.ContentType.URLENC
import java.util.logging.ConsoleHandler
import java.util.logging.Level
import java.util.logging.Logger
// MUST ENTER VALID FACEBOOK EMAIL AND PASSWORD BELOW !!!
def email=''
def pass=''
// Remove default loggers
def logger=Logger.getLogger('')
def handlers=logger.handlers
handlers.each() { handler->logger.removeHandler(handler) }
// Log ALL to Console
logger.setLevel Level.ALL
def consoleHandler=new ConsoleHandler()
consoleHandler.setLevel Level.ALL
logger.addHandler(consoleHandler)
// Facebook - need to get main page to capture cookies
def http = new HTTPBuilder()
http.get(uri:'http://www.facebook.com')
// Login
def html=http.post(uri:'https://login.facebook.com/login.php?login_attempt=1',body:[email:email,pass:pass])
assert html==null
// Why null?
html=http.post(uri:'https://login.facebook.com/login.php?login_attempt=1',body:[email:email,pass:pass]) { resp,reader->
assert resp.statusLine.statusCode==302
// Shouldn't we be redirected???
// http://groovy.codehaus.org/modules/http-builder/doc/handlers.html
// "In cases where a response sends a redirect status code, this is handled internally by Apache HttpClient, which by default will simply follow the redirect by re-sending the request to the new URL. You do not need to do anything special in order to follow 302 responses. "
}
Here are relevant logs:
FINE: Receiving response: HTTP/1.1 302 Found
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << HTTP/1.1 302 Found
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Expires: Sat, 01 Jan 2000 00:00:00 GMT
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Location: http://www.facebook.com/home.php?
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << P3P: CP="DSP LAW"
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Pragma: no-cache
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: datr=1275687438-9ff6ae60a89d444d0fd9917abf56e085d370277a6e9ed50c1ba79; expires=Sun, 03-Jun-2012 21:37:24 GMT; path=/; domain=.facebook.com
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: lxe=koshelev%40post.harvard.edu; expires=Tue, 28-Sep-2010 15:24:04 GMT; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: lxr=deleted; expires=Thu, 04-Jun-2009 21:37:23 GMT; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Set-Cookie: pk=183883c0a9afab1608e95d59164cc7dd; path=/; domain=.facebook.com; httponly
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Content-Type: text/html; charset=utf-8
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << X-Cnection: close
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Date: Fri, 04 Jun 2010 21:37:24 GMT
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.DefaultClientConnection receiveResponseHeader
FINE: << Content-Length: 0
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: datr][value: 1275687438-9ff6ae60a89d444d0fd9917abf56e085d370277a6e9ed50c1ba79][domain: .facebook.com][path: /][expiry: Sun Jun 03 16:37:24 CDT 2012]".
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: lxe][value: koshelev%40post.harvard.edu][domain: .facebook.com][path: /][expiry: Tue Sep 28 10:24:04 CDT 2010]".
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: lxr][value: deleted][domain: .facebook.com][path: /][expiry: Thu Jun 04 16:37:23 CDT 2009]".
Jun 4, 2010 4:37:22 PM org.apache.http.client.protocol.ResponseProcessCookies processCookies
FINE: Cookie accepted: "[version: 0][name: pk][value: 183883c0a9afab1608e95d59164cc7dd][domain: .facebook.com][path: /][expiry: null]".
Jun 4, 2010 4:37:22 PM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Connection can be kept alive indefinitely
Jun 4, 2010 4:37:22 PM groovyx.net.http.HTTPBuilder doRequest
FINE: Response code: 302; found handler: post302$_run_closure2#7023d08b
Jun 4, 2010 4:37:22 PM groovyx.net.http.HTTPBuilder doRequest
FINEST: response handler result: null
Jun 4, 2010 4:37:22 PM org.apache.http.impl.conn.SingleClientConnManager releaseConnection
FINE: Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter#605b28c9
You can see there is clearly a location argument.
Thank you
Misha

I've had the same problem with HTTPBuilder until I realized that the HTTP/1.1 spec states:
Redirection 3xx
[..]
This class of status code indicates
that further action needs to be
taken by the user agent in order to
fulfill the request. The action
required MAY be carried out by the
user agent without interaction with
the user if and only if the method
used in the second request is GET
or HEAD.
302 Found
[..]
If the 302 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
Essentially this means that the request following a POST and 302 redirect won't work automatically and will require user intervention if the HTTP/1.1 spec is followed by the letter. Not all Http clients follow this practice, in fact most browsers do not. However the Apache Http Client (which is the underlying Http client for HttpBuilder) is spec compliant. There is an issue in the Apache Http Client bugtracker that contains more information and a possible solution for the problem.

void test_myPage_shouldRedirectToLogin() {
def baseURI = "http://servername"
def httpBuilder = new HTTPBuilder(baseURI)
// Make sure that HttpClient doesn't perform a redirect
def dontHandleRedirectStrategy = [
getRedirect : { request, response, context -> null},
isRedirected : { request, response, context -> false}
]
httpBuilder.client.setRedirectStrategy(dontHandleRedirectStrategy as RedirectStrategy)
// Execute a GET request and expect a redirect
httpBuilder.request(Method.GET, ContentType.TEXT) {
req ->
uri.path = '/webapp/de/de/myPage'
response.success = { response, reader ->
assertThat response.statusLine.statusCode, is(302)
assertThat response.headers['Location'].value, startsWith("${baseURI}/webapp/login")
}
response.failure = { response, reader ->
fail("Expected redirect but received ${response.statusLine} \n ${reader}")
}
}
}
302 status coming because, after action on any link redirected url not follow by HttpBuilder so we need to add "RedirectStrategy" explicitly.

What other headers do you see when you process the 302 response? If you were to turn on http client logging you'd expect to see HttpClient process the 302 response and automatically request URL in the Location header. What do you see when you process that URL? Does it work for any URL?
Try http://www.sun.com (it redirects to Oracle now.) I'm just wondering if the server you're working with is doing something wonky like sending a 302 with no Location header.