We Keep Coding

When HTTP redirect occurs, apache HttpClient returns header from first response and not the last response

This is my java program
private static final String SAMPLE_URL = "https://www.dropbox.com/s/<something>/test_out4.mp4";
public static void main(String[] args) throws IOException, URISyntaxException {
HttpClient client = HttpClientBuilder.create().build();
HttpHead request = new HttpHead(new URI(SAMPLE_URL));
HttpResponse response = client.execute(request);
System.out.println(response.getStatusLine());
for (Header header : response.getAllHeaders()) {
System.out.println(header.getName() + ": " + header.getValue());
}
}
See a snippet of the output of the java program. The status line says it is HTTP/1.1 200 OK. However the header fields that are printed doesn't match what i get when i run curl manually. It seems to take the header values from the first response and not from the last response. Even more important Content Length field which is present in the last response is not set the response structure.
HTTP/1.1 200 OK <<< Status is 200
Server: nginx
Date: Thu, 20 Jun 2019 02:22:58 GMT
Content-Type: text/html; charset=utf-8 << Content type is char
When i run curl the output is correct. Is there any setting in HttpClient to return the most recent headers?
curl -I https://www.dropbox.com/s/<something>/test_out4.mp4
HTTP/1.1 301 Moved Permanently <<< Status 301
Server: nginx
Date: Thu, 20 Jun 2019 02:20:50 GMT
Content-Type: text/html; charset=utf-8 <<< Content type text
Connection: keep-alive
....
HTTP/1.1 302 Found << second redirect
Server: nginx
Date: Thu, 20 Jun 2019 02:36:03 GMT
Content-Type: text/html; charset=utf-8
....
HTTP/1.1 200 OK <<< Status finally 200
Server: nginx
Date: Thu, 20 Jun 2019 02:36:04 GMT
Content-Type: video/mp4 << content type correct
Content-Length: 92894175 << length correct
Connection: keep-alive

AsyncHttpClient does not use provided proxy data

I've written a query tool which at it's heart uses this block to gather info from an external url:
AsyncHttpClientConfig proxiedCF = new DefaultAsyncHttpClientConfig.Builder().setUserAgent(pickUserAgent()).build();
AsyncHttpClient asyncHttpClient = new DefaultAsyncHttpClient(proxiedCF);
Future<Response> f = asyncHttpClient.prepareGet(url).setProxyServer(new ProxyServer.Builder(pickProxyServer(), 80)).execute();
It works fine. However it works even when an invalid proxy is provided, which is a bit suspicious and I feel my elaborate proxy configuration is not used at all.
I stumbled upon this by having pickProxyServer() return a String "1.1.1.1", which is obviously not a valid web proxy.
I use SLF4J for logging and it looks pretty normal:
20:31:47.454 [AsyncHttpClient-7-1] DEBUG o.a.n.channel.NettyConnectListener - Using new Channel '[id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]' for 'GET' to '[[url removed by me]]'
20:31:47.586 [AsyncHttpClient-7-1] DEBUG o.a.netty.handler.HttpHandler -
Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET [[url removed by me]] HTTP/1.1
Host: [[url removed by me]]
Accept: */*
User-Agent: burning_dandelion
Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Fri, 13 Oct 2017 18:31:49 GMT
Expires: Fri, 13 Oct 2017 18:31:49 GMT
Cache-Control: private, max-age=3600
Content-Type: text/xml; charset=ISO-8859-1
P3P: CP="This is not a P3P policy!"
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-10-13-18; expires=Fri, 20-Oct-2017 18:31:49 GMT; path=/; domain=[[url removed by me]]
Set-Cookie: NID=114=qiVBv02cmXYHh2RfLQbhBfESWIoaGlf3d2jlSbAdQ8yWPDsCpOeK9aYbvfq0HWsER68W1oE53jiriM_fivTc1bJi1F2sfCi0wMptKI-9U3ueVKITtFvYYZx2T0rJf1kQ; expires=Sat, 14-Apr-2018 18:31:49 GMT; path=/; domain=[[url removed by me]]; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
20:31:47.587 [AsyncHttpClient-7-1] DEBUG o.a.netty.channel.ChannelManager - Adding key: ProxyPartitionKey(proxyHost=1.1.1.1, proxyPort=80, secured=false, targetHostBaseUrl=[[url removed by me]]:80 for channel [id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]
Can someone point me towards my error? Obviously, I want an I/O exception or any kind of notice when an invalid proxy is called upon.

Ok it's the middle of the night but I found something. When the port is anything but :80, the proxy is correctly used. This does the trick for me.

IBM Social Business Toolkit - no support for IBM Connections 5.5?

I'm asking myself whether IBM Connections 5.5 is not supported by IBM SBT.
The version I use is "1.1.11.20151208-1200".
My test procedure is the following:
public static void testCreateCommunity() throws ClientServicesException {
String g = Variables.aCommunityService.createCommunity( "TEST", "TESTDESCRIPTION", "public" );
if (g.isEmpty()) {
System.out.println( "x0[Failed] Creating Community has failed." );
}
else
{
System.out.println( "Creating Community successfully done." );
}
}
This code does perfectly work in Connections 5.0 CR 3 , but does not work
in IBM Connections 5.5.
I always get:
com.ibm.sbt.services.client.ClientServicesException: Request to url https://blabla.com/communities/service/atom/communities/my returned an error response 400:Bad Request HTTP/1.1 400 Bad Request [Date: Tue, 26 Jan 2016 10:20:02 GMT, X-Frame-Options: SAMEORIGIN, Strict-Transport-Security: max-age=max-age=31536000;includeSubDomains, X-XSS-Protection: 1;mode=block, X-Permitted-Cross-Domain-Policies: master-only, X-Powered-By: Servlet/3.0, Expires: Thu, 1 Jan 1970 00:00:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, X-LConn-Auth: false, X-UA-Compatible: IE=edge, Last-Modified: Tue, 26 Jan 2016 10:20:02 GMT, Set-Cookie: LtpaToken2=kx9gO87/cDI8zHT1v8iwsFCP6WAbAH7FusrA8VU7jOC78KqkTEghj1XsNPRLMDT4tmIEI+diSer+++TZw1gSiC79jveQoTerr53Ggdf/zVwOVACyzA9kcpzPsaWn2+u83SkHC4s3ZCAoDGe1eq6Mb9sF2lnrn2GDrbsSzzvCPdo+pSzx4AG+0OEOa1rPX2gVF5mCfYXeqtNxUeFMc/Eibzt0zszHX5RDXZz5pcU+D1LW98B8rnar3YJjEgp8QdLT1IvhRYIo1zQQs920c9kU0tgw+CccC97fD/SRucqsHWqh2aHhs2hlTaEzMKo21o/5lD+Qwkn3QwWYFtKZntmQGLlAlJvPBQNgR2+38E4Y8uEyFy8jaBbZE0tE6MdK9zSY9Pz6zGPZaMHSV6msS+veXncynS5mcFg7jpLdsHqbQRw0Hb9w3Pe7XChaQ+yrbwTiF+mooWrCoSOYCYkA6fEVVKUbCDF0imKFWVZXOdCaszl/Ank9DFbiBSXfNGWoiXk1pJHSnoJs8C4+jBqjhbcYebpbLLTmjtS2DytMW15r97bpDekGMqFywms539c4c9QKMmjPli6L7fgYAGVsopqlMmp8AwhhuH9tXaqc6mOtbspMAKGZTn8GmvAFIVTxqfumyYLCUQvsCOgRIhdC0WlXxx/Zq+usQcvHUXwQarFhycU=; Path=/; Domain=.blabla.com, Set-Cookie: JSESSIONID=0000H65mMCw0ijcsS5e19kYaAyB:1a9lvgg03; Path=/; HttpOnly, Vary: Accept-Encoding,User-Agent, Connection: close, Transfer-Encoding: chunked, Content-Type: text/xml;charset=UTF-8, Content-Language: de-DE]
Does anybody know whether IBM SBT generally supports Connections 5.5?

We have similar issues in Communities (and Activities) use cases. Most of the functionality works fine, but you stumbled upon an issue that we also encountered.
For now we are using the REST API's to work around this.
I know we could contribute to the open source project, but that will take some time.
So I would say it generally supports 5.5, but in this specific case ...

After running into the same issue I have found a way to add the header before the createCommunity() service call. In your case it would look like this:
Variables.aCommunityService.addDefaultHeader("Content-Type","application/atom+xml")
//Then create community
https://github.com/OpenNTF/SocialSDK/issues/1772#issuecomment-239517941
HTH

Google OAuth2 PlayGround returns "Unauthorized Client"

clientId = xxxxxx
clientSecret = xxxxxxxx
applicationHost = xxxxxxxxx
My authorization code request:
OAuthClientRequest oAuthClientRequest = OAuthClientRequest
.authorizationProvider(OAuthProviderType.GOOGLE)
.setResponseType("code")
.setClientId(clientId)
.setParameter("access_type", "online")
.setRedirectURI(applicationHost + "auth/google/callback")
.setScope("https://www.googleapis.com/auth/plus.login")
.buildQueryMessage();
response.sendRedirect(oAuthClientRequest.getLocationUri());
I am getting an authorization code with this. but whenever I send a request for the access_token using this code I am getting an error. (Code 400)
My access_token request:
OAuthClientRequest oAuthClientRequest = OAuthClientRequest
.tokenProvider(OAuthProviderType.GOOGLE)
.setGrantType(GrantType.AUTHORIZATION_CODE)
.setClientId(clientId)
.setClientSecret(clientSecret)
.setParameter("access_type", "online")
.setRedirectURI(applicationHost + "auth/google/callback")
.setCode(code)
.buildQueryMessage();
GitHubTokenResponse oAuthResponse = oAuthClient.accessToken(
oAuthClientRequest, GitHubTokenResponse.class);
return oAuthResponse.getAccessToken();
OAuth2 Playground response:
HTTP/1.1 400 Bad Request
Alternate-protocol: 443:quic
Content-length: 37
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
Via: HTTP/1.1 GWA
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Mon, 17 Feb 2014 09:03:52 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
{
"error": "unauthorized_client"
}
Please help me out. Thanks in advance.

You're taking an auth code from your application (ie. client id XXXXX) and pasting that into a different app (oauth playground with client id YYYYY) and expecting it to work?
That's not gonna work.
It might work if you go into the Gear option and enter your app's credentials. But I'm slightly confused why you're doing this. What is the problem you are trying to solve?
This answer might help How do I authorise an app (web or installed) without user intervention? (canonical ?)

Redirect with Curl/HttpClient: header "Location" missing

when I get the following url with curl
curl -D headers.http "http://www.springerlink.com/index/10.1007/s00453-007-9157-8"
the file headers.http contains a "Location" header:
HTTP/1.1 302 Found
Date: Tue, 27 Oct 2009 17:00:20 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: http://www.springerlink.com/link.asp?id=c104731297q64224
Set-Cookie: CookiesSupported=True; expires=Wed, 27-Oct-2010 17:00:20 GMT; path=/
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 173
but when I used the apache httpclient library this "Location:" header was missing (?).
int status = httpClient.executeMethod(method);
if(status!=HttpStatus.SC_OK &&
status!=HttpStatus.SC_MOVED_TEMPORARILY &&
status!=HttpStatus.SC_MOVED_PERMANENTLY
)
{
throw new IOException("connection failure for "+url+" status:"+status);
}
Header header=method.getResponseHeader("Location");
if(header==null )
{
for(Header h:method.getResponseHeaders())
{
LOG.info(h.toString());
}
throw new IOException(
"Expected a redirect for "+url
);
}
I've listed the headers below:
INFO: Date: Tue, 27 Oct 2009 17:05:13 GMT
INFO: Server: Microsoft-IIS/6.0
INFO: X-Powered-By: ASP.NET
INFO: X-AspNet-Version: 2.0.50727
INFO: Set-Cookie: ASP.NET_SessionId=js1o5wqnuhuh24islnvkyr45; path=/; HttpOnly
INFO: Cache-Control: private
INFO: Content-Type: text/html; charset=utf-8
INFO: Content-Length: 17245
uhh ???

What's going on is that with curl , you are getting a 302 which is actually a redirect, to the URL in the location header.
With the Apache httpclient it is doing the redirect for you, and returning the headers from the request to the redirected-to location.
To demonstrate this try
curl -D headers.http "http://www.springerlink.com/link.asp?id=c104731297q64224"
and compare the response.
edit: There are actually about 4 redirects in there if you follow each location header through with curl.

http://www.springerlink.com/index/10.1007/s00453-007-9157-8 is actually a redirect. Since the -D option means "headers only", the first one is not redirecting to the specified Location: ..., while the second one is. Take a look at the Content-Length, it's much different.
What happens when you leave out the -D?

Add this
method.setFollowRedirects(false);
Before you execute the method.
HttpClient follows the redirect automatically by default but Curl doesn't.