I'm using Apache HttpComponents to GET some web pages for some crawled URLs. Many of those URLs actually redirect to different URLs (e.g. because they have been processed with a URL shortener). Additionally to downloading the content, I would like to resolve the final URLs (i.e. the URL which provided the downloaded content), or even better, all URLs in the redirect chain.
I have been looking through the API docs, but got no clue, where I could hook. Any hints would be greatly appreciated.
One way is to turn off automatic redirect handling by setting the relevant parameter, and do it yourself by checking for 3xx responses, and manually extracting the redirect location from the responses "Location" header.
Here's a full demo of how to do it using Apache HttpComponents.
Important Details
You'll need to extend DefaultRedirectStrategy like so:
class SpyStrategy extends DefaultRedirectStrategy {
public final Deque<URI> history = new LinkedList<>();
public SpyStrategy(URI uri) {
history.push(uri);
}
#Override
public HttpUriRequest getRedirect(
HttpRequest request,
HttpResponse response,
HttpContext context) throws ProtocolException {
HttpUriRequest redirect = super.getRedirect(request, response, context);
history.push(redirect.getURI());
return redirect;
}
}
expand method sends a HEAD request which causes client to collect URIs in spy.history deque as it follows redirects automatically:
public static Deque<URI> expand(String uri) {
try {
HttpHead head = new HttpHead(uri);
SpyStrategy spy = new SpyStrategy(head.getURI());
DefaultHttpClient client = new DefaultHttpClient();
client.setRedirectStrategy(spy);
// FIXME: the following completely ignores HTTP errors:
client.execute(head);
return spy.history;
}
catch (IOException e) {
throw new RuntimeException(e);
}
}
You may want to set maximum number of redirects followed to something reasonable (instead of the default of 100) like so:
BasicHttpParams params = new BasicHttpParams();
params.setIntParameter(ClientPNames.MAX_REDIRECTS, 5);
DefaultHttpClient client = new DefaultHttpClient(params);
Related
I'm currently migrating an application from jersey 1 to 2. In the old app, we used a ClientFilter for all jersey clients that automatically refreshed expired OAuth tokens like this:
#Override
public ClientResponse handle(ClientRequest cr) {
ClientResponse resp = getNext().handle(cr);
if (resp.getStatus() == Status.UNAUTHORIZED.getStatusCode()) {
// Try to refresh the token
boolean refreshed = refreshToken(oAuthInfo);
if (refreshed) {
resp = getNext().handle(cr);
}
}
return resp;
}
It might not haven been the most elegant way, but the benefit was that rest client users did not have to care about expired tokens themselves.
With the ContainerResponseFilter for jersey 2, this does not seem to be that simple anymore. The only option I currently see is to use the ClientRequestContext and try to re-create the original request using getClient, getHeaders etc... and then update the result in ContainerResponseContext. This however seems a bit clunky so I was wondering if there is any more convenient way to refresh an OAuth token without having to deal with this wherever a jersey client is used?
It looks like there is not more convenient way than intercepting the response with a client filter, refreshing the token if needed and trying to repeat the exact same request with the new token. In fact, this approach is also used by jersey own filter classes.
Sample code for repeating the original rest call from within a filter class can be found in jerseys HttpAuthenticationFilter:
static boolean repeatRequest(ClientRequestContext request, ClientResponseContext response, String newAuthorizationHeader) {
Client client = request.getClient();
String method = request.getMethod();
MediaType mediaType = request.getMediaType();
URI lUri = request.getUri();
WebTarget resourceTarget = client.target(lUri);
Invocation.Builder builder = resourceTarget.request(mediaType);
MultivaluedMap<String, Object> newHeaders = new MultivaluedHashMap<String, Object>();
for (Map.Entry<String, List<Object>> entry : request.getHeaders().entrySet()) {
if (HttpHeaders.AUTHORIZATION.equals(entry.getKey())) {
continue;
}
newHeaders.put(entry.getKey(), entry.getValue());
}
newHeaders.add(HttpHeaders.AUTHORIZATION, newAuthorizationHeader);
builder.headers(newHeaders);
builder.property(REQUEST_PROPERTY_FILTER_REUSED, "true");
Invocation invocation;
if (request.getEntity() == null) {
invocation = builder.build(method);
} else {
invocation = builder.build(method,
Entity.entity(request.getEntity(), request.getMediaType()));
}
Response nextResponse = invocation.invoke();
if (nextResponse.hasEntity()) {
response.setEntityStream(nextResponse.readEntity(InputStream.class));
}
MultivaluedMap<String, String> headers = response.getHeaders();
headers.clear();
headers.putAll(nextResponse.getStringHeaders());
response.setStatus(nextResponse.getStatus());
return response.getStatus() != Response.Status.UNAUTHORIZED.getStatusCode();
}
This code is used for example in DigestAuthenticator or BasicAuthenticator to repeat a request with provided credentials in case an Unauthorised response is received from server.
Let's say I have 5 API endpoints and 4 of them require the #Header "Authorization"; the other one doesn't.
Is there a drawback or anything of the sort when I use an Interceptor to insert the header for all API calls even though one of the endpoints does not require it? :)
Probably would be better use a more customizable approach that's retrofit2 provides - dynamic headers. A dynamic header is passed like a parameter to the method. The provided parameter value gets mapped by Retrofit before executing the request. Example:
#GET("/yourEndpoint")
Call<List<Obj>> getSomth(#Header("Your-Header") String yourHeader);
But if you only worrying about extra header passing, I don't see any drawback in your particular case.
#Override
public okhttp3.Response intercept(Chain chain) throws IOException {
Request.Builder builder = chain.request().newBuilder();
Request request = chain.request();
if (!request.url().toString().contains("/tapi/login/login")) {
String cookie = SharedObj.getCookie();
builder.addHeader("Cookie", cookie);
}
return chain.proceed(builder.build());
}
This is my solution.All api need cookie except login api('/tapi/login/login') ,So I judge url to decide to wether to add cookie in request.
There is no problem using Interceptor with header for all API.
Set headers in interceptors and use to all API. If the API method is with or without auth it will be worked.
Like this - Create ones, use everywhere.
OkHttpClient.Builder httpClient = new OkHttpClient.Builder();
httpClient.addInterceptor(chain -> {
Request original = chain.request();
Request.Builder requestBuilder = original.newBuilder();
//Add headers here using requestbuilder.
String authToken =//Your Authtoken
if (authToken != null)
requestBuilder.header("Authorization", authToken);
requestBuilder.method(original.method(), original.body());
return chain.proceed(requestBuilder.build());
});
In Retrofit
Retrofit retrofit= new Retrofit.Builder().baseUrl("baseUrl").client(httpClient.build()).build();
Is Websocket supported in gargoylesoftware library? I want to get websocket object in webclient.
Yes, WebSocket is supported since version 2.11. However, it is always recommended to use the latest version.
Please ensure you use BrowserVersion with recent browser, e.g. CHROME, FIREFOX_38, or INTERNET_EXPLORER_11.
E.g.:
try (final WebClient webClient = new WebClient(BrowserVersion.CHROME)) {
HtmlUnit will automatically handle the JavaScript with WebSocket.
Update:
To intercept the requests and responses, you can use:
new WebConnectionWrapper(webClient) {
public WebResponse getResponse(WebRequest request) throws IOException {
WebResponse response = super.getResponse(request);
if (request.getUrl().toExternalForm().contains("my_url")) {
String content = response.getContentAsString("UTF-8");
//change content
WebResponseData data = new WebResponseData(content.getBytes("UTF-8"),
response.getStatusCode(), response.getStatusMessage(), response.getResponseHeaders());
response = new WebResponse(data, request, response.getLoadTime());
}
return response;
}
};
I'm in the midst of testing my application which is using an HTTP-server. Instead of mocking I decided to go with a HTTP server fixture. Meaning that I do not have to mock any productional code. To accomplish this goal I currently chose for a free to use 3rd party library fixd.
I was able to successfully create several unit tests - which are working by means of a GET request. Most are quite simple, i.e.:
#Test
public void verifyConnectionTest()
{
try
{
final String body = FileUtils.readFileToString(RESOURCE);
final String path = "/";
this.server.handle(Method.GET, path).with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response)
{
response.setStatusCode(200);
response.setContentType("text/xml");
response.setBody(body);
}
});
// Setting up my HTTP client
// Execute some tasks
// asserting of everything was valid
}
catch (final IOException e)
{
fail(e.getMessage());
}
}
But I now have to send a POST request with multipart/form-data. Which does not make much of a difference other than changing the method and content-type:
#Test
public void executeStepTest()
{
try
{
final String body = FileUtils.readFileToString(SERVICE_RESPONSE);
final String path = "/";
this.server.handle(Method.POST, path, "multipart/form-data").with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response)
{
response.setStatusCode(200);
response.setContentType("text/xml");
response.setBody(body);
}
});
// Setting up my HTTP client
// Execute some tasks
// asserting of everything was valid
}
catch (final IOException e)
{
fail(e.getMessage());
}
}
However I get the following error: [ERROR] could not find a handler for POST - / - multipart/form-data; boundary=bqCBI7t-VW1xaJW7BADmTiGMg9w_YM2sHH8ukJYx and my guess is that fixd doesn't recognize the boundary-party. Since the documentation does not show an example I'm quite stuck on this part.
I tried using some wildcards such as '*', no succes. Thus; I need a way to either tell fixd to accept that boundary or use some wildcards I didn't yet discover. Any help would be great, thanks!
I've been making some debug and it seems to be that the problem is in the fixd core.
Basically, fixd indexes every RequestHandlerImpl by a HandlerKey (which includes ContentType as part of the key) in the map handlerMap. See method org.bigtesting.fixd.core.FixtureContainer#resolve.
...
HandlerKey key = new HandlerKey(method, route, contentType);
RequestHandlerImpl handler = handlerMap.get(key);
if (handler == null) {
// Error
}
...
Problem: When the request is multipart/form-data, boundary data (which it's generated dinamically every request) is part of the content type. So, any handler is found in handlerMap because the key changes with every running.
I've made a little test only to check that this is the cause of the problem, passing the contentType to fixd server.handle after the creation of the multipart request, and it works fine.
See the test below:
#Test
public void verifyConnectionTest_multipart() {
try {
// 1. Create multipart request (example with http-commons 3.1)
PostMethod filePost = new PostMethod(url);
Part[] parts = { new StringPart("param", "value") };
MultipartRequestEntity request = new MultipartRequestEntity(parts, filePost.getParams());
filePost.setRequestEntity(request);
// 2. fixd server handle (passing the request content type)
this.server.handle(Method.POST, "/", request.getContentType()).with(
new HttpRequestHandler() {
#Override
public void handle(final HttpRequest request,
final HttpResponse response) {
response.setStatusCode(200);
response.setContentType("text/xml");
}
});
// 3. Execute multipart request
HttpClient client = new HttpClient();
int status = client.executeMethod(filePost);
// 4. Assertions
Assert.assertEquals(200, status);
} catch (Exception e) {
Assert.fail(e.getMessage());
}
}
Hope it helps you to clarify the problem. Cheers
This was a bug in fixd, and has been fixed in version 1.0.3. Your original code should work using this new version of fixd.
I am using CRest to deserialize a JSON stream on Android. My first steps looks very promising.
To get the JSON stream from the server and not the XML one I use the following construct:
(Accept: application/json)
#EndPoint("http://localhost:8080/myserver/rest")
#Param(name = "Accept", value = "application/json", dest = Destination.HEADER)
public interface VersionService {
#ConnectionTimeout(10000)
#Path("/version")
VersionTO getVersion();
}
This works but it's a bit annoying to copy the "Param thing" for every service.
Is there a better way to configure all Services at one place only to return JSON?
Well I'm afraid there isn't any simple way in the current version, but feel free to open a request on the issue tracker.
Cheers,
Laurent
I faced a similar situation where I used a custom HTTP client. In your case it could look like as follows:
DefaultHttpClient client = new DefaultHttpClient();
client.addRequestInterceptor(new HttpRequestInterceptor() {
public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
request.addHeader("Accept", "application/json");
}
});
CRestBuilder builder = new CRestBuilder();
builder.expectsJson();
builder.setRestService(new HttpClientRestService(client));
Another option is to set default parameter for ClientPNames.DEFAULT_HEADERS of the custom HttpClient instance.
Details can be found on http://hc.apache.org/httpcomponents-client-ga/tutorial/html/httpagent.html.