I need to get the API response contents from an API server to my local server, via a proxy authentication. The API request will be like:
http://api.example.com/nav_page/head?locale=in&pointer=old&cont=/resque/&scope=old&release=new
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
public class ClientProxyAuthentication {
public static void main(String[] args) throws Exception {
DefaultHttpClient httpclient = new DefaultHttpClient();
try {
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("proxy.example.com", 8080),
new UsernamePasswordCredentials("Domain\user", "pass"));
HttpHost targetHost = new HttpHost("api.example.com", 80, "http");
HttpHost proxy = new HttpHost("proxy.example.com", 8080);
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
HttpGet httpget = new HttpGet("/");
System.out.println("executing request: " + httpget.getRequestLine());
System.out.println("via proxy: " + proxy);
System.out.println("to target: " + targetHost);
HttpResponse response = httpclient.execute(targetHost, httpget);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
System.out.println(EntityUtils.toString(entity));
}
Header[] headers = response.getAllHeaders();
for (int i = 0; i<headers.length; i++) {
System.out.println(headers[i]);
}
EntityUtils.consume(entity);
} finally {
// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}
}
In this code, if target URL is specified as api.example.com, The authentication succeeds and getting the resource (but there is no resource in base dir, hence nothing useful received). But to get correct response, I need to provide the navigation path inside that host.
Hence if i append the path like
HttpHost targetHost = new HttpHost("api.example.com/nav_page/head?locale=in&pointer=old&cont=/resque/&scope=old&release=new", 80, "http");
I am getting some error response, but not correct one.
HTTP/1.1 503 Service Unavailable
Response content length: 442
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<p>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
I am able to get the response, if i hit URL directly in browser.
May i know how to send the request for the navigation path?
Related
I Have a webservice that uploads a file. (details in the first link)
To communicate with this webservice I use org.apache.http library (Apache HttpComponents). I have found most of this code if not all of it here.
Sadly the solution only works with images and despite the efforts when trying to upload a video is shows the error of Content Length too long. To try and fix this I decided to replace it with what's in current use. At first using HttpClientBuilder
HttpClient httpclient = HttpClientBuilder.create().disableContentCompression().build();
My try failed as it still gives the same error and after research I found out that I need to use the CloseableHttpClient as follow to disable the automatic headers
CloseableHttpClient httpclient = HttpClients.custom()
.setHttpProcessor(HttpProcessorBuilder.create().build())
.build();
That worked but when setting the headers something doesn't pass I can't figure it out. I did it as follow
httppost.addHeader("Keep-Alive", "timeout=5, max=100");
httppost.addHeader("Connection", "Keep-Alive");
httppost.addHeader("Content-Type", "text/html; charset=UTF-8");
But when tracing the response this is what I have
Http Client Builder Trace
Date,Tue, 28 Mar 2017 18:30:50 GMT
Server,Apache/2.4.23 (Win64) PHP/7.0.10
X-Powered-By,PHP/7.0.10
Content-Length,43
Keep-Alive,timeout=5, max=100
Connection,Keep-Alive
Content-Type,text/html; charset=UTF-8
Closable Response Trace
Date,Tue, 28 Mar 2017 18:31:27 GMT
Server,Apache/2.4.23 (Win64) PHP/7.0.10
Content-Length,311
Connection,close
Content-Type,text/html; charset=iso-8859-1
I tried to set the headers to the response but I am haven't figured how to instantiate it properly to set the headers. And also I am trying to understand why the httppost is not being taken in consideration. Did I miss something?
One last thing when I use Closable The way I did it doesn't upload anymore due to missing headers I guess.
EDIT
Error when uploading video
Exception in thread "main" org.apache.http.ContentTooLongException: Content length is too long: 16363535
at org.apache.http.entity.mime.MultipartFormEntity.getContent(MultipartFormEntity.java:103)
at org.apache.http.util.EntityUtils.toString(EntityUtils.java:199)
at org.apache.http.util.EntityUtils.toString(EntityUtils.java:306)
at testfileupload.PostFile.main(PostFile.java:100)
C:\Users\User\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 0 seconds)
EDIT
I changed the max file size for php following this but it still shows the same error
Update
This is how I handle the multipart (I use multipartform)
HttpEntity resEntity = MultipartEntityBuilder.create()
.addBinaryBody("userfile", file, ContentType.create("video/mp4"), file.getName())
.build();
I rememeber reading something about it (not sure) And as for the php.ini params that need to be of higher value upload_max_filesize = 40M and post_max_size = 40M and memory_limit = 128M which I am kinda sure its enough (using wamp64).
I kept trying to udnerstand the content length issue since ..
I found out that the HttpEntity sets calculates the length and I checked the value its 2 bits off I believe but no matter the conenction type it just doesn't work and sticks to content length error.
I also tried
HttpClient httpclient = HttpClients.custom().setDefaultHeaders(Arrays.asList(CustomHeader)).build();
But still nothing. At this point I highly doubt its about setting the header and more related to HttpEntity with the Multipart or the HttpPost but It could be else.
Upload
I tried with a 9s Video of a size ~350Kb but it still gives the content length error which means there might be a default content set or none at all
So I dont know what I did / changed but now its working with the error.
I would really want to know why this cause an error it doesn't make sense so far unless its calling something that can't take too high of a value (method of param type given value more than it can)
To be clear the error is generated when I use this (verifcation area)
System.out.println(response.getStatusLine());
if (resEntity != null) {
// error from below, kinda forgot about it
System.out.println(EntityUtils.toString(resEntity));
}
if (resEntity != null) {
// resEntity.consumeContent();
}
The webservice is the same as in the first link.
Below is my code (dont mind the comments I didn't clear the decrepated / changed / replaced / test / different approaches)
import java.io.File;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.HttpVersion;
import org.apache.http.ProtocolVersion;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntity;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.ContentBody;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.params.CoreProtocolPNames;
import org.apache.http.protocol.HTTP;
import org.apache.http.protocol.HttpProcessorBuilder;
import org.apache.http.util.EntityUtils;
public class PostFile {
public static void main(String[] args) throws Exception {
// HttpClient httpclient = new DefaultHttpClient();
// CloseableHttpClient httpclient = HttpClients.custom()
// .setHttpProcessor(HttpProcessorBuilder.create().build())
// .build();
// Header headerCustom = new BasicHeader(HttpHeaders.CONTENT_LENGTH, "15");
// HttpClient httpclient = HttpClients.custom().build();
// HttpClient httpclient = HttpClientBuilder.create().disableContentCompression().build();
HttpClient httpclient = HttpClientBuilder.create().build();
// httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
HttpPost httppost = new HttpPost("http://localhost/userVids/upload.php");
httppost.setProtocolVersion(new ProtocolVersion("HTTP", 1, 1));
File file = new File("C:/Users/User/Desktop/test_video.mp4");
// httppost.addHeader("Content-Length", String.valueOf(file.length()));
// MultipartEntity mpEntity = new MultipartEntity();
// ContentBody cbFile = new FileBody(file, ContentType.MULTIPART_FORM_DATA);
// mpEntity.addPart("userfile", cbFile);
// System.out.println(mpEntity.getContentLength());
//
// httppost.setEntity(mpEntity);
/*
Date,Tue, 28 Mar 2017 18:30:50 GMT
Server,Apache/2.4.23 (Win64) PHP/7.0.10
X-Powered-By,PHP/7.0.10
Content-Length,43
Keep-Alive,timeout=5, max=100
Connection,Keep-Alive
Content-Type,text/html; charset=UTF-8
*/
/*
Date,Tue, 28 Mar 2017 18:31:27 GMT
Server,Apache/2.4.23 (Win64) PHP/7.0.10
Content-Length,311
Connection,close
Content-Type,text/html; charset=iso-8859-1
*/
HttpEntity resEntity = MultipartEntityBuilder.create()
.addBinaryBody("userfile", file, ContentType.create("video/mp4"), file.getName())
.build();
System.out.println("Entity Length " + resEntity.getContentLength());
// httppost.addHeader("Keep-Alive", "timeout=5, max=100");
// httppost.addHeader("Connection", "Keep-Alive");
// httppost.addHeader("Content-Type", "text/html; charset=UTF-8");
// httppost.addHeader("Content-Disposition", "form-data; name=\"userfile\"; filename=\"star_empty.png\"");
// httppost.addHeader("Content-Type", "image/png");
// httppost.addHeader("Content-Transfer-Encoding", "binary");
// httppost.addHeader("Content-Length", "99999");
httppost.setEntity(resEntity);
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = httpclient.execute(httppost);
// HttpEntity resEntity = response.getEntity();
System.out.println("Headers (name,value)");
List<Header> httpHeaders = Arrays.asList(response.getAllHeaders());
httpHeaders.forEach((header) -> {
System.out.println(header.getName() + "," + header.getValue());
});
System.out.println(response.getStatusLine());
if (resEntity != null) {
System.out.println(EntityUtils.toString(resEntity));
}
if (resEntity != null) {
// resEntity.consumeContent();
}
// httpclient.getConnectionManager().shutdown();
}
}
BUT A HUGE NOTICE I had my php.ini (changed all of them in wamp didn't know if it would consider only the running version or reads all (didn't test) ) and changed the values as Lucas Holt (he fixed it i.e, I am just providing for who needs or maybe explanation) said and he deserves credit for it. (I am sure it didn't work before that I just tested a lot and dont remember really.)
I have one url, which make 2 internal redirects then finally returns ok response.
First URL: : http://www.someurl.com/
Redirect URL 1: : http://www.someurl_1.com/ with response 302
Redirect URL 2: : http://www.someurl_2.com/ with response 302
Final URL: : http://www.finalurl.com/ with response 200
Internally Redirect URL 1 send some cookie to Redirect URL 2.
What I have to do is get cookie of which set for Redirect URL 2:.
Here is my java code.
HttpClient client = HttpClientBuilder.create().build();
HttpGet get = new HttpGet(myurl);
get.setHeader("User-Agent", "Mozilla");
get.setHeader("Accept"," text/html,application/xhtml+xml,application/xml;");
get.setHeader("Accept-Language", "en-US,en;q=0.8");
get.setHeader("Accept-Encoding"," gzip, deflate");
get.setHeader("Connection","keep-alive");
get.setHeader("Cookie",JSESSIONID+");
// get.setHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
Header[] requestheaders = get.getAllHeaders();
System.out.println("requestheaders >>> ");
for(Header header: requestheaders){
System.out.println(header.getName()+"-------------------- "+header.getValue());
}
HttpResponse response = client.execute(get);
System.out.println("response 7 "+response);
System.out.println("Headers are");
Header[] headers = response.getAllHeaders();
for (int i = 0; i < headers.length; i++) {
System.out.println((headers[i].getName()+"___________________"+headers[i].getValue()));
}
This code gives me final response rather than intermediate redirected response. So can anyone please suggest me what is better way of doing it.
Other thing I have checked is:-
I have disable redirect, this response gives me very first url of this process.
I have used Jsoup to disable redirect which gives same out put as above.
Whereas fetching such redirected cookie is possible in ruby , i have done that.
But I have to do this in java.
httpclient 4.5
This worked for me
HttpClient client = HttpClientBuilder.create().setRedirectStrategy(new DefaultRedirectStrategy() {
public boolean isRedirected(HttpRequest request, HttpResponse response, HttpContext context) {
boolean isRedirect=false;
try {
isRedirect = super.isRedirected(request, response, context);
Header[] requestheaders = response.getAllHeaders();
System.out.println("getAuthToken >>> ");
for(Header header: requestheaders){
System.out.println(header.getName()+"-------------------- "+header.getValue());
if(header.getName().equalsIgnoreCase("Set-Cookie") && header.getValue().startsWith("auth-token")){
System.out.println("Auth_Cookie "+header.getValue().split(";")[0]);
auth_token = header.getValue().split(";")[0];
}
}
} catch (ProtocolException e) {
e.printStackTrace();
}
if (!isRedirect) {
int responseCode = response.getStatusLine().getStatusCode();
if (responseCode == 301 || responseCode == 302) {
return true;
}
}
return false;
}
}).build();
HttpGet get = new HttpGet(url);
client.execute(get);
For more technical detail check my blog here.
If you want more control over the redirection behavior, you can probably change the RedirectStrategy of the http client.
You can create your own RedirectStrategy and use it:
HttpClient instance = HttpClientBuilder.create()
.setRedirectStrategy(new LaxRedirectStrategy()).build();
I had the same issue where I had a number of redirects but I needed the cookies from each response to be passed on to the redirects. I could have probably done more work to figure out what cookies each redirect really needed but I just built up a map of all the cookies and each redirect got the full set of cookies from the combined redirects before it. Here is my code...
import java.util.HashMap;
import java.util.Map;
import org.apache.http.Header;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.ProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.RedirectStrategy;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.impl.client.DefaultRedirectStrategy;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.protocol.HttpContext;
import org.springframework.http.HttpEntity;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;
...
RestTemplate restTemplate = new RestTemplate();
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
RedirectStrategy rs = new DefaultRedirectStrategy() {
Map<String, Header> cookies = new HashMap<>();
#Override
public HttpUriRequest getRedirect(HttpRequest request, HttpResponse response,
HttpContext context) throws ProtocolException {
//
// Get the cookies out of the response so we can inject them into the redirect.
//
for (Header header : response.getHeaders("Set-Cookie")) {
this.cookies.put(header.getName(), header);
}
HttpUriRequest redirect = super.getRedirect(request, response, context);
for (Header cookie : this.cookies.values()) {
redirect.addHeader("Cookie", cookie.getValue());
}
return redirect;
}
};
final HttpClient httpClient = HttpClientBuilder.create().setRedirectStrategy(rs).build();
factory.setHttpClient(httpClient);
restTemplate.setRequestFactory(factory);
ResponseEntity<String> response = restTemplate.getForEntity("<my_url>", String.class);
I have been trying to connect to CRM using the below Java code, but I am getting an authentication issue.
package com.raj;
import java.io.IOException;
import org.apache.commons.io.IOUtils;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.NTCredentials;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
public class MSDynaOData {
/**
* #param args
*/
public static void main(String[] args) throws ClientProtocolException, IOException {
DefaultHttpClient httpclient = new DefaultHttpClient();
NTCredentials creds = new NTCredentials("XXXXXXXXXX", "XXXXXXX", "", "");
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost("XXXXXXXXX", 80, "http");
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
String url = "/XRMServices/2011/Organization.svc/Account";
url += "?$select=Name";
HttpGet httpget = new HttpGet(url);
httpget.addHeader("Accept", "application/json");
HttpResponse response = httpclient.execute(target, httpget, localContext);
HttpEntity entity = response.getEntity();
System.out.println(" Status :: " + response.getStatusLine());
for (Header header : response.getAllHeaders()) {
System.out.println(header.getName() + " : " + header.getValue());
}
System.out.println(IOUtils.toString(entity.getContent()));
}
}
Response contains :
Constants.TokenizedStringMsgs.GENERIC_ERROR = "<H1>Sorry, but we're having trouble signing you in</H1><p>Please try again in a few minutes. If this doesn't work, you might want to contact your admin and report the following error: #~#ErrorCode#~#.</p>";
Constants.TokenizedStringMsgs.UPN_DISAMBIGUATE_MESSAGE = "It looks like #~#MemberName_LS#~# is used with more than one account. Which account do you want to use?";
I am able to login to CRM 2011 online through the credentials used in Java code successfully, but when I used same credentials in the code I am getting the above login issue response.
Please let me know if I am missing anything in the above code.
I am tring to perform Digest Authentication using the HttpClient library, but I keep getting: HTTP/1.1 401 Unauthorized.
When I try the request from Firefox it works fine and I get a response correctly, so I know the server authentication is working fine.
Update: moved Working code to answer.
The following code worked for me
import java.util.Random;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.AuthCache;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.auth.DigestScheme;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.impl.client.BasicAuthCache;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.util.EntityUtils;
/**
* A simple example that uses HttpClient to execute an HTTP request against a
* target site that requires user authentication.
*/
public class RestClient {
public static void main(String args[]) throws Exception {
HttpHost targetHost = new HttpHost("localhost", 8001, "http");
DefaultHttpClient httpclient = new DefaultHttpClient();
final String userName = "admin";
final String password = "password";
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("localhost", 8001),
new UsernamePasswordCredentials(userName, password));
// Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// Generate DIGEST scheme object, initialize it and add it to the local
// auth cache
DigestScheme digestAuth = new DigestScheme();
// Suppose we already know the realm name
digestAuth.overrideParamter("realm", "some realm");
// Suppose we already know the expected nonce value
digestAuth.overrideParamter("nonce", "whatever");
authCache.put(targetHost, digestAuth);
// Add AuthCache to the execution context
BasicHttpContext localcontext = new BasicHttpContext();
localcontext.setAttribute(ClientContext.AUTH_CACHE, authCache);
HttpGet httpget = new HttpGet("http://localhost:8001/rest/test");
try {
HttpResponse response = httpclient.execute(targetHost, httpget, localcontext);
HttpEntity entity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
}
EntityUtils.consume(entity);
} finally {
httpclient.getConnectionManager().shutdown();
}
}
}
How to make HTTPClient use custom User-Agent header?
The following code submits empty user-agent. What am I missing?
import java.io.IOException;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.CoreProtocolPNames;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
public class TestHTTP {
public static void main(String[] args) throws ClientProtocolException, IOException {
HttpGet request = new HttpGet("http://tool.keepmeapi.com/echo");
HttpContext HTTP_CONTEXT = new BasicHttpContext();
HTTP_CONTEXT.setAttribute(CoreProtocolPNames.USER_AGENT, "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13");
request.setHeader("Referer", "http://www.google.com");
HttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(request, HTTP_CONTEXT);
if (response.getStatusLine().getStatusCode() < 200 || response.getStatusLine().getStatusCode() >= 400) {
throw new IOException("Got bad response, error code = " + response.getStatusLine().getStatusCode());
}
HttpEntity entity = response.getEntity();
if (entity != null) {
System.out.println(EntityUtils.toString(entity));
EntityUtils.consume(entity);
}
}
}
Note: The solution is for users using the old httpcomponents 4.2 and before.
The line
request.setHeader("User-Agent", "MySuperUserAgent");
is missing. Add it and enjoy.
Note: Starting from httpcomponents 4.3 this solution is deprecated.
You can also set a global user agent value instead of per request:
String userAgent = "NewUseAgent/1.0";
HttpClient httpClient = new DefaultHttpClient();
httpclient.getParams().setParameter(CoreProtocolPNames.USER_AGENT, userAgent);
With httpcomponents 4.3 you should use the client builder to set the user agent:
HttpClient httpClient = HttpClients.custom()
.setUserAgent("my UserAgent 5.0")
.build();
httpClient.execute(new HttpGet("http://www.google.de"));