My problem is i'm trying to get into scopus using a crawler but it requires my crawler to enter the site through my school proxy server. I tried authenticating but it keep responding with 401 status.
public void testConnection() throws ClientProtocolException, IOException {
DefaultHttpClient httpclient = new DefaultHttpClient();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.NTLM);
httpclient.getParams().setParameter(AuthPNames.TARGET_AUTH_PREF, authpref);
NTCredentials creds = new NTCredentials("username","password","ezlibproxy1.ntu.edu.sg","ntu.edu.sg");//this is correct
httpclient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpHost target = new HttpHost("ezlibproxy1.ntu.edu.sg", 443, "https");//this is correct
// Make sure the same context is used to execute logically related requests
HttpContext localContext = new BasicHttpContext();
// Execute a cheap method first. This will trigger NTLM authentication
HttpGet httpget = new HttpGet("http://www-scopus-com.ezlibproxy1.ntu.edu.sg/authid/detail.url?authorId=14831850700");
HttpResponse response = httpclient.execute(target, httpget, localContext);
HttpEntity entity = response.getEntity();
System.out.println(EntityUtils.toString(entity));
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("Status Code:" + statusCode);
}
The status code respond is 401 (unauthorised).
Any suggestion on this?
Related
Using SOAP UI, I am able to successfully invoke the REST GET call with basic authentication
GET https://app.test.com/testing/rest/authentication-point/authentication HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Basic aPOjYVclOmIzABFhZjVpJES=
Host: app.test.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
I received response code as 200.
Similar request, When I tried to invoke via java client, it is giving 400 status code.
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("User", "Password"));
final HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = null;
response = client.execute(
new HttpGet("https://app.test.com/testing/rest/authentication-point/authentication"),
context);
int statusCode = response.getStatusLine().getStatusCode();
This code worked properly when the host was HTTP. Recently a VIP is added and made as HTTPS, after which it is not working. Please suggest a fix for this.
You need to use ssl with http client:
TrustStrategy acceptingTrustStrategy = (cert, authType) -> true;
SSLSocketFactory sf = new SSLSocketFactory(
acceptingTrustStrategy, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("https", 8443, sf));
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("User", "Password"));
final HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
DefaultHttpClient httpClient = new DefaultHttpClient(ccm);
HttpGet getMethod = new HttpGet(new HttpGet("https://app.test.com/testing/rest/authentication-point/authentication"),context);
HttpResponse response = httpClient.execute(getMethod);
int statusCode = response.getStatusLine().getStatusCode();
After trying out so many options, below code worked for me.
HttpHost targetHost = new HttpHost("app.test.com", 443, "https");
AuthCache authCache = new BasicAuthCache();
authCache.put(targetHost, new BasicScheme());
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("User", "Password"));
final HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credsProvider);
context.setAuthCache(authCache);
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = null;
response = client.execute(
new HttpGet("https://app.test.com/testing/rest/authentication-point/authentication"),
context);
After defining a HTTP host with scheme as https, and setting them to context using AuthCache, the call went through successfully.
its seems i have a bug but i cant find any clue in google.
i have the following code:
DefaultHttpClient httpClient = new DefaultHttpClient();
setupHTTPS(httpClient);
HttpContext context = new BasicHttpContext();
// set proxy data, if available
if (proxyHost != null) {
httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyHost, proxyPort), new UsernamePasswordCredentials(proxyUsername, proxyPassword));
HttpHost proxy = new HttpHost(proxyHost, proxyPort);
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
}
RedirectStrategy redirectStrategy = new DefaultRedirectStrategy();
httpClient.setRedirectStrategy(redirectStrategy);
HttpGet request = new HttpGet(url);
// allow circle redirect with limiting the numbers of redirects
httpClient.getParams().setParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, true);
httpClient.getParams().setParameter(ClientPNames.MAX_REDIRECTS, 50);
// set userAgent , if available
if(userAgent!=null){
setUserAgentForHttpClient(userAgent,httpClient);
}
HttpResponse response = httpClient.execute(request, context);
RedirectLocations locations = (RedirectLocations) context.getAttribute(DefaultRedirectStrategy.REDIRECT_LOCATIONS);
and i tried to run it on url: http://annemergmed.com/action/consumeSharedSessionAction?JSESSIONID=aaaiJSrshFbnYctW3Q4Rv&MAID=I1JsdfY3Lc%2FzCjqdBXuiSQ%3D%3D&SERVER=WZ6myaEXBLHj3ZzqSv9HPw%3D%3D&ORIGIN=378011271&RD=RD&acw=&utt=
but i got different location from what i see in firebug. totaly different! (i am runing firebug on privat browsing)
how can it be? what am i missing?
I am trying to use httpclient to make make a call to Jenkins to get a list of jobs.
When I run my code, I get an UnknownHostException.
I tried to make the same request using curl and I was able to get the result. I am not sure how to interpret this.
void nwe() throws ClientProtocolException, IOException {
HttpHost target = new HttpHost("https://<JENKINS_URL>/api");
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
new AuthScope(target.getHostName(), target.getPort()),
new UsernamePasswordCredentials("username", "password"));
CloseableHttpClient httpclient = HttpClients.custom().setDefaultCredentialsProvider(credsProvider).build();
HttpGet httpGet = new HttpGet("/json");
httpGet.setHeader("Content-type", "application/json");
BasicScheme basicAuth = new BasicScheme();
HttpClientContext localContext = HttpClientContext.create();
CloseableHttpResponse response1 = httpclient.execute(target, httpGet, localContext);
System.out.println(response1.getStatusLine());
}
The CURL command on the same URL gives me the expected output
Thanks,
Amar
Read the JavaDoc for HttpHost:
Parameters: hostname - the hostname (IP or DNS name)
So you should use just (omit the protocol and context):
HttpHost target = new HttpHost( "<JENKINS_URL>" );
and then HttpGet the /api/json part.
Cheers,
I am trying to make a simple GET request for a website, but I am getting unknown host exception.
Given below is my code--
DefaultHttpClient client = new DefaultHttpClient();
HttpHost targetHost=null;
targetHost= new HttpHost("google.com/", 80, "http");
HttpGet httpget = new HttpGet("about-us.html");
BasicHttpContext localcontext = new BasicHttpContext();
try {
HttpResponse response = client.execute(targetHost, httpget, localcontext);
It looks like you have a simple problem here.
The URL for your 'HttpHost' object is malformed. You need to drop the '/' from "google.com/".
It should work after that. I used your code with that single modification & it worked.
DefaultHttpClient client = new DefaultHttpClient();
HttpHost targetHost = new HttpHost("google.com", 80, "http");
HttpGet httpget = new HttpGet("about-us.html");
BasicHttpContext localContext = new BasicHttpContext();
HttpResponse response = null;
try { response = client.execute(targetHost, httpget, localContext);
System.out.println(response.getStatusLine()
}
catch(Exception e){
// Enter error-handling code here.
}
I am using the HttpClient 4.1.1 to test my server's REST API.
I can manage to login seem to work fine but when I try to do anything else I am failing.
Most likely I have a problem setting the cookie in the next request.
Here is my code currently:
HttpGet httpGet = new HttpGet(<my server login URL>);
httpResponse = httpClient.execute(httpGet)
sessionID = httpResponse.getFirstHeader("Set-Cookie").getValue();
httpGet.addHeader("Cookie", sessionID);
httpClient.execute(httpGet);
Is there a better way to manage the session/cookies setting in the HttpClient package?
The correct way is to prepare a CookieStore which you need to set in the HttpContext which you in turn pass on every HttpClient#execute() call.
HttpClient httpClient = new DefaultHttpClient();
CookieStore cookieStore = new BasicCookieStore();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
// ...
HttpResponse response1 = httpClient.execute(method1, httpContext);
// ...
HttpResponse response2 = httpClient.execute(method2, httpContext);
// ...