try{
HttpEntity entity=null;
HttpGet httpget=null;
HttpResponse response=null;
httpclient = new DefaultHttpClient();
httpget = new HttpGet(credentialsURL);
httpget.setHeader(HttpHeaders.USER_AGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2");
response = httpclient.execute(httpget);
entity = response.getEntity();
if (entity != null) {
EntityUtils.consume(entity);
}
}
catch(Exception e){
}
return httpclient;
I am using HttpClient to handle session.Here is my code to handle session using httpclient. But httpclient won't handle ajax calls.
I am planning to move to httpunit to overcome my problem.
Basically I have a referral URL(or authentication server) from which I need to get cookies and session and store it in client.
And use this client across website to get logged user information.
What is the appropriate solution for this using httpunit.
credentialsURL="https://www.,,,,.com./sapLogin.aspx?HOOK_URL=https://authentication.server.address&username=&password=
";
Related
I'm using FileUtils.copyURLToFile(URL, File), an Apache Commons IO 2.4 part, to download and save the file on my computer. The problem is that some sites refuse connection without referrer and user agent data.
My questions:
Is there any way to specify user agent and referrer to the copyURLToFile method?
Or should I use another approach to download a file and then save a given InputStream to file?
I've re-implement the functionality with HttpComponents instead of Commons-IO. This code allows you to download a file in Java according to its URL and save it at the specific destination.
The final code:
public static boolean saveFile(URL imgURL, String imgSavePath) {
boolean isSucceed = true;
CloseableHttpClient httpClient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet(imgURL.toString());
httpGet.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.11 Safari/537.36");
httpGet.addHeader("Referer", "https://www.google.com");
try {
CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
HttpEntity imageEntity = httpResponse.getEntity();
if (imageEntity != null) {
FileUtils.copyInputStreamToFile(imageEntity.getContent(), new File(imgSavePath));
}
} catch (IOException e) {
isSucceed = false;
}
httpGet.releaseConnection();
return isSucceed;
}
Of course, the code above takes more space then just single line of code:
FileUtils.copyURLToFile(imgURL, new File(imgSavePath),
URLS_FETCH_TIMEOUT, URLS_FETCH_TIMEOUT);
but it will give you more control over a process and let you specify not only timeouts but User-Agent and Referer values, which are critical for many web-sites.
Completing the accepted answer on how to handle timeouts:
If you want to set timeouts, you have to create the CloseableHttpClient like this:
RequestConfig config = RequestConfig.custom()
.setConnectTimeout(connectionTimeout)
.setConnectionRequestTimeout(readDataTimeout)
.setSocketTimeout(readDataTimeout)
.build();
CloseableHttpClient httpClient = HttpClientBuilder
.create()
.setDefaultRequestConfig(config)
.build();
And, it may be a good idea to create your CloseableHttpClient using a try-with-resource statement to handle its closing:
try (CloseableHttpClient httpClient = HttpClientBuilder.create().setDefaultRequestConfig(config).build()) {
... rest of the code using httpClient
}
Probably not, unless you can get a hold of the underlying mechanism that opens the URL.
I recommend to use the https://hc.apache.org/ library. This has a lot of features regarding headers etc.
When I try to send a POST request with an HttpClient to a website which uses CloudFlare, I don't get the website page content.
It looks like I get "blocked" from CloudFlare.
How can I get a solution?
This is the code I use:
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://website/");
// Request parameters and other properties.
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("user", "Bob"));
try {
httpPost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
} catch (UnsupportedEncodingException e) {
// writing error to Log
e.printStackTrace();
}
/*
* Execute the HTTP Request
*/
try {
HttpResponse response = httpClient.execute(httpPost);
HttpEntity respEntity = response.getEntity();
if (respEntity != null) {
// EntityUtils to get the response content
String content = EntityUtils.toString(respEntity);
}
} catch (ClientProtocolException e) {
// writing exception to log
e.printStackTrace();
} catch (IOException e) {
// writing exception to log
e.printStackTrace();
}
First check and make sure you are sending required parameters in your request and try adding user agent in your request :
params.add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0");
If you really want to check which parameters are being sent in request when you are doing it from browser, one way I know is to use Firefox extension Tamper Data. It will show you all parameters of header and post data and will allow you to modify them also.
check your server not blocked post requests in (CORS) by default
I need to login to external web site, and use HttpClient in combination with CookieStore. As a guide I used http://svn.apache.org/repos/asf/httpcomponents/httpclient/trunk/httpclient/src/examples/org/apache/http/examples/client/ClientCustomContext.java and http://hc.apache.org/httpcomponents-client-4.4.x/tutorial/html/statemgmt.html
Here is my code:
RequestConfig globalConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.BEST_MATCH).build();
CookieStore cookieStore = new BasicCookieStore();
HttpClientContext context = HttpClientContext.create();
context.setCookieStore(cookieStore);
CloseableHttpClient httpClient = HttpClients.custom()
.setDefaultRequestConfig(globalConfig)
.setDefaultCookieStore(cookieStore)
.build();
HttpGet httpGet = new HttpGet(urlAddress);
httpGet.addHeader("Accept", "text/html, application/xhtml+xml, */*");
httpGet.addHeader("Accept-Language", "en-US,en;q=0.7,ru;q=0.3");
httpGet.addHeader("User-Agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
httpGet.addHeader("Accept-Encoding","gzip, deflate");
//httpGet.addHeader("Host", host); // should I copy host here?
httpGet.addHeader("Connection", "Keep-Alive");
CloseableHttpResponse response1 = httpClient.execute(httpGet);
List<Cookie> cookies1 = context.getCookieStore().getCookies();
try {
System.out.println(response1.getStatusLine());
HttpEntity entity1 = response1.getEntity();
// do something useful with the response body
streamToFile(entity1.getContent(), "./login_result_1.html");
// and ensure it is fully consumed
EntityUtils.consume(entity1);
} finally {
response1.close();
}
That works fine.
When I do further request to the site, should I do any special code to pass cookies obtained in the request above?
Should I regenerate cookies or that will be done by Apache?
I was not able to find clear answer. Also my further request is not handled by web site properly and it doesn't allow me to login.
HttpPost httpPost = new HttpPost(urlAddress);
httpPost.addHeader("Accept", "text/html, application/xhtml+xml, */*");
httpPost.addHeader("Accept-Language", "en-US,en;q=0.7,ru;q=0.3");
httpPost.addHeader("Referer", "http://virtonomica.ru/");
httpPost.addHeader("User-Agent", "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)");
httpPost.addHeader("Accept-Encoding","gzip, deflate");
httpPost.addHeader("Content-Type", "application/x-www-form-urlencoded");
httpPost.addHeader("Accept-Encoding", "gzip, deflate");
//httpPost.addHeader("Host", "virtonomica.ru");
//httpPost.addHeader("Content-Length", "51");//LENGTH
httpPost.addHeader("Connection", "Keep-Alive");
httpPost.addHeader("Cache-Control", "no-cache");
httpPost.setEntity(new UrlEncodedFormEntity(nvps)); // nvps is List<NameValuePair> which contains POST data
CloseableHttpResponse response = httpClient.execute(httpPost);
Please advise.
i am doing android application it need a server url call which need only response code for identify the call has successfully made ( like 200 ) but no need of any other data like json.
in other words i dont want to implement the reader objects (inputStreamReader) and bla bla bla..
so the code will be like
call url
get response code
exit
You can do something like that:
HttpGet httpRequest = new HttpGet(myUri);
HttpClient httpclient = new DefaultHttpClient();
HttpResponse response = httpclient.execute(httpRequest);
response.getStatusLine().getStatusCode(); // Your status-code
Use the HTTP HEAD request, for example using apache http client.
Use status code:
HttpGet get = new HttpGet(url);
HttpClient httpclient = new DefaultHttpClient();
HttpResponse response = httpclient.execute(get);
int code = response.getStatusLine().getStatusCode()
Try the below code:
URL obj = new URL("http://www.google.com/");
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// you can set the required parameter for your connection object
con.setRequestMethod("GET");
int responseCode = con.getResponseCode();
Try this please on your post execute method from asyntask method:
StringBuilder builder = new StringBuilder();
// Set up HTTP post
// HttpClient is more then less deprecated. Need to change to URLConnection
HttpClient client = new DefaultHttpClient( );
HttpGet httpGet = new HttpGet(params[0]);
HttpResponse response = client.execute(httpGet);
StatusLine statusLine = response.getStatusLine();
int statusCode = statusLine.getStatusCode();
if (statusCode == 200) {
//do some work
}
I'm porting an Android app on GAE.
The app compile and post a login form to enter in a
password protected site.
I'm using UrlFetchTransport instead of DefaultHttpClient.
I can get web page without problem using the following piece of code:
HttpTransport HTTP_TRANSPORT = new UrlFetchTransport();
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
GenericUrl genericUrl = new GenericUrl(url);
HttpRequest request = requestFactory.buildGetRequest(genericUrl);
com.google.api.client.http.HttpHeaders httpHeaders = request.getHeaders();
httpHeaders.setUserAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
httpHeaders.setAccept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
String paginaRicevuta = request.execute().parseAsString();
I would try a POST request using the following:
UrlFetchTransport HTTP_TRANSPORT = new UrlFetchTransport();
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
GenericUrl genericUrl = new GenericUrl("https://www.......");
List<BasicNameValuePair> nvps = new ArrayList<BasicNameValuePair>();
nvps.add(new BasicNameValuePair("ReturnUrl", ""));
nvps.add(new BasicNameValuePair("userName", "username"));
nvps.add(new BasicNameValuePair("password","password"));
//HERE HOW DEFINE httpContent????
HttpContent httpContent=null;
final HttpRequest postRequest = requestFactory.buildPostRequest(genericUrl,httpContent);
postRequest.setContent(httpContent);
postRequest.setFollowRedirects(true);
HttpHeaders httpHeaders = postRequest.getHeaders();
httpHeaders.setUserAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
httpHeaders.setAccept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpHeaders.setContentType("application/x-www-form-urlencoded");
postRequest.setHeaders(httpHeaders);
String pagePost = postRequest.execute().parseAsString();
but I don't know how format a right POST request?
Thanks for your help.