I am trying to implement a Java Code that fetch urls like this one:
http://trendistic.indextank.com/_trending-topics-archive/2011-12-25
The code I wrote is based in HTTPClient as follow:
HttpClient httpclient = new DefaultHttpClient();
// Create a local instance of cookie store
CookieStore cookieStore = new BasicCookieStore();
// Create local HTTP context
HttpContext localContext = new BasicHttpContext();
// Bind custom cookie store to the local context
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpGet httpget = new HttpGet(urls);
try {
System.out.println("executing request " + httpget.getURI());
HttpResponse response = httpclient.execute(httpget, localContext);
String line = "";
StringBuilder total = new StringBuilder();
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while ((line = rd.readLine()) != null) {
total.append(line+"\n");
}
System.out.println(total);
return (total.toString());
} catch (Exception e) {
e.printStackTrace();
return(e.getMessage());
}
}
Nevertheless, the response I got is:
if (tz != tzless.readCookie('local_tz')) {
// no cookies!
document.write("<h1>Cookies are not enabled for this site</h1>");
document.write("<p>In order for Trendistic to work in Internet Explorer you need to enable cookies.</p>");
document.write("<ul><li>Please enable cookies for this site and refresh this page</li><li>Or try Trendistic in a different browser such as Safari, Firefox or Chrome.</li></ul>");
} else {
self.location.reload();
}
How can I enable cookies for fetching the HTML with the dynamical results.
Related
I want to automate REST API using selenium(java), is it possible ? if it have header and body part in json form
In Java you can use ApacheHttpClient for example lerned from https://www.mkyong.com/java/apache-httpclient-examples/
For instance a method in ApacheHttpClientPost could be like that:
public static String post(String tokenMobile, String method, String version, String body) throws Exception{
try {
HttpClient httpClient = HttpClientBuilder.create().build();
URIBuilder builder = new URIBuilder();
builder.setScheme("https").setHost(host).setPath(method)
.setParameter("", ""); //Params
URI uri = builder.build();
HttpGet httpget = new HttpGet(uri);
HttpPost postRequest = new HttpPost(httpget.getURI()); //Header
postRequest.addHeader("Content-Type", "application/json");
postRequest.addHeader("version", version);
postRequest.addHeader("Authorization", "Bearer "+tokenMobile);
StringEntity input = new StringEntity(body); //Body in json
input.setContentType("application/json");
postRequest.setEntity(input);
HttpResponse response = httpClient.execute(postRequest);
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String output;
while ((output = br.readLine()) != null) {
StringBuilder stringBuilder = new StringBuilder();
outputs = stringBuilder.append(output).toString();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return outputs;
}
Selenium is a tool which is designed for automation of UI or e2e test cases. You can integrate the Selenium test case with API test cases but that is always a bad idea.
Try something like Rest-Assured, Postman, HTTPClient if you want to automate the API test cases.
I'm building a wrapper for an API http://www.sptrans.com.br/desenvolvedores/APIOlhoVivo/Documentacao.aspx?1#docApi-autenticacao (it's in portuguese, but you get the idea).
I'm getting response code 404 when making a POST request and I have no idea why.
This is what is being printed:
Response Code : 404 {"Message":"No HTTP resource was found that
matches the request URI
'http://api.olhovivo.sptrans.com.br/v0/Login/Autenticar'."}
public static String executePost() {
CloseableHttpClient client = HttpClientBuilder.create().build();
String targetURL = "http://api.olhovivo.sptrans.com.br/v0/Login/Autenticar";
List<NameValuePair> urlParameters = new ArrayList<>();
urlParameters.add(new BasicNameValuePair("token","3de5ce998806e0c0750b1434e17454b6490ccf0a595f3884795da34460a7e7b3"));
try {
HttpPost post = new HttpPost(targetURL);
post.setEntity(new UrlEncodedFormEntity(urlParameters));
HttpResponse response = client.execute(post);
System.out.println("Response Code : "
+ response.getStatusLine().getStatusCode());
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) result.append(line);
System.out.println(result.toString());
return result.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
It looks to me from the API documentation (albeit, I can't read Portugese), that the token needs to be in the URL, not POSTed to it:
POST /Login/Autenticar?token={token}
I think you are POSTing a form to this endpoint.
You should try this:
String targetURL = "http://api.olhovivo.sptrans.com.br/v0/Login/Autenticar?token=3de5ce998806e0c0750b1434e17454b6490ccf0a595f3884795da34460a7e7b3";
And don't call post.setEntity(...).
I am programing a Browser using HTTPClient to access HTTPS on Android.
The following source code has implemented access HTTPS by HTTPClient.
// Proxy Host and Port
String proxyHost = "127.0.0.1";
int proxyPort = 10000;
// HTTPS Site
String strURL_https = "https://172.17.4.37:8443/apache.html";
URL url_https = null;
HttpClient httpClient = null;
HttpGet get = null;
try{
url_https = new URL(strURL_https);
MySSLSocketFactory sf = MySSLSocketFactory.getSocketFactory();
get = new HttpGet(strURL_https);
try {
HttpParams params = new BasicHttpParams();
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
// Add two scheme for Port 443 and 8443
httpClient = new DefaultHttpClient();
Scheme sch = new Scheme("https", sf, 443);
Scheme sch2 = new Scheme("https", sf, 8443);
httpClient.getConnectionManager().getSchemeRegistry().register(sch);
httpClient.getConnectionManager().getSchemeRegistry().register(sch2);
// Set Proxy
httpClient.getParams().setParameter(ConnRouteParams.DEFAULT_PROXY, new HttpHost(proxyHost, proxyPort));
} catch (Exception e) {
e.printStackTrace();
}
HttpContext context = new BasicHttpContext();
// Execute GET Method
HttpResponse response = httpClient.execute(get, context);
HttpEntity entity = response.getEntity();
String charSet = null;
String contentType = null;
StringBuffer sb = null;
if(entity!= null){
charSet=EntityUtils.getContentCharSet(entity);
contentType=entity.getContentType().getValue();
InputStream is = entity.getContent();
BufferedReader br = new BufferedReader(new InputStreamReader(is, "utf8"));
String text = null;
String line = br.readLine();
sb = new StringBuffer();
while(line != null) {
sb.append(line+"\r\n");
line = br.readLine();
}
}
// Load Data on Webview
webview.loadDataWithBaseURL(strURL_https, sb.toString(), "text/html", charSet, null);
But when do httpClient.execute(get, context), my proxy server will receive HTTP CONNECT Method as the following:
CONNECT 172.17.4.37:8443 HTTP/1.1
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)
Host: 172.17.4.37:8443
My proxy server send the HTTP Method to VPN, but cannot get any response.
In my opinion, I should get like HTTP/1.0 200 Connection Established.
My question is how to do HTTP CONNECT Method by HTTPClient without proxy.
After create connection established, set proxy to HTTPClient.
I am looking forward your answer.
Thank you.
In my Android application I'm using a web view to access some web mapping data provided by a server. The server requires some HTTP form based authentication to allow access to those data. Due to the fact that the site doesn't have a mobile version, displaying the login page (or any other pages) looks pretty bad . Unfortunately the site is hardly into my reach so I've thought of the following approach:
use a native user interface to collect the username and password
thought a Http post send those information to the server
after the response is received get the cookies the server is sending
set the cookies to the the web view
try to finally access the desired data
For now I'm just trying to pass the login phase.
Is this a viable solution , or is just plain wrong and I should try something else ?
For completeness I post the code below
A. The authentication part
private String authenticate() throws Exception
{
// Create a new HttpClient and Post Header
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://mySite/login_form");
HttpResponse response = null;
BufferedReader in = null;
String resultContent = null;
try
{
// Add data
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("came_from", ""));
nameValuePairs.add(new BasicNameValuePair("form.submitted", "1"));
nameValuePairs.add(new BasicNameValuePair("js_enabled", "0"));
nameValuePairs.add(new BasicNameValuePair("cookies_enabled", ""));
nameValuePairs.add(new BasicNameValuePair("login_name", ""));
nameValuePairs.add(new BasicNameValuePair("pwd_empty", "0"));
nameValuePairs.add(new BasicNameValuePair("name", "username"));
nameValuePairs.add(new BasicNameValuePair("password", "password"));
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
// Create a local instance of cookie store
CookieStore cookieStore = new BasicCookieStore();
// Create local HTTP context
HttpContext localContext = new BasicHttpContext();
// Bind custom cookie store to the local context
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
// Execute HTTP Post Request
response = httpclient.execute(httppost,localContext);
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String line = "";
String NL = System.getProperty("line.separator");
while ((line = in.readLine()) != null)
{
sb.append(line + NL);
}
in.close();
resultContent = sb.toString();
Log.i("mytag","result :"+resultContent);
cookies = new java.util.ArrayList();
cookies = cookieStore.getCookies();
}
catch (ClientProtocolException e)
{
Log.i("mytag","Client protocol exception");
}
catch (IOException e)
{
Log.i("mytag","IOException");
}
catch(Exception e)
{
Log.i("mytag","Exception");
Log.i("mytag",e.toString());
}
return resultContent;
}
B. Setting the cookies and loading the desired page
private void init()
{
CookieSyncManager.createInstance(this);
CookieManager cookieMan= CookieManager.getInstance();
cookieMan.setAcceptCookie(true);
cookies = StartupActivity.listAfter;
if(cookies != null)
{
for (int i = 0; i<cookies.size(); i++)
{
Cookie cookie = cookies.get(i);
cookieMan.setCookie("cookie.getDomain()",cookie.getValue());
}
}
CookieSyncManager.getInstance().sync();
webView = (WebView)findViewById(R.id.web_view);
webView.getSettings().setJavaScriptEnabled(true);
webView.getSettings().setBuiltInZoomControls(true);
webView.setWebViewClient(new HelloWebViewClient());
}
protected void onResume()
{
super.onResume();
// test if the we logged in
webView.loadUrl("mySite/myDesiredFeature");
}
The results of loading that page is that the login_page form is displayed
1) Try first to make HttpGet request, to get cookies, then perform HttpPost. I think this way you should not add cookies manually.
Use one HttpClient to do this.
2) Instead of
in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
StringBuffer sb = new StringBuffer("");
String line = "";
String NL = System.getProperty("line.separator");
while ((line = in.readLine()) != null)
{
sb.append(line + NL);
}
in.close();
resultContent = sb.toString();
use
EntityUtils.toString(response.getEntity()).
I've written some code for my Android device to login to a web site over HTTPS and parse some data out of the resulting pages. An HttpGet happens first to get some info needed for login, then an HttpPost to do the actual login process.
The code below works great in a Java project within Eclipse which has the following JAR files on the build path: httpcore-4.1-beta2.jar, httpclient-4.1-alpha2.jar, httpmime-4.1-alpha2.jar, and commons-logging-1.1.1.jar.
public static MyBean gatherData(String username, String password) {
MyBean myBean = new MyBean();
try {
HttpResponse response = doHttpGet(URL_PAGE_LOGIN, null, null);
System.out.println("Got login page");
String content = EntityUtils.toString(response.getEntity());
String token = ContentParser.getToken(content);
String cookie = getCookie(response);
System.out.println("Performing login");
System.out.println("token = "+token +" || cookie = "+cookie);
response = doLoginPost(username,password,cookie, token);
int respCode = response.getStatusLine().getStatusCode();
if (respCode != 302) {
System.out.println("ERROR: not a 302 redirect!: code is \""+ respCode+"\"");
if (respCode == 200) {
System.out.println(getHeaders(response));
System.out.println(EntityUtils.toString(response.getEntity()).substring(0, 500));
}
} else {
System.out.println("Logged in OK, loading account home");
// redirect handler and rest of parse removed
}
}catch (Exception e) {
System.out.println("ERROR in gatherdata: "+e.toString());
e.printStackTrace();
}
return myBean;
}
private static HttpResponse doHttpGet(String url, String cookie, String referrer) {
try {
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
client.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, "UTF-8");
HttpGet httpGet = new HttpGet(url);
httpGet.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
httpGet.setHeader(HEADER_USER_AGENT,HEADER_USER_AGENT_VALUE);
if (referrer != null && !referrer.equals("")) httpGet.setHeader(HEADER_REFERER,referrer);
if (cookie != null && !cookie.equals("")) httpGet.setHeader(HEADER_COOKIE,cookie);
return client.execute(httpGet);
} catch (Exception e) {
e.printStackTrace();
throw new ConnectException("Failed to read content from response");
}
}
private static HttpResponse doLoginPost(String username, String password, String cookie, String token) throws ClientProtocolException, IOException {
try {
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
client.getParams().setParameter(CoreProtocolPNames.HTTP_CONTENT_CHARSET, "UTF-8");
HttpPost post = new HttpPost(URL_LOGIN_SUBMIT);
post.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
post.setHeader(HEADER_USER_AGENT,HEADER_USER_AGENT_VALUE);
post.setHeader(HEADER_REFERER, URL_PAGE_LOGIN);
post.setHeader(HEADER_COOKIE, cookie);
post.setHeader("Content-Type","application/x-www-form-urlencoded");
List<NameValuePair> formParams = new ArrayList<NameValuePair>();
formParams.add(new BasicNameValuePair("org.apache.struts.taglib.html.TOKEN", token));
formParams.add(new BasicNameValuePair("showLogin", "true"));
formParams.add(new BasicNameValuePair("upgrade", ""));
formParams.add(new BasicNameValuePair("username", username));
formParams.add(new BasicNameValuePair("password", password));
formParams.add(new BasicNameValuePair("submit", "Secure+Log+in"));
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(formParams,HTTP.UTF_8);
post.setEntity(entity);
return client.execute(post);
} catch (Exception e) {
e.printStackTrace();
throw new ConnectException("ERROR in doLoginPost(): "+e.getMessage());
}
}
The server (which is not under my control) returns a 302 redirect when the login was successful, and 200 if it fails and re-loads the login page. When run with the above JAR files I get the 302 redirect, however if I run the exact same code from an Android project with the 1.6 Android JAR file on the build path I get the 200 response from the server. I get the same 200 response when running the code on my 2.2 device.
My android application has internet permissions, and the HttpGet works fine. I'm assuming that the problem lies in the fact that HttpPost (or some other class) is different in some significant way between the Android JAR version and the newer Apache versions.
I've tried adding the Apache libraries to the build path of the Android project, but due to the duplicate classes I get messages like: INFO/dalvikvm(390): DexOpt: not resolving ambiguous class 'Lorg/apache/http/impl/client/DefaultHttpClient;' in the log. I've also tried using a MultipartEntity instead of the UrlEncodedFormEntity but I get the same 200 result.
So, I have a few questions:
Can I force the code running under Android to use the newer Apache libraries in preference to the Android versions?
If not, does anyone have any ideas how can I alter my code so that it works with the Android JAR file?
Are there any other, totally different approaches to doing an HttpPost in Android?
Any other ideas?
I've read a lot of posts and code, but I'm not getting anywhere.
I have now given up on getting the HttpClient route to give the expected response from the server when run on Android. Instead I rewrote the doPost method above to use an HttpsURLConnection instead. Here's the new (working) version in the hope that it's useful to someone.
private static LoginBean altPost(String username, String password, String cookie, String token){
LoginBean loginBean = new LoginBean();
HttpsURLConnection urlc = null;
OutputStreamWriter out = null;
DataOutputStream dataout = null;
BufferedReader in = null;
try {
URL url = new URL(URL_LOGIN_SUBMIT);
urlc = (HttpsURLConnection) url.openConnection();
urlc.setRequestMethod("POST");
urlc.setDoOutput(true);
urlc.setDoInput(true);
urlc.setUseCaches(false);
urlc.setAllowUserInteraction(false);
urlc.setRequestProperty(HEADER_USER_AGENT, HEADER_USER_AGENT_VALUE_FF);
urlc.setRequestProperty("Cookie", cookie);
urlc.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
String output = "org.apache.struts.taglib.html.TOKEN="+ URLEncoder.encode(token, HTTP.UTF_8)
+"&showLogin=true&upgrade=&username="+ URLEncoder.encode(username, HTTP.UTF_8)
+"&password="+ URLEncoder.encode(password, HTTP.UTF_8)+"&submit="
+URLEncoder.encode("Secure+Log+in", HTTP.UTF_8);
dataout = new DataOutputStream(urlc.getOutputStream());
// perform POST operation
dataout.writeBytes(output);
// get response info
loginBean.setResponseCode(urlc.getResponseCode());
// get required headers
String headerName = null;
StringBuffer newCookie = new StringBuffer(100);
String redirectLocation = "";
for (int i=1; (headerName = urlc.getHeaderField(i)) != null;i++) {
if (headerName.indexOf(COOKIE_VALUE_SESSION) > -1) {
if (newCookie.length() > 0) {newCookie.append("; ");}
newCookie.append(headerName);
}
if (headerName.indexOf(COOKIE_VALUE_AUTH) > -1) {
if (newCookie.length() > 0) {newCookie.append("; ");}
newCookie.append(headerName);
}
if (headerName.indexOf("https://") > -1) {
redirectLocation = headerName;
}
}
loginBean.setCookie(newCookie.toString());
loginBean.setRedirectUrl(redirectLocation);
in = new BufferedReader(new InputStreamReader(urlc.getInputStream()),8096);
String response;
// write html to System.out for debug
while ((response = in.readLine()) != null) {
System.out.println(response);
}
in.close();
} catch (ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (out != null) {
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (in != null) {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return loginBean;
}
I still have no idea why the HttpClient way didn't work properly.
To avoid the collisions, use this JAR file for httpclient:
httplib
And this post would also be very useful:
An answer to Stack Overflow question Apache HTTP client or URLConnection
Is it possible that this website does user-agent detection and actually returns different results because it's Android? Given that 200 implies success, why must it give a 302 instead of a 200? Have you printed out the result that you get when it returns a 200, and does it give any additional information?
Check the RedirectHandler, override the default one and do some logging in it, I had problems with that when going to Android...