Basically i just want to get cookies from server. Here's simple code, but i guess it's not enough to get every cookie like e.g sessionID etc.
import java.net.*;
import java.util.*;
public class Cookie {
public static void main(String[] args) {
try {
URL url = new URL("http://www.google.com/");
URLConnection conn = url.openConnection();
System.out.println(con.getHeaderField("Set-Cookie"));
} catch (Exception e) { }
}
}
Result shows that i get only one cookie:
NID=61=nNSsAl4g7DfxqHE7t__ghfoCc_J-RmY7PaTNoPOd_khLdvvuqI-fnteHgnQXMzmxj_C5HdMozcGfOTt8PvePLKpbDdUlzdVZiRKwNpQIej6_T69jt9C7Oi6E4F-gWPHD; expires=Mon, 14-Jan-2013 17:16:40 GMT; path=/; domain=.google.pl; HttpOnly
Firefox gets NID and PREF, so it's not working.
Is there any way to get PREF cookie?
Can anyone tell me is this proper way(code below) to send POST.
import java.io.*;
import java.net.URL;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import javax.net.ssl.HttpsURLConnection;
public class Cookie_Example {
public static void main(String[] args){
try{
String https_url = "https://ssl.filmweb.pl/j_login";
URL url = new URL(https_url);
String post = "_login_redirect_url=http%253A%252F%252Fwww.filmweb.pl%252F&j_username=test.filmweb%40gmail.com&j_password=test.filmweb&_rememberMe=on&pass=zaloguj";
HttpsURLConnection con = (HttpsURLConnection)url.openConnection();
con.setInstanceFollowRedirects(false);
con.setDoInput(true);
con.setDoOutput(true);
con.setRequestMethod("GET");
con.setRequestProperty("Host", "ssl.filmweb.pl");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1");
con.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Accept-Language","pl,en-us;q=0.7,en;q=0.3");
con.setRequestProperty("Accept-Encoding","gzip, deflate");
con.setRequestProperty("Connection","keep-alive");
con.setRequestProperty("Referer","https://ssl.filmweb.pl/login");
con.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
con.setRequestProperty("Content-Length",post.length()+"");
DataOutput output = new DataOutputStream(new BufferedOutputStream(con.getOutputStream()));
output.write(post.getBytes());
Map<String, List<String>> hf = con.getHeaderFields();
Iterator it = hf.entrySet().iterator();
while (it.hasNext()) {
Map.Entry pairs = (Map.Entry)it.next();
System.out.println(pairs.getKey() + " = " + pairs.getValue());
}
} catch (Exception e) {
}
}
}
Result is empty cookie,
Set-Cookie = [_artuser_rememberMe=;Path=/;Domain=filmweb.pl;Expires=Thu, 01-Jan-1970 00:00:00 GMT]
http://img72.imageshack.us/img72/1134/testgof.jpg
Screen from Tamper Data
It means this code also doesn't work. How Can I get __utma, __utmz etc. cookie?
Can anyone help me? Which books will be helpful?
I think that with your statement, con.getHeaderField("Set-Cookie")" you only retrieve one header. But maybe the server sends you two cookies, in two different headers.
I use another statement for retrieve all headers of cookies:
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
To see what you send to server, and what the server sends to you, I recommend use a proxy between your program (or web navigator) and the server. Maybe Charles proxy can help you.
Related
I want to call GET and POST API in java without using any framework. I need to use basic authentication. Can anybody help me with some tutorial link. In google I found code only in spring framework, But I am not using Spring. I am looking for code to call API with basic authentication.
I have to add new url with authentication in the below code. What modification is required if API is secured with basic auth and it is POST method. I am new to java so not much aware.
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.URL;
import java.net.URLConnection;
public class NetClientGet {
public static void main(String[] args) {
try
{
System.out.println("Inside the main function");
URL weburl=new URL("http://dummy.restapiexample.com/api/v1/employees");
HttpURLConnection conn = (HttpURLConnection) weburl.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
System.out.println("Output is: "+conn.getResponseCode());
System.out.println("Output is: ");
System.setProperty("http.proxyHost", null);
//conn.setConnectTimeout(60000);
if(conn.getResponseCode()!=200)
{
System.out.println(conn.getResponseCode());
throw new RuntimeException("Failed : HTTP Error Code: "+conn.getResponseCode());
}
System.out.println("After the 2 call ");
InputStreamReader in=new InputStreamReader(conn.getInputStream());
BufferedReader br =new BufferedReader(in);
String output;
while((output=br.readLine())!=null)
{
System.out.println(output);
}
conn.disconnect();
}
catch(Exception e)
{
System.out.println(e.getMessage());
}
}
}
Basic Authentication
See the RFC #2617 section 2: Basic Authentication Scheme
Add Authentication header into the request. Here's an example:
String username = "john";
String password = "pass";
// ...
URL weburl=new URL("http://dummy.restapiexample.com/api/v1/employees");
HttpURLConnection conn = (HttpURLConnection) weburl.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
// snippet begins
conn.setRequestProperty("Authorization",
"Basic " + Base64.getEncoder().encodeToString(
(username + ":" + password).getBytes()
)
);
// snippet ends
System.out.println("Output is: "+conn.getResponseCode());
POST Method
See this answer for more information about using POST method with HttpURLConnection.
I'm trying to make a request with java on this OData2 API => https://scihub.copernicus.eu/dhus/odata/v1/ for a project. But I can't without authentication. I have personal logs, as a user I don't have any problems. When I tried with java it gave an error 401.
I try this:
String auth = user + ":" + password;
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
String basicAuth = "Basic " + new String(new Base64().encode(auth.getBytes()));
connection.setRequestProperty("Authorization", basicAuth);
connection.connect();
int responseCode = connection.getResponseCode();
System.out.println(responseCode);
System.out.println(url.toString());
But it doesn't works. When i print the responseCode i have a 400 error and i try also another code it was a 401 error.
With PostMan i only need the BasicAuth to have an access and it works.
And i'm using Olingo2.
I'm new on java web and i don't have any idea.
In the first step i only want to have the authentication.
And then doing queries.
Thank you!
For the 401 case there is something wrong with the authorization,otherwise the general approach is right.
For the 400 case, browsers and tools like postman will automatically send additional headers, which the code will be missing. I reused the about code and passed an additional header Accept : application/xml and was able to retrieve the response. Below is the working code. Cheers!
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
public class Test1 {
public static void main(String[] args) throws IOException {
String user = "ENTER YOUR USERNAME";
String password = "ENTER YOUR PASSWORD";
String auth = user + ":" + password;
URL url = new URL("https://scihub.copernicus.eu/dhus/odata/v1/");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setRequestProperty("Accept", "application/xml");
String basicAuth = "Basic " + Base64.getEncoder().encodeToString((auth).getBytes(StandardCharsets.UTF_8)); // Java
connection.setRequestProperty("Authorization", basicAuth);
connection.connect();
System.out.println(connection.getResponseCode());
System.out.println(connection.getContent());
}
}
I'm trying to establish a Connection via HTTPS. I also set the "Authorization" property in the Request Header to Basic and provide an encoded auth string accordingly.
I checked with the Firefox Plugin HttpRequester and everythign works fine, which means I entered the url, choose "GET" as request method, add the Authorization to the header and after pressing submit I get back some xml which only a properly authorized user should get.
Unfortunately I can neither provide you with the actual auth info nor the real url in the SSCCE. However, I can tell you, that the Auth seems to work, since I get a 200 response. I also changed the Auth to a wrong value and get a "401 Authorization Required" response then.
It actually seems like the "?myparam=xyz" is somehow cut off, because when I remove this parameter from the url and test with Firefox HttpRequester again I get the same response as in Java.
Unfortunately I have no access to "theirdomain.com", so I don't know what's happending on the server side. But since it works with the Firefox HttpRequester, it should also work with Java.
What could be the reason? Thanks for your help!
EDIT:
I changed the url to "https://www.google.com/search?q=foo" and commented this line:
//con.setRequestProperty("Authorization", auth);
I can see from the returned string, that google received the "foo". So apparently the combination of Authorization and get parameter seems to be the problem, since both separately work fine.
SSCCE:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class HttpRequest
{
/**
* #param args
*/
public static void main(final String[] args)
{
System.out.println("start request");
final String urlString = "https://theirdomain.com/foo/bar/bob?myparam=xyz";
final String auth = "Basic XyzxYzxYZxYzxyzXYzxY==";
HttpsURLConnection con;
try
{
final URL url = new URL(urlString);
con = (HttpsURLConnection) url.openConnection();
con.setRequestProperty("Authorization", auth);
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
con.setRequestMethod("GET");
// con.setDoOutput(true);
con.connect();
final int responseCode = con.getResponseCode();
if (responseCode != 200)
System.out.println("Server responded with code " + responseCode + " " + con.getResponseMessage());
else
{
System.out.println("Starting to read...");
final InputStream inStream = con.getInputStream();
final ByteArrayOutputStream baos = new ByteArrayOutputStream();
int c;
while (inStream != null && (c = inStream.read()) != -1)
{
baos.write(c);
}
System.out.println(new String(baos.toByteArray()));
}
}
catch (final IOException e)
{
System.out.println("could not open an HTTP connection to url: " + urlString);
e.printStackTrace();
}
finally
{
System.out.println("end request");
}
}
}
Have you tried adding
con.setRequestProperty("myparam", "xyz"); to your code?
I am trying to do a PHP GET request to a website:
The problem is that this website will only process my request if I attach Cookie information to the header of the request.
Or in picture terms, if I disable cookies in my browser, I get this:
Which means the website recognises that it's my first time 'visiting' the site.
Problem is, that if I now use the search bar on the top right, it will not process this request:
it will just show the same (general) screen.
E.g.: if I have cookies disabled and I search for "AAPL", it will not show any results.
Now if I have cookies enabled, the request is handled just fine:
And so the "AAPL" results are shown.
You can try this yourself as well:
With cookies enabled, visit http://www.pennystocktweets.com/user_posts/feeds?cat=search&lptyp=prep&usrstk=AAPL
With cookies disabled, visit the link again: http://www.pennystocktweets.com/user_posts/feeds?cat=search&lptyp=prep&usrstk=AAPL
Now compare the responses, only the first one is correct.
This means that the website only works after the client has downloaded a cookie, and then has made another (new) GET request to the server with this Cookie information attached.
(Does this imply that the website needs a session-cookie to function correctly?)
Now what I'm trying to do is imitate the request with Apache HttpClient like so:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.CookieHandler;
import java.net.CookieManager;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Date;
import java.util.List;
import java.util.StringTokenizer;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
public class downloadTweets {
private String cookies;
private HttpClient client = new DefaultHttpClient();
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String[] args) throws Exception {
String ticker = "AAPL";
String lptyp = "prep";
int opid = 0;
int lpid = 0;
downloadTweets test = new downloadTweets();
String url = test.constructURL(ticker, lptyp, opid, lpid);
// make sure cookies is turn on
CookieHandler.setDefault(new CookieManager());
downloadTweets http = new downloadTweets();
String page = http.GetPageContent(url, ticker);
System.out.println(page);
}
public String constructURL(String ticker, String lptyp, int opid, int lpid)
{
String link = "http://www.pennystocktweets.com/user_posts/feeds?cat=search" +
"&lptyp=" + lptyp +
"&usrstk=" + ticker;
if (opid != 0)
{
link = link +
"&opid=" + opid +
"&lpid=" + lpid;
}
return link;
}
private String GetPageContent(String url, String ticker) throws Exception {
HttpGet request = new HttpGet(url);
String RefererLink = "http://www.pennystocktweets.com/search/post/" + ticker.toUpperCase();
request.setHeader("Host", "www.pennystocktweets.com");
request.setHeader("Connection", "Keep-alive");
request.setHeader("Accept", "*/*");
request.setHeader("X-Requested-With", "XMLHttpRequest");
request.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
request.setHeader("Referer", RefererLink);
request.setHeader("Accept-Language", "nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4,fr;q=0.2");
HttpResponse response = client.execute(request);
int responseCode = response.getStatusLine().getStatusCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
// set cookies
setCookies(response.getFirstHeader("Set-Cookie") == null ? "" :
response.getFirstHeader("Set-Cookie").toString());
return result.toString();
}
public String getCookies() {
return cookies;
}
public void setCookies(String cookies) {
this.cookies = cookies;
}
}
Now, the same thing holds: if I attach (my) cookie information, the response works just fine, and if I don't the response doesn't work.
But I don't know how to get the cookie information and then use it in a new GET request.
So my question is:
How can I make 2 requests to a website such that:
On the first GET request, I get cookie information from the website and store this in my Java program
On the second GET request, I use the stored cookie information (as a Header) to make a new request.
Note: I don't know if the cookie is a normal cookie or a session cookie but I suspect it's a session-cookie!
All help is greatly appreciated!
As the documents of Apache commons httpclient states in the HttpClient Cookie handling part:
HttpClient supports automatic management of cookies, including allowing the server to set cookies and automatically return them to the server when required. It is also possible to manually set cookies to be sent to the server.
Whenever the http client receives cookies they are persisted into HttpState and added automatically to the new request. This is the default behavior.
In the following example code, we can see the cookies returned by two GET requests. We can't see directly the cookies sent to the server, but we can use a tool such as a protocol/net sniffer or ngrep to see the data transmitted over the network:
import java.io.IOException;
import org.apache.commons.httpclient.Cookie;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.cookie.CookiePolicy;
import org.apache.commons.httpclient.methods.GetMethod;
public class HttpTest {
public static void main(String[] args) throws HttpException, IOException {
String url = "http://www.whatarecookies.com/cookietest.asp";
HttpClient client = new HttpClient();
client.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
HttpMethod method = new GetMethod(url);
int res = client.executeMethod(method);
System.out.println("Result: " + res);
printCookies(client.getState());
method = new GetMethod(url);
res = client.executeMethod(method);
System.out.println("Result: " + res);
printCookies(client.getState());
}
public static void printCookies(HttpState state){
System.out.println("Cookies:");
Cookie[] cookies = state.getCookies();
for (Cookie cookie : cookies){
System.out.println(" " + cookie.getName() + ": " + cookie.getValue());
}
}
}
This is the output:
Result: 200
Cookies:
active_template::468: %2Fresponsive%2Fthree_column_inner_ad3b74de5a1c2f311bee7bca5c368aaa4e:b326b5062b2f0e69046810717534cb09
Result: 200
Cookies:
active_template::468: %2Fresponsive%2Fthree_column_inner_ad%2C+3b74de5a1c2f311bee7bca5c368aaa4e%3Db326b5062b2f0e69046810717534cb09
3b74de5a1c2f311bee7bca5c368aaa4e: b326b5062b2f0e69046810717534cb09
Here is an excerpt of ngrep:
MacBook$ sudo ngrep -W byline -d en0 "" host www.whatarecookies.com
interface: en0 (192.168.11.0/255.255.255.0)
filter: (ip) and ( dst host www.whatarecookies.com )
#####
T 192.168.11.70:56267 -> 54.228.218.117:80 [AP]
GET /cookietest.asp HTTP/1.1.
User-Agent: Jakarta Commons-HttpClient/3.1.
Host: www.whatarecookies.com.
.
####
T 54.228.218.117:80 -> 192.168.11.70:56267 [A]
HTTP/1.1 200 OK.
Server: nginx/1.4.0.
Date: Wed, 27 Nov 2013 10:22:14 GMT.
Content-Type: text/html; charset=iso-8859-1.
Content-Length: 36397.
Connection: keep-alive.
Vary: Accept-Encoding.
Vary: Cookie,Host,Accept-Encoding.
Set-Cookie: active_template::468=%2Fresponsive%2Fthree_column_inner_ad; expires=Fri, 29-Nov-2013 10:22:01 GMT; path=/; domain=whatarecookies.com; httponly.
Set-Cookie: 3b74de5a1c2f311bee7bca5c368aaa4e=b326b5062b2f0e69046810717534cb09; expires=Thu, 27-Nov-2014 10:22:01 GMT.
X-Middleton-Response: 200.
Cache-Control: max-age=0, no-cache.
X-Mod-Pagespeed: 1.7.30.1-3609.
.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/1998/REC-html40-19980424/loose.dtd">
...
##
T 192.168.11.70:56267 -> 54.228.218.117:80 [AP]
GET /cookietest.asp HTTP/1.1.
User-Agent: Jakarta Commons-HttpClient/3.1.
Host: www.whatarecookies.com.
Cookie: active_template::468=%2Fresponsive%2Fthree_column_inner_ad.
Cookie: 3b74de5a1c2f311bee7bca5c368aaa4e=b326b5062b2f0e69046810717534cb09.
.
##
T 54.228.218.117:80 -> 192.168.11.70:56267 [A]
HTTP/1.1 200 OK.
Server: nginx/1.4.0.
Date: Wed, 27 Nov 2013 10:22:18 GMT.
Content-Type: text/html; charset=iso-8859-1.
Content-Length: 54474.
Connection: keep-alive.
Vary: Accept-Encoding.
Vary: Cookie,Host,Accept-Encoding.
Set-Cookie: active_template::468=%2Fresponsive%2Fthree_column_inner_ad%2C+3b74de5a1c2f311bee7bca5c368aaa4e%3Db326b5062b2f0e69046810717534cb09; expires=Fri, 29-Nov-2013 10:22:05 GMT; path=/; domain=whatarecookies.com; httponly.
Set-Cookie: 3b74de5a1c2f311bee7bca5c368aaa4e=b326b5062b2f0e69046810717534cb09; expires=Thu, 27-Nov-2014 10:22:05 GMT.
X-Middleton-Response: 200.
Cache-Control: max-age=0, no-cache.
X-Mod-Pagespeed: 1.7.30.1-3609.
.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/1998/REC-html40-19980424/loose.dtd">
...
I am trying to Connect to url using URLConnection, in java with username and password.
This is the following code I am using:
package com.nivi.org.client;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import com.sun.jersey.core.util.Base64;
public class GetURLContent {
public static void main(String[] args) {
URL url;
try {
url = new URL("http://sampleurl.co.uk");
URLConnection conn = url.openConnection();
String username = "username";
String password = "password";
String Token = "zzzzzzzzzzzzzzzzzzzzz";
System.setProperty("http.proxyHost", "proxyhostname");
System.setProperty("http.proxyPort", "8080");
String userpass = username + ":" + password;
String basicAuth = "Basic " + new String(new Base64().encode(userpass.getBytes()));
conn.setRequestProperty ("Authorization", basicAuth);
conn.setRequestProperty ("Token", Token);
BufferedReader br = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
System.out.println("br............."+br.toString());
br.close();
System.out.println("Done");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=======================================
I am getting the following error
java.io.IOException: Server returned HTTP response code: 403 for URL:
http://sampleurl.co.uk
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at com.nivi.org.client.GetURLContent.main(GetURLContent.java:63)
=======================================
First of all my question is
Are these following lines in the code are correct?
conn.setRequestProperty ("Authorization", basicAuth);
conn.setRequestProperty ("Token", Token);
Are these following lines in the code are correct?
It depends on how the actual website you are accessing implements its user authentication.
What you appear to be doing is a combination of HTTP Basic Authentication (i.e. the Authorization header), and something involving a non-standard header called Token. This may be sufficient, if the website supports Basic Authentication.
You should probably read the website's programmer documentation of how their web APIs work. If there isn't any such documentation available to you, use your browsers web development tools to identify the mechanism that the site is using when you log in via a web browser ... and attempt to get your client to behave the same way.
One thing to note is that sending a Basic authorization response in an HTTP request is insecure. Use an HTTPS request if that is an option.