I'm using the following code on the client side to take response of a protected website, which works pretty fine whenever there is no authentication request from server side:
final String targetURL = "http://************";
final String username = "********";
final String password = "******";
final int connectTimeout = 30000;
final String username_password = username + ":" + password;
final byte[] usernamepassword = (byte[])username_password.getBytes();
final String encoded_username_password = java.util.Base64.getEncoder().encodeToString(usernamepassword);
final URL url = new URL(targetURL);
final URLConnection connection = url.openConnection();
connection.setConnectTimeout(connectTimeout);
connection.setRequestProperty("Authorization", "Basic " + encoded_username_password);
connection.setRequestProperty("Accept", "text/html");
// response headers
final Map<String, List<String>> headerFields = connection.getHeaderFields();
for (final Map.Entry<String, List<String>> entry : headerFields.entrySet()) {
System.out.println("Key: " + entry.getKey() + ", Value: " + entry.getValue());
}
However, when I try to connect with a specific (protected) webserver, the connection is refused with the 401 http status code.
Key: null, Value: [HTTP/1.1 401 Unauthorized]
Key: WWW-Authenticate, Value: [Digest realm="Login to ********", qop="auth", nonce="*******", opaque=""]
Key: Connection, Value: [close] Key: Content-Length, Value: [0]
Is it possible to infer from the above logged response what I'm missing ?
I mean, the snippet above is a well known Java basic-authentication template, there are many examples everywhere with the same structure, so the issue seems likely related to the parameters issued as argument of the setRequestProperty() method.
This is what I took on request header by using Mozilla >> Web Developer >> Debug (where some fields are hidden by me):
Host: xxx.xxx.xxx.xxx
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
Authorization: Digest username="", realm="Login to xxxxxxx", nonce="xxx-xxx-xxx-xxx-xxx", uri="xxxxxxx", response="xxxxx", qop=auth, nc=00000001, cnonce="b2b43c8d9631354d"
I am not sure to understand your question. My guess is you would like to filter the request based on the parameters and url so you can chhose to either serve content or reject and return 401.
In this case your question should be: How to filter which http requests requires authentication.
Depending on what framework your are using, in spring-boot, there is something called WebSecurityConfigurerAdapter. It basically helps you with your problem. You can filter all the requests coming to your server.
It Simply because you are not authenticated to their server,
you need the use that protected server authentication, if you want to enable yourself to access their resource,
maybe u can check their website, or docs to get some some api key or
some token (basic, bearer or whatever token)
then follow their guideline where to put that authentication in your request (usually if token will be in request payload, if api key or basic auth will be in request header)
hope this helps
Related
I am sending a JSON request via http post. Backend requires me to send two headers, namely - Authorization and Content-Type. Then a raw JSON request as body. I send the request via Post method via Postman tool and the message gets through successfully. But when I send the same request with same headers via code, I get a bad request - Error.
I minimized the scope of testing to check.
I have sent only the two headers via Postman and here's the response:
Headers
Authorization: "bearer " + token. The structure is "bearer" keyword + space + Okta token.
Content-Type: application/json
Response:
Headers
Content-Type →application/json; charset=utf-8
Strict-Transport-Security →max-age=31536000
X-Content-Type-Options →nosniff
Request-Context →appId=cid-v1:dc220dda-e515-4dd4-b324-c078a1b8fac4
Date →Sun, 26 May 2019 13:23:01 GMT
Content-Length →29
Body
{
"Message": "No Request Body"
}
But when I send the same thing via code, I get a Bad-Request Error. Please help :(
HttpClient httpclient1;
HttpPost httppost1;
//ArrayList<NameValuePair> postParameters1 = null;
httpclient1 = HttpClientBuilder.create().build
httppost1 = new HttpPost(Constants.URL);
// httppost1.setEntity(new UrlEncodedFormEntity(postParameters1,"UTF-8"));
// httppost1.setEntity(new StringEntity(request,"UTF-8"));
httppost1.setHeader("Authorization", token);
httppost1.setHeader("Content-Type", "application/json");
// httppost1.setHeader("Content-Length", strlength);
// httppost1.setHeader("Accept", "application/json");
// StringEntity entity = new StringEntity(request,ContentType.APPLICATION_JSON);
// httppost1.setEntity(entity);
HttpResponse response1 = httpclient1.execute(httppost1);
InputStream is = response1.getEntity().getContent();
HttpEntity entity1 = httppost1.getEntity();
Header[] headers = httppost1.getAllHeaders();
System.out.println(headers.length);
for(int i=0; i<headers.length;i++)
{
System.out.println(headers[i]);
}
String result = convertInputStreamToStringCommonIO(is);
System.out.println();
System.out.println(result);
int code1 = response1.getStatusLine().getStatusCode();
System.out.println("Code: "+code1+"");
String reason1 = response1.getStatusLine().getReasonPhrase();
System.out.println("Description: "+reason1 +"");
Temporary Request Headers from Postman:
UserAgent: PostmanRuntime/7.13.0
Accept: /
Cache-Contro: no-cache
Postman-Token: *****
Host: *****
accept-encoding: gzip,dflate
content-length:
Connection: keep-alive
Here's the error I am getting:
{"errorCode":"E0000021","errorSummary":"Bad request. Accept and/or Content-Type headers likely do not match supported values.","errorLink":"E0000021","errorId":"oae5BpTGp_5RjaFtGn_Zm_mhw","errorCauses":[]}
Code: 400
Description: Bad Request
Any idea why it's failing via code? Thanks.
I would like to login into a website and keep the cookies. I have issues with the login part using Jsoup;
My code ;
Connection.Response loginForm = Jsoup.connect("URL" + "/login/").cookies(cookies).method(Connection.Method.GET).execute();
formData.put("username", "#######");
formData.put("pwd", "########");
formData.put("hidden","69");
formData.put("token", loginForm.parse().select("input#token").first().attr("value"));
Connection.Response homePage = Jsoup.connect("URL" + "/login/")
.referrer("URL" + "/login/")
.followRedirects(true)
.cookies(cookies)
.data(formData)
.method(Connection.Method.POST)
.execute();
cookies_2.putAll(homePage.cookies()); // save the cookies, this will be passed on to next request
If i go to the login page and use the developper tool there are ;
Google chrome developper tool
Google chrome developper tool
Edit 1;
The problem now is I get homepage of the website but without the section of login, I keep the "Connect or register" button.
I output the cookies_2 and it is the same that the cookies in the Chrome logs ;
Jsoup cookies_2 output
Chrome cookies when login
Now, what I don't understand is why, I don't get loggin if I have the right cookies ?
Edit 2;
I chanded my code with the final solution and it work !
Thanks for your help !
Few remarks:
You're getting HTTP status 302. It may be a good idea to use Jsoup.connect(...).followRedirects(true)
Some servers check the site you're coming from so it's recommened to set referrer header: Jsoup.connect(...).referrer(URL + "/login")
Are you sure the hidden value is always 69? Maybe it's different for each request. You can get it dynamically like this: formData.put("hidden",html.select("co_js").first().attr("value"));
I don't like your way of getting the token. Let's use Jsoup to extract it:
String authToken = html.select("input#token").first().attr("value");
Edit:
I tried to do this from scratch and submitted login form was missing Content-Type definition. Try using:
Jsoup.connect(...).header("Content-Type","application/x-www-form-urlencoded")
I got the idea by analyzing headers in Chrome's dev tools. Now I can successfully login.
You can also check if you're logged in using this code:
Document doc = homePage.parse();
System.out.println("Logged in as: " + doc.select(".dropdown-toggle").text());
By letting server know you can handle compressed pages you can decrease downloaded page size. For every request use: Jsoup.connect(...).header("accept-encoding", "gzip, deflate") It's transparent and you don't have to do anything special to handle it, but it works internally.
Edit 2:
Providing final solution based on previous advices:
import java.util.HashMap;
import java.util.Map;
import org.jsoup.Connection;
import org.jsoup.Connection.Response;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class Stackoverflow51734840 {
private static final String URL = "https://time2watch.in";
private static final String URL_LOGIN = URL + "/login/";
static String userAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
public static void main(final String[] args) throws Exception {
Map<String, String> headers = new HashMap<String, String>();
headers.put("accept-encoding", "gzip, deflate");
Connection.Response loginForm = Jsoup.connect(URL + "/login").userAgent(userAgent).headers(headers).execute();
Map<String, String> cookies = loginForm.cookies();
Document html = loginForm.parse();
String authToken = html.select("input#token").first().attr("value");
System.out.println("Found authToken:" + authToken);
Map<String, String> formData = new HashMap<String, String>();
formData.put("username", "!!!!!!!!!!!!!!!!!!");
formData.put("pwd", "!!!!!!!!!!!!!!!!!!");
formData.put("hidden", "69");
formData.put("token", authToken);
headers.put("Content-Type", "application/x-www-form-urlencoded");
System.out.println("cookies before login:");
System.out.println(cookies);
System.out.println(" Logged in cookie present? " + cookies.containsKey("s4icookuser"));
Connection.Response afterLoginPage = Jsoup.connect(URL_LOGIN).cookies(cookies).headers(headers)
.userAgent(userAgent).data(formData).method(Connection.Method.POST).referrer(URL_LOGIN).execute();
// update cookies
cookies = afterLoginPage.cookies();
System.out.println("cookies after login:");
System.out.println(cookies);
System.out.println(" Logged in cookie present? " + cookies.containsKey("s4icookuser"));
Response homePage = Jsoup.connect(URL).cookies(cookies).method(Connection.Method.GET).userAgent(userAgent)
.referrer(URL_LOGIN).followRedirects(true).referrer(URL_LOGIN).headers(headers).execute();
Document doc = homePage.parse();
System.out.println("Error? " + doc.text().contains("Erreur"));
System.out.println("OK? " + !doc.text().contains("Se connecter"));
System.out.println("Logged in as: " + doc.select(".dropdown-toggle").text());
}
}
Here is my code to run the POST request
// Instantiate HttpClient
HttpClient httpClient = new HttpClient();
// Configure HttpClient, for example:
httpClient.setFollowRedirects(false);
// Start HttpClient
httpClient.start();
ContentResponse response = httpClient.POST("url")
.agent("Mozilla/5.0")
.method(HttpMethod.POST)
.param("do","login")
.param("url","")
.param("vb_login_md5password", this.password)
.param("vb_login_md5password_utf", this.password)
.param("s", "")
.param("vb_login_username", this.username)
.param("vb_login_password", "")
.send();
for(HttpField h : response.getHeaders())
System.out.println(h.getName() + " = " + h.getValue());
Here is the output
Date = Mon, 14 Dec 2015 08:17:08 GMT
Content-Type = text/html
Transfer-Encoding = chunked
Connection = keep-alive
Set-Cookie = __cfduid=dbb11c371945319b1d5943aad06c1977e1450081028; expires=Tue, 13-Dec-16 08:17:08 GMT; path=/; domain=.sythe.org; HttpOnly
Server = cloudflare-nginx
CF-RAY = 2548787ba9dc21b6-EWR
I look at the ContentResponse text of the post request by doing
response.getContentAsString()
However for some reason I keep getting a 411 error which tells me my Content-Length is zero or not specified. Is there a way to set the content length header of the post?
Thanks!
EDIT : Doing this makes no difference in my post request
.header(HttpHeader.CONTENT_LENGTH, "0")
With .param("do","login") you specify a query param to be added. For the POST to actually contain a body I'd think you need to set content() on the request. Maybe a FormContentProvider will work for you?
I have read extensively about how do this and I have tried a number of different variations, but I can't get it to work.
Basically, I just want to login to the ConEdison website and scrape my billing history. Here is what I have:
Connection.Response loginForm = Jsoup.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.data("_LASTFOCUS","")
.data("_EVENTTARGET","")
.data("_EVENTARGUMENT","")
.data("_VIEWSTATE", viewState)
.data("_EVENTVALIDATION", eventValidation)
.data("ctl00$Main$Login1$UserName", username)
.data("ctl00$Main$Login1$Password", password)
.data("ctl00$Main$Login1$LoginButton", "Sign In")
.userAgent("Mozilla/5.0")
.method(Method.POST)
.execute();
Map<String, String> loginCookies = loginForm.cookies();
Document document = Jsoup.connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
.cookies(loginCookies)
.get();
Elements data = document.select("table.ctl00_Main_lvBillHistory_Table1");
//checking if it found the right page
System.out.println("document: " + document);
//checking if it found the table
System.out.println("data: " + data);
I know the information is correct (though I don't know if I really need to pass the data parameters with no values).
I am not getting any errors, just printing out the login page (https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng)
Any help would be greatly appreciated.
Thanks
EDIT
So, I am now convinced that I was not able to get to the internal page, because after the POST to https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng, 3 cookies are set, but then it sends GET requests to https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET then to https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/MainHome.asp?src=DOTNET&frm= for additional cookies before going to the homepage
Does anyone know how I can follow all these redirects and get the cookies in the end?
Here is what I currently have, but I cannot get the cookies from the POST call.
Response response = Jsoup
.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.method(Method.GET)
.execute();
Map<String, String> cookies = response.cookies();
cookies.put("NSC_DpoFe_Bqqt-TTM-pme", response.cookie("NSC_DpoFe_Bqqt-TTM-ofx"));
System.out.println("response cookies: " + cookies);
response = Jsoup
.connect("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.header("Accept-Encoding", "gzip, deflate")
.header("Accept-Language", "en-US,en;q=0.8")
.header("Connection", "keep-alive")
.cookies(cookies)
.header("Host", "apps.coned.com")
.referrer("https://apps.coned.com/cemyaccount/NonMemberPages/Login.aspx?lang=eng&login=0")
.userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36")
.data("_LASTFOCUS", "")
.data("_EVENTTARGET", "")
.data("_EVENTARGUMENT", "")
.data("_VIEWSTATE", viewState)
.data("_EVENTVALIDATION", eventValidation)
.data("ctl00$Main$Login1$UserName", username)
.data("ctl00$Main$Login1$Password", password)
.data("ctl00$Main$Login1$LoginButton", "Sign In")
.followRedirects(false)
.method(Method.POST)
.execute();
System.out.println("post cookies: " + response.cookies());
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
response = Jsoup
.connect("https://apps.coned.com/cemyaccount/SessionTransfer.aspx?dir=2asp&url=https:"
+ "//apps.coned.com/csol/MainHome.asp?src=DOTNET")
.cookies(cookies)
.followRedirects(false)
.method(Method.GET)
.execute();
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
String guid = response.header("location");
response = Jsoup
.connect("https://apps.coned.com/csol/SessionTransfer.asp?dir=2asp&guid="
+ guid + "&url=https://apps.coned.com/csol/MainHome.asp"
+ "?src=DOTNET&frm=")
.cookies(cookies)
.method(Method.GET)
.execute();
cookies.putAll(response.cookies());
System.out.println("response cookies: " + cookies);
Document dataPage = Jsoup
.connect("https://apps.coned.com/CEMyAccount/CSOL/BillHistory.aspx?lang=eng")
.cookies(cookies)
.get();
System.out.println("data page: " + dataPage);
Elements data = dataPage.select("table.ctl00_Main_lvBillHistory_Table1");
System.out.println("data: " + data);
In the output I get all the cookies, except the POST cookies which are blank.
Open Development tools (press F12).
Select Network tab option.
Up and left there is a round button. If it's not red, click it. This will record the traffic.
Visit the login page. Enter your credentials and login.
Check what happens in Development tools. There is a table there,
showing all the files that were received.
Check the Type column. Search for the row that has the value
document. Select it. This will open a new screen.
Select Headers and scroll down until you locate the Request
Headers. There you will find the request that was made from your
browser. You will find there all the values that were send to the
server.
Search for the parameters you need. Take the values and hardcode them
to your code. Use the same user-agent (just in case) and in general
try to imitate this request with your code.
All the steps above are for the Chrome browser.
The answer was painfully simple - the headers with the underscores had 2 underscores, I was only using 1. Doh
I have my cookies in a CookieStore Object, but to stream a video I need to convert my CookieStore to a String, but my API in NodeJS (using express, cookie-parser and passport-local) never recognizes the value of the cookie, whenever I try to encode/decode it before. I think it's a stupid problem just I'm not really good in HTTP Headers so I'm doing it wrong.
There is the code:
Method setVideoURIMethod = videovvw.getClass().getMethod("setVideoURI", Uri.class, Map.class);
Map<String, String> params = new HashMap<String, String>(1);
params.put("Cookie", ARequestAPI.getCookieString());
setVideoURIMethod.invoke(videovvw, uri[0], params);
I had the code from here so normally it should work perfectly: Authentication for Videoview in android
I think the problem comes from "ARequest.getCookieString()":
protected static CookieStore _cookies = null;
...
...
...
public static String getCookieString() {
String cookieString = "";
Log.v("Debug", _cookies.toString());
for (Cookie cookie : _cookies.getCookies()) {
try {
Log.v("Debug", "Decode: " + URLDecoder.decode(cookie.getValue(), "UTF-8"));
Log.v("Debug", "Encode: " + URLEncoder.encode(cookie.getValue(), "UTF-8"));
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
cookieString += cookie.getName() + "=" + cookie.getValue()+ ";";
}
return cookieString;
}
The display in LogCat:
And the log on my API:
When I do a request using CookieStore:
{ 'connect.sid': 's:NUTf8t9o8cepR1yYItMexMxy.WFv/ZlktryfpVZHweVozabW1US4UBvGlWxQR7G7Aamc' }
When I do the request with my function which convert the CookieStore to a String:
{ 'connect.sid': 's:AZXpZmQGX7eJgej9hVA1qaAk.7vWP756Flwbte/qxBRcLOhl/CXMlVO3HVvmsvsEBpzA' }
I tried all the options, even encode(decode()), ... but it is never a good one on my API.
The weird thing is that I tried to decode 's%3ANUTf...' (so cookie.getValue()) with the javascript function "decodeURIComponent()" and I found exactly 's:NUTf8t9o8cepR1yYItMexMxy.WFv/ZlktryfpVZHweVozabW1US4UBvGlWxQR7G7Aamc', so my cookie is good, but I think I'm doing something wrong on the encode/decode part.
Thanks for your help.
Edit: After many tests, I saw that on the API the value for the cookie is always the same, whenever I use URLDecoder.decode(), URLDecoder.decode() or event directly cookie.getValue(), the only time I get the right value is when I "choose/write" the value myself like "totoauzoo" for example with this wring I get exactly the same value on the API. But it's not working with my old cookie value like "s:DMCBao7zeS9B2jwIfeQoDZtl.3XPIYIm7y2Bz9/o468v4wxvFZmjDrc6hKk4ty89sIX4".
Edit2: I probably found what is wrong, I get that on the API:
request with HttpGet and CookieStore:
headers { host: 'xxx',
connection: 'Keep-Alive',
'user-agent': 'Apache-HttpClient/UNAVAILABLE (java 1.4)',
cookie: 'connect.sid=s%3AGRGciNxOcR4BXHrUG8PikHMX.MyWa6vjW%2BBlcUaaCHHendqc7DEK4aoNFDzm5aabOkDM',
cookie2: '$Version=1' }
cookies { 'connect.sid': 's:GRGciNxOcR4BXHrUG8PikHMX.MyWa6vjW+BlcUaaCHHendqc7DEK4aoNFDzm5aabOkDM' }
request with setVideoURI:
headers { host: 'xxx',
connection: 'keep-alive',
cookie: 'connect.sid=s%3AKWKdcuXqUpzBIMv0sOGpPxqM.xb14kPsGKvn%2Fv%2BVcfUDzxWsye8QdJfuQgonNocsX3k8',
'user-agent': 'stagefright/1.2 (Linux;Android 4.1.2)',
'accept-encoding': 'gzip,deflate' }
cookies { 'connect.sid': 's:KWKdcuXqUpzBIMv0sOGpPxqM.xb14kPsGKvn/v+VcfUDzxWsye8QdJfuQgonNocsX3k8' }
How can I fix that?
I would suggest inspecting the network traffic itself.
Cookies are nothing else than HTTP headers ...
HTTP/1.0 200 OK
Content-type: text/html
Set-Cookie: name=value
Set-Cookie: name2=value2; Expires=Wed, 09 Jun 2021 10:18:14 GMT
or
GET /spec.html HTTP/1.1
Host: www.example.org
Cookie: name=value; name2=value2
Accept: */*
I'm not familiar with the classes you are using to set the cookies, but I would inspect two places:
1:
params.put("Cookie", ARequestAPI.getCookieString());
Is this call setting the headers or are you setting POST parameters? This should be visible in the network traffic. What are params? Headers, query string?
2:
ARequestAPI.getCookieString()
Is the output of this function in correct cookie value format, or what is it?
Most probably in one of those places lies the error.