Understanding Fiddler grabbed network traffic to build HTTP Client - java

I am trying to reverse engineer a web app (which uses flash object to communicate with server). I have grabbed the network traffic via fiddler i.e., Browsed the app using IE and captured network traffic in Fiddler. This first time i am doing something this so i might be asking very basic questions :(
Now, I have those events/requests in Fiddler but I am having hard time understanding them (besides basic HTTP requests). So I am going to post the traffic flow and then its respective traffic here and at the end the questions
FLOW ON IE
Entered the url website.com/app/app-subdomain/web-app
An HTML page is displayed with user & password request
After login, an HTML page is displayed with Flash object in it (the original app)
IN FIDDLER
(requests in order)
The first thing i see is request to URL:www.website.com:443 which results in 200 status. Fiddler shows there are no cookies or whatsoever. Only the "Clients" are there
The second request i see is to URL:app/app-subdomain/web-app. However, here is the part where i am confused at. In fiddler request body, I see a cookie. which is something like this
GET https://www.website.com/app/app-subdomain/web-app HTTP/1.1
Accept: text/html, application/xhtml+xml, \*/\*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Host: www.website.com
Cookie:
NAME-WEB-SESSION=akcxxxpxkxfaxdxccajkjumxax;
hl=us;
dxxxxcxx=0;
NAMESSO=fdxdfxdfdxcfabxxxcxxcdbfexxxxfxfxxxefxecxxaxxxxxxefxxxxxxxxfxaxx;
XSRF-TOKEN=vXXnjhHE-ptnvmYfKfQVxscHyrGrfbwxyxkGzfWZGoU
So far, the thing which is confusing me, is who generated this cookie ? so lets say i am using Apache HTTPClient, would this cookie be generated by it or do I have to ? If i have to, then how to generate the values of those key-value cookie ?

You didn't specify what the first request was exactly, but more than likely this was just a HTTP CONNECT tunnel through which secure traffic flows. You should NEVER see a cookie on a CONNECT tunnel. Have you cleared your browser's cookies and cache? If not, the cookie you saw was likely set on a previous visit to the site and stored in the client's cookie jar. If you have cleared the cache and cookies, that implies that something on the client (e.g. Flash) generated the cookie via some other, non-standard, process.

Related

Spring Boot and security: How to extend response for 302 status code?

Setup:
Spring Boot application
OAuth2 security
ReactJS for UI implementation
Use case:
Login to application
open other tab with same application in same browser
Logout from application in one of the tabs, user are redirected to Login view
Go to first tab(user already logout and if I refresh page I will get login form) and do any action that triggers POST/PUT/PATCH request. Example of request and response below:
Request:
Request URL:http://localhost:8080/api/info
Request Method:PUT
Status Code:302 Found
Remote Address:[::1]:8080
Referrer Policy:no-referrer-when-downgrade
accept:application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8,sv;q=0.6,ru;q=0.4,uk;q=0.2,fr;q=0.2
Cache-Control:no-cache
Connection:keep-alive
Content-Length:66
Content-Type:application/json
Cookie:_ga=GA1.1.1868465923.1505828166; _gid=GA1.1.612220229.1507272075; session=e4oSW4Kq; prod_user_session=4d6b615f-521704; user_session=g3ggLxJDomyZ
Host:localhost:8080
mode:cors
Origin:http://localhost:8080
Pragma:no-cache
Referer:http://localhost:8080/profile
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Response:
Cache-Control:no-cache, no-store, max-age=0, must-revalidate
Connection:keep-alive
Content-Length:0
Date:Fri, 06 Oct 2017 12:12:26 GMT
Expires:0
Location:http://localhost:8080/login
Pragma:no-cache
X-Content-Type-Options:nosniff
X-Frame-Options:DENY
X-XSS-Protection:1; mode=block
After this response system triggers PUT request to http://localhost:8080/login and fails because PUT method not allowed for http://localhost:8080/login request.
Question:
I understands that I'm getting 302 status and Location:http://localhost:8080/login header because I'm already logged out. I want extend response for this case with JSON body or at least ensure that for this case I will get 401 Unauthorised status code instead of 302.
If I understood your question correctly, it sounds like you need two different responses for unauthorized requests coming via a regular webpage load (produces="text/html") vs an AJAX call (produces="application/json"). Currently your unauthorized AJAX call gets redirected to the login page which is a legit page, hence no 401 response code. Here's an example of a setup that accomplishes what you want using XML configs Spring Security Ajax login and #Configuration config https://stackoverflow.com/a/27300215/1718213
Another option that is even more user-friendly is to use Spring's WebSocket support to signal logout events to all the tabs a given user might have open (across all devices and browsers) that would trigger each tab to redirect to the login page.

Fake being a browser to avoid a 403 error

I'm using JSoup to connect to a webpage and scrape data from it, but it's giving me an HTTP 403 response (unlike my browser, which loads it successfully). From what I understand, I have to make my scraper pretend that it's a browser. I've tried two things: setting the user agent to be a browser, and setting the referrer to be the same website (both of these I got by browsing StackOverflow). I still, however, get a 403. My code looks like this (I know the browser is old, I just copypasted it, surely it shouldn't matter?):
Document doc = Jsoup.connect("http://www.website.com/subpage/")
.userAgent("\"User-Agent\", \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11\"")
.referrer("http://www.website.com")
.get();
What else do I need to do to trick the server into thinking that I'm a browser?
Since you can load the page successfully(a 200?) with your browser, you can use that information to create a Jsoup connection.
Open up your browser's network tab in development view, have a look at the request and imitate it. For example, a GET to this page looks like
Host: stackoverflow.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0)
Gecko/20100101 Firefox/46.0
Accept: application/json, text/javascript; q=0.01
Accept-Language: sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: http://stackoverflow.com/questions/37134906/fake-being-a-browser-to-avoid-a-403-error
Content-Length: 263
Cookie: x; prov=x; acct=t=wx
DNT: 1
Connection: keep-alive
All these have corresponding Jsoup methods. This should be enough to fool the web server.
If you are still experiencing trouble you should log the actual request to see it is sent as is expected.
There are several ways to distinguish web browsers from robot user agents. One possibility that comes to mind is checking for the Accept header content.
I suggest that you use Firefox developer tools to inspect your requests and add headers/cookies to your scraper application.
Additionally you can use a packet sniffer (ngrep, wireshark) and compare your requests with the requests of a real browser session to determine what signals are used.
A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server can be reached and understood the request, but refuses to take any further action. Status code 403 responses are the result of the web server being configured to deny access, for some reason, to the requested resource by the client.
It is working in browser, in browser he might take some header or
cookies.
Please check which are the header or any other param require using Fiddler or simple browser & set those value in Jsoup it will resolve your issue.

Gmail Authentication using XOAuth 2.0

I want to implement XOAuth 2.0 for my web application in Java.
My application asks for username and password from the user. Now using these credentials I want to authenticate the user through Gmail.
Please suggest libraries, and also how to implement it??
I am new to authentication using Gmail. Need some suggestions.
as I know the purpose of the OAuth 2.0 is to prevent leak of the user's credentials into 3rd party applications.
The abstract flow shows how client (3rd party app) should use the access_token (retrieved with one of the grant types) to access resource.
The specification also defines the basic HTTPS calls to use for example in case of an authorization code grant type (I think this flow best suits for you). This has 2 request, one for the code (user agent redirection is used to send the code to the 3rd party app) and one for the access_token.
Authorization code request&response
GET /authorize?response_type=code&client_id=s6BhdRkqt3&state=xyz
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb HTTP/1.1
Host: server.example.com
HTTP/1.1 302 Found
Location: https://client.example.com/cb?code=SplxlOBeZQQYbYS6WxSbIA
&state=xyz
Token request&response:
POST /token HTTP/1.1
Host: server.example.com
Authorization: Basic czZCaGRSa3F0MzpnWDFmQmF0M2JW
Content-Type: application/x-www-form-urlencoded
grant_type=authorization_code&code=SplxlOBeZQQYbYS6WxSbIA
&redirect_uri=https%3A%2F%2Fclient%2Eexample%2Ecom%2Fcb
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"example",
"expires_in":3600,
"refresh_token":"tGzv3JOkF0XG5Qx2TlKWIA",
"example_parameter":"example_value"
}
So you do not need to ask a user for the GMail credentials.
Here you can find a spring-security example for an OAuth 2.0 compliant application.
I hope I could help.

HttpUrlConnection not working and shows different status code for GET and POST call

I am getting FileNotFoundException when making GET call to REST API. Here HTTP status code I get is 403.
For POST call I get IOException : No authentication challenges found, whereas I pass Authorization header. Here I get HTTP status code 401.
Look at my already asked question to see the code and logcat screen shot for POST call.
Below I am attaching logcat screen shot for GET call :
Note :
1) I have tried using Authenticator.
2) Tried different base64 flags such as NO_WRAP, URL_SAFE, DEFAULT.
3) My simple call to www.google.com works.
4) When I log urlConnection.getErrorStream(), last line in image is printed. I don't understand what is that and what does it mean. I have specified Content-type to application/json in header.
UPDATE : I tried using Burp and found that headers "Accept", "Content-type" were different. I used the same as in iOS app. But still it does not work.
Things to note :
1) It always throws an Exception on the line in = urlConnection.getInputStream();.
2) I logged few things and according to it, content-length is 114, which is not null. content-type is application/json; charser=utf-8.
There's a difference between your HTTP traffic for iOS and for Android. This is guaranteed, otherwise you'd get identical behaviour from the server. The difference is probably in HTTP header(s) &/or parameter(s).
This is very difficult to debug remotely via SO Q&A - E.g. we don't know what headers & parameters your iOS client is successfully using nor how your server is configured & programmed.
How to diagnose the problem & correct yourself:
Trace your working HTTP traffic: iOS client <-> server
Trace your non-working HTTP traffic: Android client <-> server
Compare (2) and (3). For the most thorough comparison, save each HTTP request and response message as a separate file for (2) and (3), then diff the corresponding files.
Recommended HTTP tracing tools:
Fiddler2 (windows only) See also Documentation
Burp (JVM-based: windows, linux, OSX, etc) See also Getting Started
WebScarab (JVM-based: windows, linux, OSX, etc) See also Getting Started
UPDATE
Seems you have the same problem for both GET and POST: the server is configured for BASIC authentication, but the client is not following the authentication protocol correctly. I think it just shows as a slightly different sympton in the two cases: for GET it says 'resource not found' (because you're not authenticated) and for POST the resource is given by you, but the server says you're not authorized to change the resource on the server. I suggest you've done enough (good!) debugging of request contents and now you should stop and focus on getting authentication working.
Send you GET/POST request to the server without Authorization header
Allow the server to prompt you for authentication with a 401 response with an WWW-Authenticate header containing a challenge string (e.g. WWW-Authenticate: Basic realm="Protected" see RFC 2617 HTTP Basic Authentication and Digest Authentication)
Now send an additional GET request to server that (either without/with the original request contents), but includes the Authentication header, with Base 64 encoded username:password (Authorization: Basic ZnJhbms6ZmllZGxlcg==)
I solved my problem and it is something I never tried to focus on while solving the problem.
I need not pass Authorization header. The thing is that there are 2 credentials come into the picture. one is server's authorization and second is credentials for login API. In my app, user creates an account and login to it. To authenticate the user I pass credentials to server and server authenticate it.
So when user enters correct credentials then response received is correct. And, in case of wrong credentials, my server passes a error message You are not authorized person, which I want to display to user(as in my iOS app). So the problem is here that HTTP status code (in case of wrong credentals) is 401 and that is why I don't receive the message sent by the server (and receive No authentication challenges found message).
The reason why I don't receive server message is that HttpUrlConnection don't give server response when HTTP status code is >= 400.
The only option to get error details in case status code 400 and above is to use getErrorStream() method and using that I was receiving No authentication challenges found message.
Finally, either I had to handle each status code, that is equal and above 400, at client side or I can use HttpClient, instead of HttpUrlConnection. And now I am moving to HttpClient.
I had a similar problem, and solved by passing the authenticated session cookie. Not sure if that is possible in your situation.
AuthUser="foobar"
AuthPass="password"
URL targetUrl = new URL("http://www.google.com/");
HttpURLConnection connection = (HttpURLConnection) targetUrl.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
String authStr = Base64Variants.MIME_NO_LINEFEEDS.encode((AuthUser+":"+AuthPass).getBytes());
connection.addRequestProperty("Authorization", "Basic "+authStr);
InputStream inputStream= connection.getInputStream();
I just run something like this and it worked perfectly for me. Just make sure you use Base64Variants.MIME_NO_LINEFEEDS and you should be able to create a proper authentication header. If that doesn't work, then you might have some problem on the server side.

Authentication in SOAP UI while testing web services

I am using SOAP UI for testing the web services. In the Authentication tab, I have entered the username and password and domain name.
Sample:
WSDL URL: http://localhost:8080/soap/helloTest?wsdl
Username:hello
pass:helloworld
domain: http://mysysfullname:8080
I have given the correct credentials. I also replaced domain as http://localhost:8080.
Even then I am getting invalid credentials only. The username and password are correct and they are not my windows credentials.
Can some one help how to give the credentials.
Adding Raw Request
POST http://localhost:8080/soap/helloTest HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: ""
User-Agent: Jakarta Commons-HttpClient/3.1
Host: localhost:8080
Content-Length: 1004
Thanks.
It also depends on the WSS-PasswordType
verify that also.
Try setting SOAP-UI to use Preemptive Authorisation Type.
As per SOAP-ui documentation:
Send Authentication headers with each request without first receiving an authentication challenge. This is a potential security hazard but will improve performance since only one request will be required for authenticated endpoints instead of two.

Categories