We Keep Coding

2 Same HTTP Requests Give Different Results?

What differentiate these 2 requests that cause them to have different results/responses from the server although they should be the same ?
Request initiated by Chrome after a simple
click/navigation(successful, response code is 302)
I simply copied
that request as a curl and imported it to Postman and then postman
hanged
I did the same with Java - HttpUrlConnection(mimicking all the request headers and cookies like Chrome sent), but it hanged and waited forever. Is this simply because of the server logic that doesn't accept non-browser client ?
Here are the steps that I tried:
1. Visited this link: https://www.tokopedia.com/p/handphone-tablet/handphone
2. I opened the inspector and opened the Network - All tab
3. I clicked one of the products
4. I clicked the top request from the Network - All tab
5. I copied it as cURL bash
6. I imported it to Postman
7. I ran that request
8. Postman hanged

Actually the problem might even go deeper than what the other answers say.
So neither the User-Agent request header nor telnet might solve that problem (unless you initialize the TLS handshake also with telnet MANUALLY, but that is near impossible to complete).
TLS fingerprinting
If the connection is an SSL/TLS connection, the server could detect which algorithm is used to generate keys, and most applications have their specific signature / cipher.
So only by the TLS handshake alone you can tell Chrome from Postman or FireFox or Java. Java usually - unless a JVM implementation REALLY wants to go off-road - has the same signature across all platforms, using the same cipher/algorithm across all implementations.
I am sorry I cannot properly recall the name of this technique. The first project I know that published this is called something like "A3" or "S3". Salesforce published an article about JA3 analysis. They describe the technique and show a list of signatures and applications so you can guesstimate what app you're talking to, without the need to even decrypt the data: https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967
My Solution
I had that same problem too, wanted to scan the NVidia or AMD servers for graphics card availability. Did not work from Java, so after a lot of research, finding the project mentioned above, I simply used Selenium to control FireFox and that got the proper server responses and I achieved my goal this way.

The only way to be sure that the exact same data is sent is to manually send it yourself through something like telnet. I had a similar problem once- it turned out that the browser was sending the data in one big chunk, while my code was sending it line-by-line. No site should have this problem, but it's possible that it exists.

The server might be checking for User-Agent request header and will block traffic that does not originate from a browser. Try setting the header in curl or your Java Code to a value corresponding to (any) browser. I've encountered such behavior on some e-shops and commercial websites.

Why is GetServerAuthCodeResult Deprecated? How can I do something equivalent in an Installed Application?

Following this post: http://android-developers.blogspot.com/2016/01/play-games-permissions-are-changing-in.html I have obtained a single use authorization code for use on my backend server as follows:
import com.google.android.gms.games.Games;
//later
Games.GetServerAuthCodeResult result = Games.getGamesServerAuthCode(gameHelper.getApiClient(), server_client_id).await();
if (result.getStatus().isSuccess()) {
String authCode = result.getCode();
// Send code to server...
This seems to works fine, but it presents a question:
1) getGamesServerAuthCode and GetServerAuthCodeResult are marked as deprecated. Why? Should I be using something else instead?
2) How would I do something equivalent in an non-Android installed Java application? I am able to obtain a token on the client application, but I also need to obtain a single use code to pass to my backend server like above. I can't find an equivalent function to get a Server Auth Code. (using com.google.api.client.extensions.java6.auth.oauth2)
I am basically trying to follow this flow: https://developers.google.com/games/services/web/serverlogin but in Java, NOT Javascript. I am attempting to do this in an Android app and a desktop Java app.

1) Yes, in Android use GetServerAuthCodeResult although it is still marked as deprecated. It is the recommended way from Google and it seems they have only forgot to remove the deprecation annotation when releasing to general public.
2) For desktop applications you can follow the instructions here: https://developers.google.com/identity/protocols/OAuth2InstalledApp
Basically from your app you open the system browser (embedded webviews are discouraged) and make a https request to the https://accounts.google.com/o/oauth2/v2/auth endpoint. In the request you supply a local redirect URI parameter i.e. http://127.0.0.1:9004 (you should query your platform for the relevant loopback IP, and start a HTTP listener on a random available port). The authorization code will be sent to your local HTTP listener when the user has given consent or an error such as error=access_denied if the user declined the request. Your application must be listening on this local web server to retrieve the response with the authcode. You also have the option to redirect to a server URI directly claimed by your app, see docs on link above. When your app receives the authorization response, for best usability, it should respond with an HTML page, instructing the user to close the browser tab and return to your app. Also, if you want the Games-scope make sure you are using the https://www.googleapis.com/auth/games as scope in the request, example below, with line breaks and spaces for readability.
https://accounts.google.com/o/oauth2/v2/auth?
scope=https://www.googleapis.com/auth/games&
redirect_uri=http://127.0.0.1:9004&
response_type=code&
client_id=812741506391-h38jh0j4fv0ce1krdkiq0hfvt6n5amrf.apps.googleusercontent.com
Please note that I think you'll have to create and link an app of type other, in the Google Play Developer Console linked-app, for the localhost redirection to work. Use type Web if you plan to redirect to server URI directly, add your server URI to Authorized redirect URIs in the API Manager under section Credentials.
Browser screenshot:

There is finally a proper answer to part 1) of this question!
In the release notes of gms 10.2.0
https://developers.google.com/android/guides/releases#february_2017_-_v102
the new method of obtaining a server code is described. A good example of how to do this is provided here:
https://github.com/playgameservices/clientserverskeleton
I ended up updating Google's baseGameUtils to follow the example above.
Still not sure the proper way to do this for part 2) of the question, at the moment I am sending the token to the server which works but is probably unsafe.

HTTP Referrer. Is it reliable? [duplicate]

This question already has an answer here:
How reliable is HTTP_REFERER?
(1 answer)
Closed 8 years ago.
I am interested in logging from where a user comes in order to access my web app.
I thought of using HTTP's referrer header for that, but from e.g.HTTP referrer wiki
it seems that this is not a accurate/reliable way since in many cases it is not send.
I was wondering is the referrer header the only way? Is there a better/standard approach?

Reliable way would be to have ?ref=somehash a GET parameter
For example:
Consider this site SO, they have list of questions, now there is a portlet which streams the recent questions to some other site for example abcd.com now to see if user clicked the link from abcd.com you pass a parameter ?ref=423jahjaghr where this string maps to abcd.com

Referrer header isn't the only way, but it is the most standard.
You can consider using Google Analytics, which has extra referrer capabilities, but you'd have to manually setup collecting the data from their services to input into your logging infrastructure.
Nothing is going to be 100% fool proof though. It's pretty straight-foward to block Google Analytics, and spoof referrers, and HTML5 will make it even easier to prevent sending referrer information.
If it's mission critical that you know the referrer of all inbound traffic, you'll have come up with a more draconian approach (like #Jigar Joshi has suggested)

Depending on the browser, you may OR may not get the referrer header. You may not get it always. You have to have a request parameter OR a form field to get the referrer.

HTTP Referer is a good way to analyze logs. And to maintain analytics on user interactions. However a browser or any other system which displays webpages and is able to traverse a webpage might not send this header entry.
You might also consider to use a 3rd party application like google analytics. But you should check if this 3rd party tool is legal in your country. Most of them have data-privacy issues.
Very important is. For analytics its ok if you have a certain error in the expected outcome. However never do any security related checks on the http referer. Someone might enter whatever he wants as referer.

How to secure a REST web service in Java EE 6

I have made a web application using Java EE 6 (using reference implementations) and I want to expose it as a REST web service.
The background is that I want to be able to retrieve data from the web application to a iOS app I made. The question is how would I secure the application? I only want my application to use the web service. Is that possible and how would I do this? I only need to know what I should search for and read and not the actual code.

Unfortunately, your webservice will never be completely secure but here are few of the basic things you can do:
Use SSL
Wrap all your (app) outbound payloads in POST requests. This will prevent casual snooping to find out how your webservice works (in order to reverse engineer the protocol).
Somehow validate your app's users. Ideally this will involve OAUTH for example using Google credentials, but you get the idea.
Now I'm going to point out why this won't be completely secure:
If someone gets a hold of your app and reverse engineers it, everything you just did is out the window. The only thing that will hold is your user validation.
Embedding a client certificate (as other people have pointed out) does nothing to help you in this scenario. If I just reverse enginneered your app, I also have your client certificate.
What can you do?
Validate the accounts on your backend and monitor them for anomalous usage.
Of course this all goes out the window when someone comes along, reverse engineers your app, builds another one to mimic it, and you wouldn't (generally) know any better. These are all just points to keep in mind.
Edit: Also, if it wasn't already obvious, use POST (or GET) requests for all app queries (to your server). This, combined with the SSL should thwart your casual snoopers.
Edit2: Seems as if I'm wrong re: POST being more secure than GET. This answer was quite useful in pointing that out. So I suppose you can use GET or POST interchangeably here.

Depends on how secure you want to make it.
If you don't really care, just embed a secret word in your application and include in all the requests.
If you care a little more do the above and only expose the service via https.
If you want it to be secure, issue a client certificate to your app and require a
valid client certificate to be present when the service is accessed.

my suggestions are:
use https instead of http. there are free ssl certificate avaliable,
get one and install.
use a complex path such as 4324234AA_fdfsaf/ as the root end point.
due to the nature of http protocol, the path part is encrypted in the https request. therefore it's very safe. there are ways to decrypt the request through man-in-the-middle attack but it requires full control over the client device including install an ilegal ssl certificate. but, i'd spend more time on my app to make it successful.

Create a rule on the machine which hosts your Web Service to only allow your application to access it through some port. In Amazon EC2, this is done creating a rule in the instance Security Group.

We have used RestEasy as a part to securing our exposed RESTful webservices. There should be lot of example out there but here is the one which might get you started.
http://howtodoinjava.com/2013/06/26/jax-rs-resteasy-basic-authentication-and-authorization-tutorial/
You can also use OAUTH:
http://oltu.apache.org/index.html

Uploading to Youtube via a proxy using the Java Youtube API

So I want to write a servlet which uploads a video to a youtube channel using the Java API, but I can't seem to find a way of specifying that I want to go through a proxy server. I've seen an example on this site where someone managed to do this using C#, but the Classes they used don't seem to exist in the Java API. Has anybody managed to successfully do this?
YouTubeService service = new YouTubeService(clientID, developerKey);

I'm new here so I'm unable to comment on posts (and a little late on this topic), but Jesper, I believe this is the C# sample that the original poster was talking about: How to upload to YouTube using the API via a Proxy Server
I can see no "direct" way of porting that example to Java though, since the GDataRequestFactory doesn't seem to have any proxy-related fields.
I was also having issues with the Java client Library with proxy in our application. Basically, the library picks up the global Java proxy settings:
System.getProperty("http.proxyHost");
System.getProperty("http.proxyPort");
but for some reason not everywhere. To be more precise, even with a proxy server properly configured in Java, YouTube authentication (calling service.setUserCredentials("login", "pwd")) would use a direct connection and ignore the proxy. But a video upload (calling service.insert(...)) would use the proxy correctly.
With the help of folks at the official YouTube API mailing list, I was able to nail this down. The issue is that the authentication is performed using SSL (HTTPS) and since there is a different set of properties for the HTTPS proxy, this didn't work. The fix is to simply set https.proxy* properties as well (in addition to http.proxy*), so that these point to a valid proxy server too:
System.getProperty("https.proxyHost");
System.getProperty("https.proxyPort");