We Keep Coding

Java https request get 403 response code for curseforge site

I try get redirect from https://www.curseforge.com/projects/291874 to https://www.curseforge.com/minecraft/mc-mods/serene-seasons but all tries response 403. I try HttpURLConnection, HttpsURLConnection and URLConnection but nothing to work.
How do request on this site?
Browser do this two steps:
CONNECT www.curseforge.com:443 HTTP/1.1
GET /projects/291874 HTTP/1.1
How do this on java?
Please check your code before answer (first link in question)
String url = "https://www.curseforge.com/projects/291874";
try {
HttpURLConnection con1 = (HttpURLConnection)(new URL(url).openConnection());
con1.setRequestProperty("Host", "www.curseforge.com");
con1.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0");
con1.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con1.setRequestProperty("Accept-Language", "ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3");
con1.setRequestProperty("Accept-Encoding", "gzip, deflate, br");
con1.setRequestProperty("DNT", "1");
con1.setRequestProperty("Connection", "keep-alive");
con1.setRequestProperty("Upgrade-Insecure-Requests", "1");
con1.connect();
//con1.getInputStream();
System.out.println(con1.getResponseCode());
System.out.println(con1.getHeaderField("Location"));
} catch (IOException e) {
e.printStackTrace();
}

Getting "java.io.IOException: Server returned HTTP response code: 403 for URL: " in java code

I am trying to validate the linkedIn profile of 100K person and wrote a dummy code but its giving "java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user"
I have tried setting different setRequestProperty but not working.
public static void main(final String[] args) {
String output = "";
int TIMEOUT_VALUE = 99999999;
HttpURLConnection conn = null;
BufferedReader br = null;
String urlEndPoint = "";
String authUser = "";
String authPwd = "";
try {
long start = System.nanoTime();
urlEndPoint = "https://www.linkedin.com/in/test.user";
authUser = "linkedin-username";
authPwd = "linkedin-password";
URL url = new URL(urlEndPoint);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("username", authUser);
conn.setRequestProperty("password", authPwd);
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Keep-Alive", "header");
conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
conn.setConnectTimeout(TIMEOUT_VALUE);
conn.setReadTimeout(TIMEOUT_VALUE);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.9,mt;q=0.8");
conn.setRequestProperty("Accept-Encoding", "gzip,deflate,br");
conn.setRequestProperty("Host", "www.linkedin.com");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36");
conn.setRequestProperty("http.agent", "Chrome/71.0.3578.80 (Windows NT 10.0; Win64; x64)");
conn.setDoOutput(true);
String userPassword = authUser + ":" + authPwd;
String encoding = Base64Encoder.encode(userPassword);
conn.setRequestProperty("Authorization", "Basic " + encoding);
OutputStream os = conn.getOutputStream();
os.flush();
conn.connect();
br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
while ((output = br.readLine()) != null) {
System.out.println(output);
}
if (br != null) {
br.close();
}
if (os != null) {
os.close();
}
long elapsed = System.nanoTime() - start;
} catch (MalformedURLException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (IOException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} finally {
try {
if (conn != null) {
conn.disconnect();
}
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
}
}
//logger.info("processPartyTerrRelationship called ends");
}
The outcode of above code is :
java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at ValidateLinkedInProfiles.main(ValidateLinkedInProfiles.java:57)

The HTTP error code 403 is an error related to the authorization to the requested resource:
HTTP 403 provides a distinct error case from HTTP 401; while HTTP 401 is returned when the client has not authenticated, and implies that a successful response may be returned following valid authentication, HTTP 403 is returned when the client is not permitted access to the resource for some reason besides authentication

It's hard to understand how you're working. The LinkedIn link requires login. But you indeed need to debug it somehow and need a raw real output to the server with the correct one otherwise you will not complete it. If you have Java example program, see if they or you have a typo, but again without screenshot or text from LinkedIn I cannot debug it. Maybe try to add the examples and I will try to help you (just make me login with my public profile to other places). Also make sure there is your real password and your user account in the correct fields of course (authUsr,authPwd shall not be copy paste unlike everything else).

HTTP 403 is a legitimate response from a server. So the behavior is valid. However, I would recommend to use some HTTP client utility rather then writing your own code to make Http request. This will reduce the chance of a problem caused by your own code. As some Http clients I would suggest Apache Http Client or OK Http client or MgntUtils Http Client (see MgntUtils HttpClient javadoc here, Full MgntUtils library on github is here and Maven repository is here). Disclaimer: MgntUtils library is written by me

HTTP 403 is a standard HTTP status code communicated to clients by an HTTP server to indicate that the server understood the request, but will not fulfill it. There are a number of sub-status error codes that provide a more specific reason for responding with the 403 status code.
You either do not have access to the site(try logging in from a browser and try to run the script from the same browser, if your access is shared across different tabs of the same browser that is also fine, but make sure you're authorized) or the request to the link contains sensitive information which the site doesn't want to share.

FileNotFoundException on getInputStream from GAE address

I'm doing an Android app with an API with Python. The API is on a Google App Engine cloud and everything works fine when I tested it with Postman.
I'm trying to do a Login with a POST method. That method returns json with the user information I keep getting that error: FileNotFoundException
Here is some of my code:
try{
String account = params[0].get(0);
String password = params[0].get(1);
URL url = new URL("http", WEB_SERVICE_URL, PORT, REST_LOGIN);
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setConnectTimeout(CONNECTION_TIMEOUT);
httpURLConnection.setRequestMethod("POST");
httpURLConnection.setDoOutput(true);
httpURLConnection.setRequestProperty("Content-Type", "application/json");
httpURLConnection.setRequestProperty("Accept", "application/json");
JSONObject json = jsonParser.serialJsonLogin(nomCompte, motPasse);
osw = new OutputStreamWriter(httpURLConnection.getOutputStream(),"UTF-8");
osw.write(json.toString());
osw.flush();
String body = readStream(httpURLConnection.getInputStream());
osw.close();
Log.i(TAG, "Return : " + body);
user = jsonParser.deserializeJsonUser(body);
}catch (Exception e) {
mException = e;
}finally {
if (mHttpURLConnection != null) {
mHttpURLConnection.disconnect();
}
}
return user;
At: String body = readStream(httpURLConnection.getInputStream()); I'm getting a java.io.FileNotFoundException: http://10.0.2.2:8080/login
My readStream method is fine, I tested it. If I look in my Google App Engine logs, I can see that there is no 404, or anything wrong. If I find the user I get a 201 if not a 403. So even if the error says FileNotFound, I see status code which means that actually the URL is right.
UPDATE: My API was giving me a 201 and getInputStream apparently doesn't work on 201 status. Changed my return status to 200 in my API and it works fine.

Response from UrlConnection Java

So far, I have this snippet.
URLConnection connection = null;
try {
connection = (new URL("some_link")).openConnection();
connection.setConnectTimeout(5000);
connection.setReadTimeout(5000);
connection.connect();
} catch (IOException e) {
}
The possible response codes are 200 and 404 and it's working fine when the response code is 200 (OK). My question is how can I find the response code received by my connection, for example: if the response code is 404, throw an exception and do smth there.

use something like this:
HttpURLConnection connection = (HttpURLConnection)new URL("URL_STRING")
.openConnection();
int statusCode = connection.getResponseCode();

Make it to a type of HTTPUrlConnection. Then you can use .getResponseCode();

HttpUrlConnection redirection to other servlet is not happening

I have the following code which will call the server through HttpUrlConnection.
String response = HttpUtil.submitRequest(json.toJSONString(), "http://ipaddr:port/SessionMgr/validateSession?sessionId=_78998348uthjae3a&showLoginPage=true");
The above lines will call the following code:
public static String submitRequest(String request, String **requestUrl**) {
try {
URL url = new URL(requestUrl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
OutputStream os = conn.getOutputStream();
os.write(request.getBytes());
os.flush();
if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
throw new RuntimeException("Failed : HTTP error code : "
+ conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
StringBuffer sb = new StringBuffer();
while ((output = br.readLine()) != null) {
sb.append(output);
}
conn.disconnect();
return sb.toString();
} catch (MalformedURLException e) {
} catch (IOException e) {
}
return "";
}
The requestUrl will go to the servlet below:
public class ValidateSessionServlet extends HttpServlet {
String session = req.getParameter(sessionId);
if (session == null) {
// redirect to servlet which will display login page.
response.setContentType("text/html");
String actionUrl = getIpPortUrl(request)
+ PropertyConfig.getInstance().getFromIdPConfig(globalStrings.getCheckSSOSession());
out.write("<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\"> \n");
out.write("<html><head><body onload=\"document.forms[0].submit()\">\n");
out.write("<form method=\"POST\" action=\"" + actionUrl + "\">\n");
out.write("<input type=\"hidden\" name=\"locale\" value=\"" + locale + "\"/>\n");
out.write("<input type=\"hidden\" name=\"Sessionrequest\" value=\"" + true + "\"/>\n");
out.write("</form>\n</body>\n</html>\n");
}
}
In the above code the form should go to the servlet as mentioned in the actionUrl, but it is again going to servlet which is in step(1).
1) May i know can we make this above html form in step(3) to submitted and redirect to the servlet in actionUrl.
As per the above code i am summarizing the requirement. If the session is null, I have to redirect the user to login page and validated against database and then the response should go to step(1), Is it possible?

If you want your HttpUrlConnection to support redirections, you need to set your HttpUrlConnection like this:
...
conn.setRequestProperty("User-agent", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1");
conn.setInstanceFollowRedirects(true);
...
Then if your server redirect your request somewhere else, conn will receiver the redirected response.

To clarify, setInstanceFollowRedirects(true) only dictates whether HTTP redirects should be automatically followed by the HttpURLConnection instance. In your particular case, it seems that you want to redirect to servlets based on whether session is null (or some other condition based on your specific application logic).
The correct (and more bug-proof solution) is to check for HTTP 3xx status code cases and manually handle the redirect. Here is a code snippet as an example:
if (responseStatusCode != HttpURLConnection.HTTP_OK) {
switch(responseStatusCode){
case HttpURLConnection.HTTP_MOVED_TEMP:
// handle 302
case HttpURLConnection.HTTP_MOVED_PERM:
// handle 301
case HttpURLConnection.HTTP_SEE_OTHER:
String newUrl = conn.getHeaderField("Location"); // use redirect url from "Location" header field
String cookies = conn.getHeaderField("Set-Cookie"); // if cookies are needed (i.e. for login)
// manually redirect using a new connection
conn = (HttpURLConnection) new URL(newUrl).openConnection();
conn.setRequestProperty("Cookie", cookies);
conn.addRequestProperty("User-agent", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1");
default:
// handle default (other) case
}
}
The above code is similar to what I use for my app's user login redirects, and I've found that it's very easy to debug. (In general, I handle HTTP status code cases manually in order to avoid bugs down the road.)
Finally, I would recommend using a good JSON lib, such as json.org, to parse your responses.