We Keep Coding

Getting exception "Connection Reset" when trying to fetch response code using httpurlconnection

I am trying to get response code using httpurlconnection but getting "Connection Reset" every time.
Here is the code I am using,
try {
String url = "https://www.northerntrust.com/asia-pac/home";
HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
conn.setRequestMethod("GET");
conn.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36");
conn.setConnectTimeout(2000);
conn.setInstanceFollowRedirects(false);
conn.setReadTimeout(100000);
conn.connect();
int responseCode = conn.getResponseCode();
} catch(Exception e) {
logger.error("Caught exception : {}", e.getMessage);
}
The exception says : "Stack trace : java.net.SocketInputStream.read(SocketInputStream.java:210)",
Connection reset
Why am I getting this error ?
This is happening for these urls as well :
https://www.comerica.com/business.html
https://www.pbcu.com/

Getting "java.io.IOException: Server returned HTTP response code: 403 for URL: " in java code

I am trying to validate the linkedIn profile of 100K person and wrote a dummy code but its giving "java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user"
I have tried setting different setRequestProperty but not working.
public static void main(final String[] args) {
String output = "";
int TIMEOUT_VALUE = 99999999;
HttpURLConnection conn = null;
BufferedReader br = null;
String urlEndPoint = "";
String authUser = "";
String authPwd = "";
try {
long start = System.nanoTime();
urlEndPoint = "https://www.linkedin.com/in/test.user";
authUser = "linkedin-username";
authPwd = "linkedin-password";
URL url = new URL(urlEndPoint);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("username", authUser);
conn.setRequestProperty("password", authPwd);
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Keep-Alive", "header");
conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
conn.setConnectTimeout(TIMEOUT_VALUE);
conn.setReadTimeout(TIMEOUT_VALUE);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.9,mt;q=0.8");
conn.setRequestProperty("Accept-Encoding", "gzip,deflate,br");
conn.setRequestProperty("Host", "www.linkedin.com");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36");
conn.setRequestProperty("http.agent", "Chrome/71.0.3578.80 (Windows NT 10.0; Win64; x64)");
conn.setDoOutput(true);
String userPassword = authUser + ":" + authPwd;
String encoding = Base64Encoder.encode(userPassword);
conn.setRequestProperty("Authorization", "Basic " + encoding);
OutputStream os = conn.getOutputStream();
os.flush();
conn.connect();
br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
while ((output = br.readLine()) != null) {
System.out.println(output);
}
if (br != null) {
br.close();
}
if (os != null) {
os.close();
}
long elapsed = System.nanoTime() - start;
} catch (MalformedURLException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (IOException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} finally {
try {
if (conn != null) {
conn.disconnect();
}
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
}
}
//logger.info("processPartyTerrRelationship called ends");
}
The outcode of above code is :
java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at ValidateLinkedInProfiles.main(ValidateLinkedInProfiles.java:57)

The HTTP error code 403 is an error related to the authorization to the requested resource:
HTTP 403 provides a distinct error case from HTTP 401; while HTTP 401 is returned when the client has not authenticated, and implies that a successful response may be returned following valid authentication, HTTP 403 is returned when the client is not permitted access to the resource for some reason besides authentication

It's hard to understand how you're working. The LinkedIn link requires login. But you indeed need to debug it somehow and need a raw real output to the server with the correct one otherwise you will not complete it. If you have Java example program, see if they or you have a typo, but again without screenshot or text from LinkedIn I cannot debug it. Maybe try to add the examples and I will try to help you (just make me login with my public profile to other places). Also make sure there is your real password and your user account in the correct fields of course (authUsr,authPwd shall not be copy paste unlike everything else).

HTTP 403 is a legitimate response from a server. So the behavior is valid. However, I would recommend to use some HTTP client utility rather then writing your own code to make Http request. This will reduce the chance of a problem caused by your own code. As some Http clients I would suggest Apache Http Client or OK Http client or MgntUtils Http Client (see MgntUtils HttpClient javadoc here, Full MgntUtils library on github is here and Maven repository is here). Disclaimer: MgntUtils library is written by me

HTTP 403 is a standard HTTP status code communicated to clients by an HTTP server to indicate that the server understood the request, but will not fulfill it. There are a number of sub-status error codes that provide a more specific reason for responding with the 403 status code.
You either do not have access to the site(try logging in from a browser and try to run the script from the same browser, if your access is shared across different tabs of the same browser that is also fine, but make sure you're authorized) or the request to the link contains sensitive information which the site doesn't want to share.

convert curl request into HttpURLConnection

I am trying to implement a client that logins first and do some staff.
This is my curl request:
curl -v https://api.example.com/api-token-auth/ \
-H "Accept: application/json" \
-d "username=myusername&password=mypassword"
I want to convert it into java code. Here is what I have tried:
HttpURLConnection conn;
URL obj = new URL("https://api.example.com/api-token-auth/");
URL obj = new URL(quoteURL);
conn = (HttpURLConnection) obj.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
String userpass = "username=myusername" + "&" + "password=mypassword";
String basicAuth = new String(Base64.getEncoder().encode(userpass.getBytes()));
conn.setRequestProperty("Authorization", basicAuth);
conn.setRequestProperty( "Accept", "*/*" );
conn.setRequestProperty( "Accept-Encoding", "gzip, deflate" );
conn.setRequestProperty( "Accept-Language", "en;q=1, fr;q=0.9, de;q=0.8,ja;q=0.7, nl;q=0.6, it;q=0.5" );
conn.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded; charset=utf-8" );
conn.setRequestProperty( "API-Version", "1.3.0" );
conn.setRequestProperty("Connection", "keep-alive");
conn.setRequestProperty( "Accept", "*/*" );
conn.setRequestProperty("User-Agent", "Mozilla/5.0");
conn.connect();
InputStreamReader inputStreamReader = new InputStreamReader(conn.getInputStream());
BufferedReader in = new BufferedReader(inputStreamReader);
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
conn.disconnect();
return response;
Then I receive this error:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 400 for URL: https://api.example.com/api-token-auth/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
I have tried several possible solutions, but no luck. I could nit find what I am doing wrong.

your curl request isn't actually doing HTTP basicAuth (which is what your example code is trying to do) - it's just POSTing the -d argument to the server (as a url encoded body)
so
get rid of all the setRequestProperty() stuff (it's not needed)
use con.setContentType("application/x-www-form-urlencoded") [which is arguably a bit cleaner]
write the userpass string to con.getOutputStream() [no need to base64 encode...again, this has nothing to do w/ http basicAuth]
for example, your curl command issues the following HTTP request
POST /api-token-auth/ HTTP/1.1
Host: api.example.com
User-Agent: curl/7.49.1
Accept: application/json
Content-Length: 39
Content-Type: application/x-www-form-urlencoded
username=myusername&password=mypassword
the following Java program would execute pretty much the EXACT same request
public class SO {
public static void main(String[] args) throws Exception {
String rsp = curl("http://axrsgpar0019:13080/api-token-auth/", "application/json", "username=myusername&password=mypassword");
}
public static String curl(String url, String accepts, String minusD) throws Exception {
HttpURLConnection con = (HttpURLConnection)new URL(url).openConnection();
con.setDoOutput(true);
con.setRequestProperty("Accept", accepts);
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
con.getOutputStream().write(minusD.getBytes());
con.getOutputStream().close();
ByteArrayOutputStream rspBuff = new ByteArrayOutputStream();
InputStream rspStream = con.getInputStream();
int c;
while ((c = rspStream.read()) > 0) {
rspBuff.write(c);
}
rspStream.close();
return new String(rspBuff.toByteArray());
}
}
generates the following HTTP request (only difference is User-Agent and keep-alive..which should be insignificant)
POST /api-token-auth/ HTTP/1.1
Accept: application/json
Content-Type: application/x-www-form-urlencoded
User-Agent: Java/1.8.0_91
Host: api.example.com
Connection: keep-alive
Content-Length: 39
username=myusername&password=mypassword

How to get a response body while getting 403. Java

I make a request to my web app, web app reports some errors.
nothing complicated. just like this:
{"password":["can't be blank"]}
and it returns it with code 403.
In my android app I use HttpURLConnection to make a request and I catch an exception white trying to getInputStream.
In that exception I can getResponseMessage(), but all I got is 'Forbidden'
How can I get an actual response body?
HttpURLConnection urlConnection = (HttpURLConnection)new URL(url).openConnection();
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0");
try {
String result = IOUtils.toString(urlConnection.getInputStream());
urlConnection.disconnect();
App.SaveToken(result);
}catch(IOException exception){
if (urlConnection.getResponseCode()==403)
{
return urlConnection.getResponseMessage();
}
}

If the response code is not 200(or 2xx), use getErrorStream() instead of getInputStream().

jsoup posting Java

Im struggling getting java submitting POST requests over HTTPS
Code used is here
try{
Response res = Jsoup.connect(LOGIN_URL)
.data("username", "blah", "password", "blah")
.method(Method.POST)
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.execute();
System.out.println(res.body());
System.out.println("Code " +res.statusCode());
}
catch (Exception e){
System.out.println(e.getMessage());
}
and also this
Document doc = Jsoup.connect(LOGIN_URL)
.data("username", "blah")
.data("password", "blah")
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0")
.header("Content-type", "application/x-www-form-urlencoded")
.method(Method.POST)
.timeout(3000)
.post();
Where LOGIN_URL = https://xxx.com/Login?val=login
When used over HTTP it seems to work, HTTPS it doesnt, But doesnt throw any exceptions
How can I POST over HTTPS
Edit:
seems there is a 302 redirect involved when the server gets a POST over HTTPS (which doesnt happen over http) How can I use jsoup to store the cookie sent with the 302 to the next page ?

this is my code:
URL form = new URL(Your_url);
connection1 = (HttpURLConnection)form.openConnection();
connection1.setRequestProperty("Cookie", your_cookie);
connection1.setReadTimeout(10000);
StringBuilder whole = new StringBuilder();
BufferedReader in = new BufferedReader(
new InputStreamReader(new BufferedInputStream(connection1.getInputStream())));
String inputLine;
while ((inputLine = in.readLine()) != null)
whole.append(inputLine);
in.close();
Document doc = Jsoup.parse(whole.toString());
String title = doc.title();
i have used this code to get the title of the new page.

Here is what you can try...
import org.jsoup.Connection;
Connection.Response res = null;
try {
res = Jsoup
.connect("your-first-page-link")
.data("username", "blah", "password", "blah")
.method(Connection.Method.POST)
.execute();
} catch (IOException e) {
e.printStackTrace();
}
Now save all your cookies and make a request to the other page you want.
//Saving Cookies
cookies = res.cookies();
Making a request to another page.
try {
Document doc = Jsoup.connect("your-second-page-link").cookies(cookies).get();
}
catch(Exception e){
e.printStackTrace();
}
Comment if further help needed.