We Keep Coding

Getting "java.io.IOException: Server returned HTTP response code: 403 for URL: " in java code

I am trying to validate the linkedIn profile of 100K person and wrote a dummy code but its giving "java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user"
I have tried setting different setRequestProperty but not working.
public static void main(final String[] args) {
String output = "";
int TIMEOUT_VALUE = 99999999;
HttpURLConnection conn = null;
BufferedReader br = null;
String urlEndPoint = "";
String authUser = "";
String authPwd = "";
try {
long start = System.nanoTime();
urlEndPoint = "https://www.linkedin.com/in/test.user";
authUser = "linkedin-username";
authPwd = "linkedin-password";
URL url = new URL(urlEndPoint);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty("username", authUser);
conn.setRequestProperty("password", authPwd);
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setRequestProperty("Keep-Alive", "header");
conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
conn.setConnectTimeout(TIMEOUT_VALUE);
conn.setReadTimeout(TIMEOUT_VALUE);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept-Language", "en-US,en;q=0.9,mt;q=0.8");
conn.setRequestProperty("Accept-Encoding", "gzip,deflate,br");
conn.setRequestProperty("Host", "www.linkedin.com");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36");
conn.setRequestProperty("http.agent", "Chrome/71.0.3578.80 (Windows NT 10.0; Win64; x64)");
conn.setDoOutput(true);
String userPassword = authUser + ":" + authPwd;
String encoding = Base64Encoder.encode(userPassword);
conn.setRequestProperty("Authorization", "Basic " + encoding);
OutputStream os = conn.getOutputStream();
os.flush();
conn.connect();
br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
while ((output = br.readLine()) != null) {
System.out.println(output);
}
if (br != null) {
br.close();
}
if (os != null) {
os.close();
}
long elapsed = System.nanoTime() - start;
} catch (MalformedURLException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (IOException e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
} finally {
try {
if (conn != null) {
conn.disconnect();
}
} catch (Exception e) {
//this.logger.error("Error occurred during processPartyTerrRelationship ", e);
e.printStackTrace();
}
}
//logger.info("processPartyTerrRelationship called ends");
}
The outcode of above code is :
java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.linkedin.com/in/test.user
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at ValidateLinkedInProfiles.main(ValidateLinkedInProfiles.java:57)

The HTTP error code 403 is an error related to the authorization to the requested resource:
HTTP 403 provides a distinct error case from HTTP 401; while HTTP 401 is returned when the client has not authenticated, and implies that a successful response may be returned following valid authentication, HTTP 403 is returned when the client is not permitted access to the resource for some reason besides authentication

It's hard to understand how you're working. The LinkedIn link requires login. But you indeed need to debug it somehow and need a raw real output to the server with the correct one otherwise you will not complete it. If you have Java example program, see if they or you have a typo, but again without screenshot or text from LinkedIn I cannot debug it. Maybe try to add the examples and I will try to help you (just make me login with my public profile to other places). Also make sure there is your real password and your user account in the correct fields of course (authUsr,authPwd shall not be copy paste unlike everything else).

HTTP 403 is a legitimate response from a server. So the behavior is valid. However, I would recommend to use some HTTP client utility rather then writing your own code to make Http request. This will reduce the chance of a problem caused by your own code. As some Http clients I would suggest Apache Http Client or OK Http client or MgntUtils Http Client (see MgntUtils HttpClient javadoc here, Full MgntUtils library on github is here and Maven repository is here). Disclaimer: MgntUtils library is written by me

HTTP 403 is a standard HTTP status code communicated to clients by an HTTP server to indicate that the server understood the request, but will not fulfill it. There are a number of sub-status error codes that provide a more specific reason for responding with the 403 status code.
You either do not have access to the site(try logging in from a browser and try to run the script from the same browser, if your access is shared across different tabs of the same browser that is also fine, but make sure you're authorized) or the request to the link contains sensitive information which the site doesn't want to share.

SOAP message to webservice - HTTP response code: 403 for URL

I try to send a SOAP message in an XML file to a webservice and than grab the binary output and decode it. Endpoint uses HTTPS protocol, so I used TrustManager in my code to avoid PKIX problems. You can see my code here:
import javax.net.ssl.*;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.security.cert.X509Certificate;
public class Main{
public static void sendSoapRequest() throws Exception {
String SOAPUrl = "URL HERE";
String xmlFile2Send = ".\\src\\request.xml";
String responseFileName = ".\\src\\response.xml";
String inputLine;
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} };
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
// Create all-trusting host name verifier
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) { return true; }
};
// Install the all-trusting host verifier
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
// Create the connection with http
URL url = new URL(SOAPUrl);
URLConnection connection = url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) connection;
FileInputStream fin = new FileInputStream(xmlFile2Send);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
copy(fin, bout);
fin.close();
byte[] b = bout.toByteArray();
StringBuffer buf=new StringBuffer();
String s=new String(b);
b=s.getBytes();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length", String.valueOf(b.length));
httpConn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
httpConn.setRequestProperty("SOAPAction", "");
httpConn.setRequestMethod("POST");
httpConn.setDoOutput(true);
OutputStream out = httpConn.getOutputStream();
out.write(b);
out.close();
// Read the response.
httpConn.connect();
System.out.println("http connection status :"+ httpConn.getResponseMessage());
InputStreamReader isr = new InputStreamReader(httpConn.getInputStream());
BufferedReader in = new BufferedReader(isr);
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
FileOutputStream fos=new FileOutputStream(responseFileName);
copy(httpConn.getInputStream(),fos);
in.close();
}
public static void copy(InputStream in, OutputStream out) throws IOException {
synchronized (in) {
synchronized (out) {
byte[] buffer = new byte[256];
while (true) {
int bytesRead = in.read(buffer);
if (bytesRead == -1)
break;
out.write(buffer, 0, bytesRead);
}
}
}
}
public static void main(String args[]) throws Exception {
sendSoapRequest();
}
}
I get following error code, when I execute this.
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL

Your implementation is alright, the problem is related to your Content-Type header, in fact.
The value text/xml; charset=utf-8 is the default Content-Type of SOAP 1.1, which is probably not the version of yours. SOAP 1.2 expects a header of type application/soap+xml; charset=utf-8, so changing your line of code to this one below is gonna make it working:
httpConn.setRequestProperty("Content-Type", "application/soap+xml; charset=utf-8");
In SoapUI, it's possible to check the headers calling the request and going to the Headers tab on the bottom of the window:
Then, you can compare the differences between your application configs and the SoapUI ones.

403 error might be related to your soap request headers being sent to the server.
All Hosts valid will allow your Java App to trust the SSL Cert for the URL.
Check if your server is expecting soap header with username/password. If you have access to this server, you can check through the web server logs on where your request is failing. Error code points to to missing Soap Header particularly Soap Headers with username and password

Wonder if your SOAP request contains any kind of authentication information in headers like SAML. One option is, in your above code where you read the file and send the data to server, instead of sending it to server you dump it to another file. Dump that byteoutputstream. Then copy text from that file and put it in SOAP UI and try running that. Does that work?

In a similar situation we have been some time before, and as long as trying TrustManager didn't work as expected, we managed to overcome this problem by installing the certificate from server to JVM's keystore (JVM used to run the application). More information about how to do it you can find in several posts, like
How to import a .cer certificate into a java keystore?
I am aware that it is a try to force JVM to accept SSL certificates, and this functionality would be better to live in application context, but as long as we were building a web application which ran in specific application servers, the implemented solution was an accepted one.

cURL GET request with body in Java

I have the following curl request:
curl -X GET http://hostname:4444/grid/api/hub -d '{"configuration":["slotCounts"]}'
which returns a JSON object.
How can I make such request and get the response in Java? I tried this:
URL url = new URL("http://hostname:4444/grid/api/hub -d '{\"configuration\":[\"slotCounts\"]}'");
try (BufferedReader reader = new BufferedReader(new InputStreamReader(
url.openStream(), "UTF-8"))) {
for (String line; (line = reader.readLine()) != null;) {
System.out.println(line);
}
}
But it returns an exception:
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://hostname:4444/grid/api/hub -d '{"configuration":["slotCounts"]}'

Based on the comments, managed to solve it myself.
private static class HttpGetWithEntity extends
HttpEntityEnclosingRequestBase {
public final static String METHOD_NAME = "GET";
#Override
public String getMethod() {
return METHOD_NAME;
}
}
private void getslotsCount() throws IOException,
URISyntaxException {
HttpGetWithEntity httpEntity = new HttpGetWithEntity();
URL slots = new URL("http://hostname:4444/grid/api/hub");
httpEntity.setURI(pendingRequests.toURI());
httpEntity
.setEntity(new StringEntity("{\"configuration\":[\""
+ PENDING_REQUEST_COUNT + "\"]}",
ContentType.APPLICATION_JSON));
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = client.execute(getPendingRequests);
BufferedReader rd = new BufferedReader(new InputStreamReader(response
.getEntity().getContent()));
// At this point I can just get the response using readLine()
System.out.println(rd.readLine());
}

That's not how sending data in Java works. The -d flag is for the CURL CLI only. In Java you should use a library like Apache HTTP Client:
https://stackoverflow.com/a/3325065/5898512
Then parse the result with JSON: https://stackoverflow.com/a/5245881/5898512

As per your exception/error log, it clearly says that the service http://hostname:4444/grid/api/hub is receiving bad request(Status code 400).
And i think you need to check the service to which you are hitting and what exactly it accepts. Ex: the service may accept only application/json / application/x-www-form-urlencoded or the parameter to service that expecting but you are not sending that.

GlassFish 3 - 400 Bad Request on GET/POST/PUT/DELETE

In a script I have, I've created a small and simple REST client. The script itself is a prototype, and therefore the code is not 'production worthy' - so ignore lazy catch expressions and alike.
There are two types of servers that contain the REST service that I fetch data from; either a WildFly 8.2.0 or a GlassFish 3.1.2.2. And the catch here is: My REST client works fine for fetching data from the Wildfly server, but the GlassFish server returns an HTTP 400 Bad Request, for any request.
I can access the REST service for both servers through a web browser, so I know that they are both working properly. I can even do a raw connection though a socket to both servers and they response with the correct data.
So, what could be the reason for GlassFish to not accept the requests?
Socket connection (for testing)
import java.net.Socket;
Socket s = new Socket("localhost", 8080);
String t = "GET /rest/appointment/appointments/search/?fromDate=2016-11-21&branchId=3 HTTP/1.1\nhost: localhost:8080\nAuthorization: Basic base64encodedUsername:PasswordHere\n\n"
OutputStream out = s.getOutputStream();
out.write(t.getBytes());
InputStream inn = s.getInputStream();
Scanner scan = new Scanner(inn);
String line;
while ((line = scan.nextLine()) != null) {
println line;
}
s.close();
REST client code:
import groovy.json.JsonSlurper;
import javax.xml.bind.DatatypeConverter;
/*
REST-client (a very simple one)
*/
public class RESTclient {
public static Object get(URL url, Map<String, String> headers) {
return http(url, "GET", null, headers);
}
public static Object post(URL url, String data, Map<String, String> headers) {
return http(url, "POST", data, headers);
}
public static Object put(URL url, String data, Map<String, String> headers) {
return http(url, "PUT", data, headers);
}
public static Object delete(URL url, String data, Map<String, String> headers) {
return http(url, "DELETE", data, headers);
}
private static Object http(URL url, String method, String data, Map<String, String> headers) {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("username", "password".toCharArray());
}
});
connection.setRequestMethod(method);
for (String header : headers.keySet()) {
connection.setRequestProperty(header, headers.get(header));
}
if (data != null) {
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
OutputStream outputStream =connection.getOutputStream();
outputStream.write(data.getBytes());
}
int responseCode = connection.getResponseCode();
switch (responseCode) {
case HttpURLConnection.HTTP_NO_CONTENT:
// This happens when the server doesn't give back content, but all was ok.
return (new HashMap());
case HttpURLConnection.HTTP_OK:
InputStream inputStream = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String response = reader.readLine();
JsonSlurper parser = new JsonSlurper();
Object jsonResponse = parser.parseText(response); // This can be either a List or a Map
// Close the connection
try { connection.close(); } catch (Exception e) { /* Already closed */ }
return jsonResponse;
default:
println "response code: " + responseCode;
println connection.getResponseMessage();
println connection.getHeaderFields();
// Close the connection
try { connection.close(); } catch (Exception e) { /* Already closed */ }
return null;
}
}
}
Usage:
URL appointmentSearchURL = new URL("http://localhost:8080/rest/appointment/appointments/search/?fromDate=2016-11-21&branchId=3");
Object response = RESTclient.get(appointmentSearchURL, new HashMap<String, String>());
println response;
All that is printed out:
response code: 400
Bad Request
[null:[HTTP/1.1 400 Bad Request], Server:[GlassFish Server Open Source Edition 3.1.2.2], Connection:[close], Set-Cookie:[rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Tue, 22-Nov-2016 08:43:29 GMT, SSOcookie=2a86cf4b-a772-435a-b92e-f12845dc20a2; Path=/; HttpOnly], Content-Length:[1090], Date:[Wed, 23 Nov 2016 08:43:28 GMT], Content-Type:[text/html], X-Powered-By:[Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2.2 Java/Oracle Corporation/1.7)]]
null

I found my answer! So, I will leave this here if any other stumble across the same issue in the future:
There was a missing Accept header, I guess the server-side only accept json content. I have not researched further on why the WildFly server does not response with a 400 bad request, but I suppose WildFly tries to guess/deduce the incoming data.
So the whole issue was resolved by adding the following:
connection.setRequestProperty("Accept", "application/json");

Http POST in BlackBerry

Greetings,
I am trying to setup a server connection from my BlackBerry Application . I was able to get a response code on the status of the server. Now i have a few values which i have to POST to the server
Its like a registration page values(username, password, age ) have to be sent to the server .
ConnectionFactory connFact = new ConnectionFactory();
ConnectionDescriptor connDesc;
connDesc = connFact.getConnection(url);
if (connDesc != null)
{
HttpConnection httpConn;
httpConn = (HttpConnection)connDesc.getConnection();
try
{
final int iResponseCode = httpConn.getResponseCode();
UiApplication.getUiApplication().invokeLater(new Runnable()
{
public void run()
{
Dialog.alert("Response code: " + Integer.toString(iResponseCode));
}
});
}
catch (IOException e)
{
System.err.println("Caught IOException: " + e.getMessage());
}
}
Thats the code i used to get the response code.
I would appreciate it if someone could help me how i can make a POST request to the server..
the server url for status was company.com/app/version/stats
when it for register it would be
company.com/app/register
Thank you

What type of a POST do you use? If you are just passing key-value pairs, then it should be a POST of a "application/x-www-form-urlencoded" content-type.
So, what lacks youe code is:
1). Set a proper content-type on your connection:
httpConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
2). Prepare the content to be sent to the server via the POST:
URLEncodedPostData encPostData = new URLEncodedPostData("UTF-8", false);
encPostData.append("username", username);
encPostData.append("password", password);
encPostData.append("age", age);
byte[] postData = encPostData.toString().getBytes("UTF-8");
3). Set content-length for the connection (this step may be optional - try without this first, probably the BB OS is smart enough to set this automatically):
httpConn.setRequestProperty("Content-Length", String.valueOf(postData.length));
4). Open an OutputStream and write the content to it (the code is simplified):
OutputStream os = httpConn.openOutputStream();
os.write(postData);
os.flush();

...
httpConn = (HttpConnection)connDesc.getConnection();
httpConn.setRequestMethod(HttpConnection.POST);
httpConn.setRequestProperty("username",name);
httpConn.setRequestProperty("password",pass);
....