We Keep Coding

Getting 403 error during GET request from Java application

I am trying to do a GET request with JAVA client using RestTemplate library, resulting in following error:
Exception in thread "main" org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden
When I am trying to hit the same URL by command line it is working fine. Posting the cURL command and Java code snippet here.
cURL :
curl -X GET -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" https://xxxx-xxxx-xxxx {"key":"value"}
JAVA snippet :
String URL="https://xyz";
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
//setting up the required headers
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
headers.set("Authorization", "Bearer "+accessToken);
HttpEntity<String> entity = new HttpEntity<String>("body",headers);
//get request
ResponseEntity<String> responseEntity = restTemplate.exchange(URL, HttpMethod.GET, entity, String.class);
P.S - Is the issue because of the URL being a HTTPS one instead of HTTP ?

Add the user agent header, try and let us know if it works.
String URL="https://xyz";
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
//setting up the required headers
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
//settting user agent
headers.add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
headers.set("Authorization", "Bearer "+accessToken);
HttpEntity<String> entity = new HttpEntity<String>("body",headers);
//get request
ResponseEntity<String> responseEntity = restTemplate.exchange(URL, HttpMethod.GET, entity, String.class);

Can get response from browser, but 401 unauthorized from resttemplate

Can get from the response from Chrome, but when I want to get it with resttemplate it gives me 401 response.
I try to add the exact Content Types with rest template but it still gives the 401 error.
here is the request that i get it from Chrome :
GET URL HTTP/1.1
Host: host
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic auth
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en,de;q=0.9,tr;q=0.8
and here is the code that i used in java to connect the service :
RestTemplate restTemplate = new RestTemplate();
List<HttpMessageConverter<?>> messageConverters = new ArrayList<>();
MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
converter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_XHTML_XML, MediaType.APPLICATION_XML, MediaType.TEXT_HTML));
messageConverters.add(converter);
restTemplate.setMessageConverters(messageConverters);
HttpEntity<Map<String, String>> entity = new HttpEntity<>(params,createHeaders(username, password));
restTemplate.getForObject("URL",String.class, entity);
public static HttpHeaders createHeaders(String username, String password) {
String auth = username + ":" + password;
byte[] encodedAuth = Base64.encodeBase64(auth.getBytes(Charset.forName("US-ASCII")));
String authHeader = "Basic " + new String(encodedAuth);
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.set("Authorization", authHeader);
return httpHeaders;
}

It appears to be a problem with encoding the authorization token generation.
If you are sure about the username and password is correct, the only possibility is that the method/encoding used to generate the token is different on the browser(or the UI application which is generating the token) and the implementation of your createHeaders() method.
Please refer to the below links for more details:
What encoding should I use for HTTP Basic Authentication?
https://github.com/request/request/issues/1334
Hope it'll help.

java.io.IOException: 403 error loading URL jsoup login

help me with jsoup login, i have to login, then redirect from login page to another page with saved session.
public void parseXhtml() throws IOException{
String sessionID=null;
Map<String, String> cookies = new HashMap<String, String>();
cookies.put("login", "login");
cookies.put("password", "password");
Connection conn=Jsoup.connect("http://localhost:8080/dir/login.xhtml");
Connection.Response res = Jsoup
.connect("http://localhost:8080/dir/login.xhtml")
.data(cookies)
.execute();
res = Jsoup.connect("http://localhost:8080/dir/dir/index.xhtml")
.cookie("JSESSIONID", res.cookies().get("JSESSIONID"))
.method(Method.GET)
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36")
.execute();
Document doc = res.parse();
System.out.println(doc.html());
sessionID = res.cookie("JSESSIONID");
Document docu = Jsoup.connect("http://localhost:8080/dir/dir/index.xhtml")
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36")
.cookie("JSESSIONID", res.cookies().get("JSESSIONID"))
.method(Connection.Method.GET)
.get();
it throws the below exception
java.io.IOException: 403 error loading URL http://localhost:8080/dir/dir/index.xhtml
if i'm doing like [Sending POST request with username and password and save session cookie it throws the same exception.

ExternalContext ec=FacesContext.getCurrentInstance().getExternalContext();
HttpServletRequest req=(HttpServletRequest) ec.getRequest();
HttpSession sess=(HttpSession) ec.getSession(true);
String url = req.getRequestURL().append(";jsessionid=").append(sess.getId()).toString();
ec.setRequest("http://localhost:8080/dir/dir/index.xhtml");
HttpServletRequest req2=(HttpServletRequest) ec.getRequest();
String url2 = req2.getRequestURL().append(";jsessionid=").append(sess.getId()).toString();
Document doc2=Jsoup.connect(url2).get();
System.out.println(doc2.html());
Finally i got it, I do not know if it is right, but works

Execute GET/POST on Google App Engine using UrlFetchTransport

I'm porting an Android app on GAE.
The app compile and post a login form to enter in a
password protected site.
I'm using UrlFetchTransport instead of DefaultHttpClient.
I can get web page without problem using the following piece of code:
HttpTransport HTTP_TRANSPORT = new UrlFetchTransport();
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
GenericUrl genericUrl = new GenericUrl(url);
HttpRequest request = requestFactory.buildGetRequest(genericUrl);
com.google.api.client.http.HttpHeaders httpHeaders = request.getHeaders();
httpHeaders.setUserAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
httpHeaders.setAccept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
String paginaRicevuta = request.execute().parseAsString();
I would try a POST request using the following:
UrlFetchTransport HTTP_TRANSPORT = new UrlFetchTransport();
HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory();
GenericUrl genericUrl = new GenericUrl("https://www.......");
List<BasicNameValuePair> nvps = new ArrayList<BasicNameValuePair>();
nvps.add(new BasicNameValuePair("ReturnUrl", ""));
nvps.add(new BasicNameValuePair("userName", "username"));
nvps.add(new BasicNameValuePair("password","password"));
//HERE HOW DEFINE httpContent????
HttpContent httpContent=null;
final HttpRequest postRequest = requestFactory.buildPostRequest(genericUrl,httpContent);
postRequest.setContent(httpContent);
postRequest.setFollowRedirects(true);
HttpHeaders httpHeaders = postRequest.getHeaders();
httpHeaders.setUserAgent("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36");
httpHeaders.setAccept("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
httpHeaders.setContentType("application/x-www-form-urlencoded");
postRequest.setHeaders(httpHeaders);
String pagePost = postRequest.execute().parseAsString();
but I don't know how format a right POST request?
Thanks for your help.

IOException - XML Parsing

I am trying to parse an xml response from the below url -
http://imdbapi.org/?type=xml&q=argo
For this, i have written the below code -
try
{
XMLReader myReader = XMLReaderFactory.createXMLReader();
xmlHandler handlerobj = new xmlHandler();
myReader.setContentHandler(handlerobj);
myReader.parse(new InputSource(new URL("http://imdbapi.org/?type=xml&q=argo").openStream()));
}
catch(Exception e)
{
System.out.println("Error");
}
xmlHandler is a class that extends DefaultHandler.
I am getting an IOException in the above code.
Stack trace -
java.io.IOException: Server returned HTTP response code: 403 for URL: http://imdbapi.org/?type=xml&q=argo
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at gui.getimdbdata(gui.java:73)
at gui.main(gui.java:64)
What is the problem with this code ?

You must set the user.agent:
System.setProperty("http.agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
(if you connect to the URL with your browser this is done automagically)

Solved the issue, thanks to #dijkstra !
The web service would only allow browser to fetch the xml data.
Following are the modifications -
url = new URL(urlString);
uc = url.openConnection();
uc.addRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
uc.connect();
uc.getInputStream();
BufferedInputStream in = new BufferedInputStream(uc.getInputStream());