We Keep Coding

Java https request get 403 response code for curseforge site

I try get redirect from https://www.curseforge.com/projects/291874 to https://www.curseforge.com/minecraft/mc-mods/serene-seasons but all tries response 403. I try HttpURLConnection, HttpsURLConnection and URLConnection but nothing to work.
How do request on this site?
Browser do this two steps:
CONNECT www.curseforge.com:443 HTTP/1.1
GET /projects/291874 HTTP/1.1
How do this on java?
Please check your code before answer (first link in question)
String url = "https://www.curseforge.com/projects/291874";
try {
HttpURLConnection con1 = (HttpURLConnection)(new URL(url).openConnection());
con1.setRequestProperty("Host", "www.curseforge.com");
con1.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0");
con1.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con1.setRequestProperty("Accept-Language", "ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3");
con1.setRequestProperty("Accept-Encoding", "gzip, deflate, br");
con1.setRequestProperty("DNT", "1");
con1.setRequestProperty("Connection", "keep-alive");
con1.setRequestProperty("Upgrade-Insecure-Requests", "1");
con1.connect();
//con1.getInputStream();
System.out.println(con1.getResponseCode());
System.out.println(con1.getHeaderField("Location"));
} catch (IOException e) {
e.printStackTrace();
}

org.springframework.web.client.HttpClientErrorException: 403 null (Spring RestTemplate)

I'm trying to send a GET request with RestTemplate. My code looks like:-
RestTemplate template = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
headers.add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36");
headers.add("_rToken", cookie.getValue());
String url = getUrl(contextUrl);
HttpEntity<Object> entity = new HttpEntity<Object>(headers);
ResponseEntity<String> response = null;
try {
response = template.exchange(url, HttpMethod.GET, entity, String.class);
}
catch(RestClientException ex) {
ex.printStackTrace();
}
return response;
On debugging, my URL looks good. What I have as entity is:-
body = null,
headers = <{Accept=[application/json],
user-agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36],
_rToken=[2a6ac90f-1dfb-4df3-8d23-6d8d948fb9b5]}>
Since my get method doesn't take a parameter, body is supposed to be null. What am I doing wrong here?
Stack trace:
org.springframework.web.client.HttpClientErrorException: 403 null at
org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)
at
org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:79)
at
org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at
org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:773)
at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:726)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:682)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:598)
at com.hm.ul.util.PlpUMCommunicator.get(PlpUMCommunicator.java:53)

java.io.IOException: 403 error loading URL jsoup login

help me with jsoup login, i have to login, then redirect from login page to another page with saved session.
public void parseXhtml() throws IOException{
String sessionID=null;
Map<String, String> cookies = new HashMap<String, String>();
cookies.put("login", "login");
cookies.put("password", "password");
Connection conn=Jsoup.connect("http://localhost:8080/dir/login.xhtml");
Connection.Response res = Jsoup
.connect("http://localhost:8080/dir/login.xhtml")
.data(cookies)
.execute();
res = Jsoup.connect("http://localhost:8080/dir/dir/index.xhtml")
.cookie("JSESSIONID", res.cookies().get("JSESSIONID"))
.method(Method.GET)
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36")
.execute();
Document doc = res.parse();
System.out.println(doc.html());
sessionID = res.cookie("JSESSIONID");
Document docu = Jsoup.connect("http://localhost:8080/dir/dir/index.xhtml")
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36")
.cookie("JSESSIONID", res.cookies().get("JSESSIONID"))
.method(Connection.Method.GET)
.get();
it throws the below exception
java.io.IOException: 403 error loading URL http://localhost:8080/dir/dir/index.xhtml
if i'm doing like [Sending POST request with username and password and save session cookie it throws the same exception.

ExternalContext ec=FacesContext.getCurrentInstance().getExternalContext();
HttpServletRequest req=(HttpServletRequest) ec.getRequest();
HttpSession sess=(HttpSession) ec.getSession(true);
String url = req.getRequestURL().append(";jsessionid=").append(sess.getId()).toString();
ec.setRequest("http://localhost:8080/dir/dir/index.xhtml");
HttpServletRequest req2=(HttpServletRequest) ec.getRequest();
String url2 = req2.getRequestURL().append(";jsessionid=").append(sess.getId()).toString();
Document doc2=Jsoup.connect(url2).get();
System.out.println(doc2.html());
Finally i got it, I do not know if it is right, but works

How to get a response body while getting 403. Java

I make a request to my web app, web app reports some errors.
nothing complicated. just like this:
{"password":["can't be blank"]}
and it returns it with code 403.
In my android app I use HttpURLConnection to make a request and I catch an exception white trying to getInputStream.
In that exception I can getResponseMessage(), but all I got is 'Forbidden'
How can I get an actual response body?
HttpURLConnection urlConnection = (HttpURLConnection)new URL(url).openConnection();
urlConnection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0");
try {
String result = IOUtils.toString(urlConnection.getInputStream());
urlConnection.disconnect();
App.SaveToken(result);
}catch(IOException exception){
if (urlConnection.getResponseCode()==403)
{
return urlConnection.getResponseMessage();
}
}

If the response code is not 200(or 2xx), use getErrorStream() instead of getInputStream().

IOException - XML Parsing

I am trying to parse an xml response from the below url -
http://imdbapi.org/?type=xml&q=argo
For this, i have written the below code -
try
{
XMLReader myReader = XMLReaderFactory.createXMLReader();
xmlHandler handlerobj = new xmlHandler();
myReader.setContentHandler(handlerobj);
myReader.parse(new InputSource(new URL("http://imdbapi.org/?type=xml&q=argo").openStream()));
}
catch(Exception e)
{
System.out.println("Error");
}
xmlHandler is a class that extends DefaultHandler.
I am getting an IOException in the above code.
Stack trace -
java.io.IOException: Server returned HTTP response code: 403 for URL: http://imdbapi.org/?type=xml&q=argo
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at gui.getimdbdata(gui.java:73)
at gui.main(gui.java:64)
What is the problem with this code ?

You must set the user.agent:
System.setProperty("http.agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
(if you connect to the URL with your browser this is done automagically)

Solved the issue, thanks to #dijkstra !
The web service would only allow browser to fetch the xml data.
Following are the modifications -
url = new URL(urlString);
uc = url.openConnection();
uc.addRequestProperty("User-Agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
uc.connect();
uc.getInputStream();
BufferedInputStream in = new BufferedInputStream(uc.getInputStream());