We Keep Coding

GET Request to a url (to accept terms and conditions) returns the same page as the response body but works in POSTMAN

There is a page I need to access after accepting the Terms and Conditions in my crawler. However, even after using the url in the source code:
'/auth/submitterms.do?randomNum=' + randomNum
the response body returned is the same terms and conditions page.
When I do the same in Postman it works fine and takes me to the next
randomNum is obtained using regex from the response body. I used the cookieHandler API to handle the session.
Code Snippet:
GET request to terms and conditions page:
connection = new URL(URL_NEXIS+"auth/ipmdelegate.do").openConnection();
res = connection.getInputStream();
try (Scanner scanner = new Scanner(res)){
response = scanner.useDelimiter("\\A").next();
System.out.println("Terms Page: \n"+response);
}
Regex used to obtain the randomNum from the response:
Pattern pattern = Pattern.compile("randomNum=[0-9].[0-9]*");
Matcher matcher = pattern.matcher(response);
if (matcher.find()) {
System.out.println(matcher.group());
randomNum = matcher.group().split("=")[1];
System.out.println(randomNum.toString());
} else {
throw new IOException("Error: Could not accept terms and conditions.");
}
GET request to URL which 'accepts' the terms.
System.out.println(URL_NEXIS + "auth/submitterms.do?randomNum=" + randomNum);
redirect_page = URL_NEXIS + "auth/submitterms.do?randomNum=" + randomNum;
connection = new URL(redirect_page).openConnection();
connection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
connection.setRequestProperty("Accept-Encoding", "gzip, deflate, br");
connection.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
connection.setRequestProperty("Connection","keep-alive");
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36");
res = connection.getInputStream();
How can this be achieved and why is the same request behaving differently with Postman?
EDIT:
I used connection.setInstanceFollowRedirect(false); and was able to obtain a 302 response code. I can now see the location header and set-cookie variable. However, I think the cookieHandler api already handles the session.
Now when I try to send a GET request to the new Url from the location header, I get taken back to the terms and conditions page.

403 forbidden for url in java but not in browser

I'm behind a corporate firewall, but i can paste the URL in my browser with and without my proxy settings enabled within the browser and can retrieve the data fine. I just can't within java.
Any ideas?
Code:
private static String getURLToString(String strUrl) throws IOException {
// LOG.debug("Calling URL: [" + strUrl + "]");
String content = "";
URLConnection connection = new URL(strUrl).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
String inputLine;
while ((inputLine = br.readLine()) != null) {
content += inputLine;
}
br.close();
return content;
}
Error:
java.io.FileNotFoundException: Response: '403: Forbidden' for url: '<url here>'
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:778)
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
Note: The '' portion is for anonymizing.

As you are receiving a "403: Forbidden" error, it means that your Java code can reach the URL, but it lacks something that is required to access it.
In the browser, press F12 (developer/debug mode) and request the URL again. Check the headers and cookies that are being sent. Most likely you will need to add one of these for you to be able to receive the content you need.

Adding "User-Agent" header fixed it for me:
connection.setRequestProperty("User-Agent", "Mozilla/5.0");

curl command line vs HttpURLConnection API service call response time differences

I am consuming a web services using HttpURLConnection which took 60 seconds to return the response but when I use CURL(command line) for same operation with same parameters then it took only 20 - 25 seconds to return the response.
what could be the issue in API service call through HttpURLConnection because it's taking longer time to return the response.
HttpURLConnection API call code :
`
url = new URL(this._serviceURL);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
connection.setRequestProperty("Accept", "application/xml;");
connection.setRequestProperty("SOAPAction", "http://www.xxtest.com/Request");
connection.setDoInput(true);
connection.setDoOutput(true);
// Send request
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(xmlRequest);
wr.flush();
wr.close();
// Get Response
responseCode = connection.getResponseCode();
String xmlResponse = "";
if (responseCode == HttpURLConnection.HTTP_OK) { // success
is = connection.getInputStream();
xmlResponse = IOUtils.toString(is);
// Decode base64 and Decompress
final GZIPInputStream gzipInputStream = new GZIPInputStream(new ByteArrayInputStream(Base64.decodeBase64(xmlResponse.getBytes())));
xmlResponse = IOUtils.toString(gzipInputStream);
}`
CURL command :
curl -XPOST -H "Content-type: text/xml" -H "SOAPAction: http://www.xxtest.com/Request" -H "Accept: application/xml;" -d #request_soap.xml 'http://www.xxtest.com/xmlservices.asmx' > response.xml
Update :
Above mentioned HttpURLConnection API call java code - when executed from a Web Application (Tomcat) then it's taking longer time(60 seconds) to return the response but when I run the same java code as standalone java program on same server then it is returning response in 20 seconds. Exactly same code. Now, I don't understand why the same code is taking longer time when it is getting executed from a Web Application.

The performance problem might be at this point
DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
wr.writeBytes(xmlRequest);
wr.close();
I would guess following will get better Performance.
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
out.write(xmlRequest);
out.close();

I have added Proxy.NO_PROXY in openConnection method in a HttpURLConnection API call code(mentioned in my question) and delayed response issue solved. The services which I was consuming were having proxy setting and because of that I was facing delayed response issue.
Please check below mentioned code snippet.
connection = (HttpURLConnection) url.openConnection(Proxy.NO_PROXY);

HttpURLConnection sends a POST request even though httpCon.setRequestMethod("GET"); is set

Here is my code:
String addr = "http://172.26.41.18:8080/domain/list";
URL url = new URL(addr);
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setDoOutput(true);
httpCon.setDoInput(true);
httpCon.setUseCaches(false);
httpCon.setAllowUserInteraction(false);
httpCon.setRequestMethod("GET");
httpCon.addRequestProperty("Authorization", "Basic YWRtaW4fYFgjkl5463");
httpCon.connect();
OutputStreamWriter out = new OutputStreamWriter(httpCon.getOutputStream());
System.out.println(httpCon.getResponseCode());
System.out.println(httpCon.getResponseMessage());
out.close();
What I see in response:
500 Server error
I open my httpCon var, and what I see:
POST /rest/platform/domain/list HTTP/1.1
Why is it set to POST even though I have used httpCon.setRequestMethod("GET"); to set it to GET?

The httpCon.setDoOutput(true); implicitly set the request method to POST because that's the default method whenever you want to send a request body.
If you want to use GET, remove that line and remove the OutputStreamWriter out = new OutputStreamWriter(httpCon.getOutputStream()); line. You don't need to send a request body for GET requests.
The following should do for a simple GET request:
String addr = "http://172.26.41.18:8080/domain/list";
URL url = new URL(addr);
HttpURLConnection httpCon = (HttpURLConnection) url.openConnection();
httpCon.setUseCaches(false);
httpCon.setAllowUserInteraction(false);
httpCon.addRequestProperty("Authorization", "Basic YWRtaW4fYFgjkl5463");
System.out.println(httpCon.getResponseCode());
System.out.println(httpCon.getResponseMessage());
See also:
Using java.net.URLConnection to fire and handle HTTP requests
Unrelated to the concrete problem, the password part of your Authorization header value doesn't seem to be properly Base64-encoded. Perhaps it's scrambled because it was examplary, but even if it wasn't I'd fix your Base64 encoding approach.

Need Java Expert Help on Executing HTTP GET with Cookie

My problem is that I want to use Java to implement an application which sends an HTTP GET request to some website. However, the target website needs one cookie to be set:
ShippingCountry=US
If this cookie is not set it returns bad indications. Below are my code segment and I get null from connect().
String urlString = "http://www1.macys.com/catalog/index.ognc?CategoryID=5449&viewall=true";
try{
URL url = new URL(urlString);
URLConnection connection = url.openConnection();
connection.addRequestProperty("Cookie", "ShippingCountry=US");
connection.connect();
// Create file
FileWriter fstream = new FileWriter("d:/out.txt");
BufferedWriter out = new BufferedWriter(fstream);
BufferedReader rd = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;
while ((line = rd.readLine()) != null)
{
out.write(line);
}
rd.close();
//Close the output stream
out.close();
}
Can somebody help me?

Just a guess but perhaps you might need setRequestProperty instead of addRequestProperty, since there can be only one Cookie string in a request.
connection.setRequestProperty("Cookie", "ShippingCountry=US");
If you need to send multiple cookie values you tack them on with colons:
connection.setRequestProperty("Cookie", "ShippingCountry=US;OtherValue=2");
Update:
I tried doing a GET request using python and it looks like I get a 500 error. Perhaps the server is blocking the request because it doesn't look like a web browser. Maybe adding the following headers will work:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: keep-alive
In particular the Accept and User-Agent headers might be required.