I am trying to retrieve the final location of a given URL (String ref) as follows:
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
int responseCode = con.getResponseCode();
return con.getURL().toString();
It works in most cases, but rarely returns a URL which yet contains another redirection.
What am I doing wrong here?
Why do I get responseCode = 3xx, even after calling setInstanceFollowRedirects(true)?
UPDATE:
OK, responseCode can sometimes be 3xx.
If it happens, then I will return con.getHeaderField("Location") instead.
The code now is:
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
int responseType = con.getResponseCode()/100;
while (responseType == 1)
{
Thread.sleep(10);
responseType = con.getResponseCode()/100;
}
if (responseType == 3)
return con.getHeaderField("Location");
return con.getURL().toString();
Will appreciate comment should anyone see anything wrong with the code above.
UPDATE
Removed the handling of code 1xx, as according to most commenters it is not necessary.
Testing if the Location header exists before returning it, in order to handle code 304.
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
if (con.getResponseCode()/100 == 3)
{
String target = con.getHeaderField("Location");
if (target != null)
return target;
}
return con.getURL().toString();
HttpURLConnection will not follow redirects if the protocol changes, such as http to https or https to http. In that case, it will return the 3xx code and you should be able to get the Location header. You may need to open a connection again in case that new url also redirects. So basically, use a loop and break it when you get a non-redirect response code. Also, watch out for infinite redirect loops, you could set a limit for the number of iterations or check if each new url has been visited already.
If you just want the redirect url, the response header should give you that:
if (con.getResponseCode() == 301) {
String redirectUrl = con.getHeaderField("Location");
}
There probably can easily be multiple levels of redirection - imagine a bit.ly pointing to a youtu.be address pointing to youtube.com. Perhaps you need to loop until you get your 200 OK or until you hit a redirection cycle.
I have trouble locating the source code to check but I believe what I said is true. See e.g. java urlconnection get the final redirected URL
You also might need to handle protocol redirects, e.g. HTTP -> HTTPS: URLConnection Doesn't Follow Redirect
I think I now understand what you want. I now think that you are trying to retrieve the final address, not the content of the final address. Please correct me if my assumption is wrong.
For doing this (not the content, but the address), you need a different approach. You need to switch off follow-redirects and you then need to handle the iterational redirect-following on your own until you find a non-redirecting response. Bear in mind that you can not reuse a URLConnection.
The approaches for finding the final address and the other approach for retrieving the content of the final address are so different, because URLConnection does not reveal the followed-to address if you switch on follow-redirects.
In your code, you seem to expect URLConnection.getURL() to return the followed-to address. This is not the behavior of this method. It returns the original URL which you used to create the URLConnection. It does this no matter if you switch on follow-redirects or not.
However, if you switch it on, you will not be able to get the followed-to URL address. This is because getHeaderField("Location"), with follow-redirects, makes no sense: it returns the redirection-target of the final redirect, which should not exist, since it's the final address.
Sometime it is loading in the field of requestURI. Use like this code:
val declaredField = con.javaClass.getDeclaredField("requestURI")
declaredField.isAccessible=true
val loc = declaredField.get(con).toString()
Related
I want to sent a Http- PUT-Request which holds a JSON-Body, but I do not know how to define that body. I have tried out a lot by know, but always getting the same problem. Btw, I have already tested the put-request in postman, and it worked as it should, but I cannot realize this request in Java.
My latest try Looks like this:
String sURL = "http://localhost/Thingworx/Things/testAC/Properties/*";
HttpURLConnection urlConn;
URL mUrl = new URL(sURL);
urlConn = (HttpURLConnection) mUrl.openConnection();
//query should be the body
String query = String.format("{\"datDate\":%s,\"txtFrom\": %s,\"txtTo\": %s,\"numFlightTimeDecimal\":%f,\"numSeatCapacity\":%d,\"numLoad\":%d,\"numDirt\":%f,\"numTotalFlightTime\":%f,\"numCummulatedDirt\":%f}", javaDate.toString(), txtFrom, txtTo, numFlightTimeDecimal, numSeatCapacity, numLoad, numDirt, numTotalFlightTime, numCumDirt);
urlConn.setRequestMethod("PUT");
urlConn.setRequestProperty("appKey", "1042fdd2-8e85-4de1-9c92-d79ac24c1ffc");
urlConn.addRequestProperty("Content-Type", "application/json");
urlConn.setDoOutput(true);
if (query != null) {
urlConn.setRequestProperty("Content-Length", Integer.toString(query.length()));
urlConn.getOutputStream().write(query.getBytes("UTF8"));
}
System.err.println(urlConn.getResponseCode());
The Response code says 200, which should be fine then. But the PUT-Request doesnt change any values on the server. The request does not show any effect.
Thank you in advance for any Kind help or advice!
Theresa
urlConn.setRequestMethod("PUT");
Sets the request method to PUT.
urlConn.setDoOutput(true);
Enables output and sets the request method to POST. See the Javadoc. You need to do this first, not last.
NB Don't set the content-length. Java will do that for you.
Given a URL (String ref), I am attempting to retrieve the redirected URL as follows:
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(false);
con.setRequestProperty("User-Agent","");
int responseType = con.getResponseCode()/100;
while (responseType == 1)
{
Thread.sleep(10);
responseType = con.getResponseCode()/100;
}
if (responseType == 3)
return con.getHeaderField("Location");
return con.getURL().toString();
I am having several (conceptual and technical) problems with it:
Conceptual problem:
It works in most cases, but I don't quite understand how.
All methods of the 'con' instance are called AFTER the connection is opened (when 'con' is instanciated).
So how do they affect the actual result?
How come calling 'setInstanceFollowRedirects' affects the returned value of 'getHeaderField'?
Is there any point calling 'getResponseCode' over and over until the returned value is not 1xx?
Bottom line, my general question here: is there another request/response sent through the connection every time one of these methods is invoked?
Technical problem:
Sometimes the response-code is 3xx, but 'getHeaderField' does not return the "final" URL.
I tried calling my code with the returned value of 'getHeaderField' until the response-code was 2xx.
But in most other cases where the response-code is 3xx, 'getHeaderField' DOES return the "final" URL, and if I call my code with this URL then I get an empty string.
Can you please advise how to approach the two problems above in order to have a "100% proof" code for retrieving the "final" URL?
Please ignore cases where the response-code is 4xx or 5xx (or anything else other than 1xx / 2xx / 3xx for that matter).
Thanks
Conceptual problems:
0.) Can one URLConnection or HttpURLConnection object be reused?
No, you can not reuse such an object. You can use it to fetch the content of one URL just once. You can not use it to retrieve another URL, nor to fetch the content twice (speaking on the network level).
If you want to fetch another URL or to fetch the URL a second time, you have to call the openConnection() method of the URL class again to instanciate a new connection object.
1.) When is the URLConnection actually connected?
The method name openConnection() is misleading. It only instanciates the connection object. It does not do anything on the network level.
The interaction on the network level starts in this line, which implicitly connects the connection (= the TCP socket under the hood is opened and data is sent and received):
int responseType = con.getResponseCode()/100;
.
Alternatively, you can use HttpURLConnection.connect() to explicitly connect the connection.
2.) How does setInstanceFollowRedirects work?
setInstanceFollowRedirects(true) causes the URLs to be fetched "under the hood" again and again until there is a non-redirect response. The response code of the non-redirect response is returned by your call to getResponseCode().
UPDATE:
Yes, this allows to write simple code if you do not want to bother about the redirects yourself. You can simply switch on to follow redirects and then you can read the final response of the location to which you get redirected as if there was no redirect taking place.
I would be more careful in evaluating the response code. Not every 3xx-code is automatically a kind of redirection. For example the code 304 just stands for "Not modified."
Look at the original definitions here.
I would like to have a second opinion on a small piece of Java code.
Will the method below always return an output string equal to the input string?
private static String func(final String url)
{
HttpURLConnection con = (HttpURLConnection)new URL(url).openConnection();
con.setInstanceFollowRedirects(true);
...
...
return con.getURL().toString();
}
The question refers to all possible scenarios, such as automatic redirection, etc.
If you look at URLConnection.getUrl() implementation, you can see that it returns the original URL passed to the constructor.
HttpURLConnection also doesn't change the original url.
To get the destination URL of a redirect you're supposed to call con.getHeaderField("Location"); - see for example: Retrieve the final location of a given URL in Java
So you get the original URL until you call connect() or some other method that results in establishing a connection.
If you set ((HttpURLConnection)con).setInstanceFollowRedirects(true); then after connect() if it really redirects you'll get the destination URL.
Redirect may not automatically happen for example when the protocol changes (e.g. http -> https).
I am developing a very simple http bot. I am using the javax.net.ssl.HttpsURLConnection class and I have to make multiple requests.
Snippet of code :
HttpURLConnection urlConnection =
(HttpURLConnection) new URL(url+"?"+firstParameters).openConnection();
urlConnection.setRequestProperty("Accept-Charset", "UTF-8");
headerFields = urlConnection.getHeaderFields();
keys = headerFields.keySet();
for(String key : keys){
if(key != null && key.contains("ookie")){
cookies = urlConnection.getHeaderField(key);
break;
}
}
for(String cookie : cookies.split(";")){
if(cookie.contains("JSESSION")){
JSESSION = cookie.split("=")[1];
break;
}
}
document = new InputSource(urlConnection.getInputStream());
parser.setDocument(document);
attributesId.put("name",new ArrayList<String>(Arrays.asList(attributesNames)));
elementsIds.put("INPUT",attributesId);
elements = parser.getValues(elementsIds);
for(String attr : attributesNames){
secondParameters = secondParameters.replaceAll("#r"+index,elements.get(attr));
}
urlConnection.getInputStream().close();
//Second call
urlConnection = (HttpURLConnection) new URL(url2).openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", "JSESSIONID="+JSESSION);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
payload = new PrintWriter(urlConnection.getOutputStream());
payload.print(secondParameters);
payload.flush();
payload.close();
Summarizing the code above, first i do a request without any payload and i am able to see the correct response from the server, but the problem is when i make the second request (now with payload and with the JSESSION cookie), what i receive it his the same response that i received in the first request, it looks like i am making the first request again.
So my question is , what i am doing wrong ?
I just need to open one connection, and then change the headers and payload ?
There is any tutorial related with multiple http requests(with mixed methods , post and get)?
Thanks in advance
I've never used HttpURLConnection before. I usually use Apache's HTTPClient code. There are a lot of docs and tutorials about it on their home page.
Couple of things that I noticed about your code:
You code does not handle multiple Cookie headers on the response. Mine seems to handle that better.
Are you sure that all you need is JSESSION? Maybe there are other cookies you are missing?
Have you debugged your code to make sure that your JSESSION cookie gets set appropriately? I added some trim() calls in my cookie processing code to make sure some spaces didn't slip in there.
I can't see the real value of your secondParameters. I have no idea if they are valid. Have you debugged your code to verify the secondParamters value looks good. You can see in my code what I'm posting to the server. Btw, I'd use a StringBuilder instead of + to build them.
Hope this helps.
can someone kindly suggest what I'm doing wrong here?
I'm trying to get the header location for a certain URL using Java
here is my code:
URLConnection conn = url.openConnection();
String location = conn.getHeaderField("Location");
it's strange since I know for sure the URL i'm refering to return a Location header and using methods like getContentType() or getContentLength() works perfectly
Perhaps Location header is returned as a part of redirect response. If so, URLConnection handles redirect automatically by issuing the second request to the pointed resource, so you need to disable it:
((HttpURLConnection) conn).setInstanceFollowRedirects(false);
EDIT:
If you actually need a URL of the redirect target and don't want to disable redirect handling, you may call getURL() instead (after connection is established).
Just a follow up to axtavt's answer... If the url has multiple redirects, you could do something like this in order to obtain the direct link:
String location = "http://www.example.com/download.php?getFile=1";
HttpURLConnection connection = null;
for (;;) {
URL url = new URL(location);
connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(false);
String redirectLocation = connection.getHeaderField("Location");
if (redirectLocation == null) break;
location = redirectLocation;
}