I am developing a very simple http bot. I am using the javax.net.ssl.HttpsURLConnection class and I have to make multiple requests.
Snippet of code :
HttpURLConnection urlConnection =
(HttpURLConnection) new URL(url+"?"+firstParameters).openConnection();
urlConnection.setRequestProperty("Accept-Charset", "UTF-8");
headerFields = urlConnection.getHeaderFields();
keys = headerFields.keySet();
for(String key : keys){
if(key != null && key.contains("ookie")){
cookies = urlConnection.getHeaderField(key);
break;
}
}
for(String cookie : cookies.split(";")){
if(cookie.contains("JSESSION")){
JSESSION = cookie.split("=")[1];
break;
}
}
document = new InputSource(urlConnection.getInputStream());
parser.setDocument(document);
attributesId.put("name",new ArrayList<String>(Arrays.asList(attributesNames)));
elementsIds.put("INPUT",attributesId);
elements = parser.getValues(elementsIds);
for(String attr : attributesNames){
secondParameters = secondParameters.replaceAll("#r"+index,elements.get(attr));
}
urlConnection.getInputStream().close();
//Second call
urlConnection = (HttpURLConnection) new URL(url2).openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", "JSESSIONID="+JSESSION);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
payload = new PrintWriter(urlConnection.getOutputStream());
payload.print(secondParameters);
payload.flush();
payload.close();
Summarizing the code above, first i do a request without any payload and i am able to see the correct response from the server, but the problem is when i make the second request (now with payload and with the JSESSION cookie), what i receive it his the same response that i received in the first request, it looks like i am making the first request again.
So my question is , what i am doing wrong ?
I just need to open one connection, and then change the headers and payload ?
There is any tutorial related with multiple http requests(with mixed methods , post and get)?
Thanks in advance
I've never used HttpURLConnection before. I usually use Apache's HTTPClient code. There are a lot of docs and tutorials about it on their home page.
Couple of things that I noticed about your code:
You code does not handle multiple Cookie headers on the response. Mine seems to handle that better.
Are you sure that all you need is JSESSION? Maybe there are other cookies you are missing?
Have you debugged your code to make sure that your JSESSION cookie gets set appropriately? I added some trim() calls in my cookie processing code to make sure some spaces didn't slip in there.
I can't see the real value of your secondParameters. I have no idea if they are valid. Have you debugged your code to verify the secondParamters value looks good. You can see in my code what I'm posting to the server. Btw, I'd use a StringBuilder instead of + to build them.
Hope this helps.
Related
I want to sent a Http- PUT-Request which holds a JSON-Body, but I do not know how to define that body. I have tried out a lot by know, but always getting the same problem. Btw, I have already tested the put-request in postman, and it worked as it should, but I cannot realize this request in Java.
My latest try Looks like this:
String sURL = "http://localhost/Thingworx/Things/testAC/Properties/*";
HttpURLConnection urlConn;
URL mUrl = new URL(sURL);
urlConn = (HttpURLConnection) mUrl.openConnection();
//query should be the body
String query = String.format("{\"datDate\":%s,\"txtFrom\": %s,\"txtTo\": %s,\"numFlightTimeDecimal\":%f,\"numSeatCapacity\":%d,\"numLoad\":%d,\"numDirt\":%f,\"numTotalFlightTime\":%f,\"numCummulatedDirt\":%f}", javaDate.toString(), txtFrom, txtTo, numFlightTimeDecimal, numSeatCapacity, numLoad, numDirt, numTotalFlightTime, numCumDirt);
urlConn.setRequestMethod("PUT");
urlConn.setRequestProperty("appKey", "1042fdd2-8e85-4de1-9c92-d79ac24c1ffc");
urlConn.addRequestProperty("Content-Type", "application/json");
urlConn.setDoOutput(true);
if (query != null) {
urlConn.setRequestProperty("Content-Length", Integer.toString(query.length()));
urlConn.getOutputStream().write(query.getBytes("UTF8"));
}
System.err.println(urlConn.getResponseCode());
The Response code says 200, which should be fine then. But the PUT-Request doesnt change any values on the server. The request does not show any effect.
Thank you in advance for any Kind help or advice!
Theresa
urlConn.setRequestMethod("PUT");
Sets the request method to PUT.
urlConn.setDoOutput(true);
Enables output and sets the request method to POST. See the Javadoc. You need to do this first, not last.
NB Don't set the content-length. Java will do that for you.
i'm running a simple java program to get HttpResponse codes, however for some reason not all codes happen to be 200. I find this odd because when checking the network tab for certain URLs like www.reddit.com, the Response is 200, but my program is returning a different value.
The code below...
try{
String urlName = "http://www.reddit.com";
URL url = new URL(urlName);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
String message = connection.getResponseMessage();
System.out.println("Message: " + message);
int code = connection.getResponseCode();
System.out.println(Integer.toString(code));
}
catch(Exception e){
e.printStackTrace();
}
Lastly, is there a reason to set the RequestMethod to GET and connection again? I get the response code whether or not I have that code because the connection executes openConnection();
Goal - make all valid connections return 200
You said that you're seeing a 301 for Reddit and a 302 for Facebook. Those status codes mean that you're getting redirected. Your browser's following them; your code isn't.
Java's built-in HTTP support is not great for end-users. I strongly recommend using a better HTTP client library, such as Apache's HttpClient, or Horizon, which is built on top of Apache (for synchronous requests) and Ning (for async).
Full disclosure: I work for HubSpot; Horizon is one of our open-source libraries.
It would be nice if you posted the error code it did give...
I ran your code myself and the error was 301, meaning moved permanently.
If you go to http://www.reddit.com yourself, you will see that you get redirected to the httpS version of reddit. Changing this in the urlName will fix your problem.
Edit: same goes for facebook as i saw in comments to your question, google does not require https always so that does work.
I have an android app and REST server made in Spring Tool Suite. REST is connecting to database and returns list of Json objects depending on the query in the GET request.
This is my REST response to GET method
#RequestMapping(value = "/short", method = RequestMethod.GET)
public Map<String,List<Temperature>> shortPeriod(#RequestParam(value = "time",required = false)
#DateTimeFormat(pattern="yyyy:MM:dd HH:mm:ss") Date time){
Map<String, List<Temperature>> results = new HashMap<String, List<Temperature>>();
List<Temperature> temperatureList = new ArrayList<Temperature>();
System.out.println("TEST!");
try {
temperatureList = dataServices.getEntityList(time,"short");
results.put("data", temperatureList);
} catch (Exception e) {
e.printStackTrace();
}
return results;
}
and this is how i connect on android
String tempUrl = new String("http://" + IP + ":8080/" + "RestServer/temperature/" + tableName + "?time=" + urlQueryTimestamp);
String encodedUrlTemp = tempUrl.replace(" ", "%20");
String encodedUrl = encodedUrlTemp.replace("-", ":");
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.connect();
I have put test prints and It's working ok till connection.connect();
The url looks something like this
http://192.168.1.106:8080/RestServer/temperature/short?time=2015:06:05%2010:20:03
I'm printing "url" variable before connecting and when I take that url and paste it into chrome I get normal response as I should. But when I'm trying to connect in my app I get "method not allowed". I have found some solutions about that exception and it's usually problem with POST method not implemented on server, but with same url i get normal response on chrome on my computer and even on my mobile chrome on smartphone.
Other thing is that I have one other server with hardcoded json data for testing and i can connect normally there when there is no query in http request, so only this
http://192.168.1.106:8080/RestServer/temperature/short
So my assumption is that chrome escapes some strings internally or sets up some header infromation by itself when using a query, and android doesn't. Server doesn't even records request as one I pasted in code above, it's like im sending wrong url.
One other thing, I tried using url encoder from android library but it escapes some letters that shouldn't be escaped and I read that it escapes it for html, so I'm doing it manually(maybe I'm doing it wrong).
I turned off the firewall, checked ip adress dozen of times and I have no ideas more.
After trying absolutely every solution I found with escapeing, changing library's and reading about URI's and URL's, I have found the solution on android developer documentation page where it says:
HttpURLConnection uses the GET method by default. It will use POST if
setDoOutput(true) has been called. Other HTTP methods (OPTIONS, HEAD,
PUT, DELETE and TRACE) can be used with setRequestMethod(String).
Hope this helps to someone
I am trying to retrieve the final location of a given URL (String ref) as follows:
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
int responseCode = con.getResponseCode();
return con.getURL().toString();
It works in most cases, but rarely returns a URL which yet contains another redirection.
What am I doing wrong here?
Why do I get responseCode = 3xx, even after calling setInstanceFollowRedirects(true)?
UPDATE:
OK, responseCode can sometimes be 3xx.
If it happens, then I will return con.getHeaderField("Location") instead.
The code now is:
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
int responseType = con.getResponseCode()/100;
while (responseType == 1)
{
Thread.sleep(10);
responseType = con.getResponseCode()/100;
}
if (responseType == 3)
return con.getHeaderField("Location");
return con.getURL().toString();
Will appreciate comment should anyone see anything wrong with the code above.
UPDATE
Removed the handling of code 1xx, as according to most commenters it is not necessary.
Testing if the Location header exists before returning it, in order to handle code 304.
HttpURLConnection con = (HttpURLConnection)new URL(ref).openConnection();
con.setInstanceFollowRedirects(true);
con.setRequestProperty("User-Agent","");
if (con.getResponseCode()/100 == 3)
{
String target = con.getHeaderField("Location");
if (target != null)
return target;
}
return con.getURL().toString();
HttpURLConnection will not follow redirects if the protocol changes, such as http to https or https to http. In that case, it will return the 3xx code and you should be able to get the Location header. You may need to open a connection again in case that new url also redirects. So basically, use a loop and break it when you get a non-redirect response code. Also, watch out for infinite redirect loops, you could set a limit for the number of iterations or check if each new url has been visited already.
If you just want the redirect url, the response header should give you that:
if (con.getResponseCode() == 301) {
String redirectUrl = con.getHeaderField("Location");
}
There probably can easily be multiple levels of redirection - imagine a bit.ly pointing to a youtu.be address pointing to youtube.com. Perhaps you need to loop until you get your 200 OK or until you hit a redirection cycle.
I have trouble locating the source code to check but I believe what I said is true. See e.g. java urlconnection get the final redirected URL
You also might need to handle protocol redirects, e.g. HTTP -> HTTPS: URLConnection Doesn't Follow Redirect
I think I now understand what you want. I now think that you are trying to retrieve the final address, not the content of the final address. Please correct me if my assumption is wrong.
For doing this (not the content, but the address), you need a different approach. You need to switch off follow-redirects and you then need to handle the iterational redirect-following on your own until you find a non-redirecting response. Bear in mind that you can not reuse a URLConnection.
The approaches for finding the final address and the other approach for retrieving the content of the final address are so different, because URLConnection does not reveal the followed-to address if you switch on follow-redirects.
In your code, you seem to expect URLConnection.getURL() to return the followed-to address. This is not the behavior of this method. It returns the original URL which you used to create the URLConnection. It does this no matter if you switch on follow-redirects or not.
However, if you switch it on, you will not be able to get the followed-to URL address. This is because getHeaderField("Location"), with follow-redirects, makes no sense: it returns the redirection-target of the final redirect, which should not exist, since it's the final address.
Sometime it is loading in the field of requestURI. Use like this code:
val declaredField = con.javaClass.getDeclaredField("requestURI")
declaredField.isAccessible=true
val loc = declaredField.get(con).toString()
I'm facing this problem with Java. I want to get some HTML informations from a URL. This code was working for so long, but suddenly, it stopped working.
When I access this URL using the browser, it opens with no problem.
The code:
URL site = new URL(this.url);
java.net.URLConnection yc = site.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
String objetivo = "<td height=\"28\" colspan=\"2\"";
while ((inputLine = in.readLine()) != null && !inputLine.contains(objetivo)) {
}
inputLine = in.readLine();
The Exception:
java.io.IOException: Server returned HTTP response code: 500 for URL: http://www.myurl.com
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at Sites.websites.Site1.getData(Site1.java:53)
at util.Util.lerArquivo(Util.java:278)
at util.Util.main(Util.java:983)
What's wrong? Did the host block me?
HTTP status code 500 usually means that the webserver code has crashed. You need to determine the status code beforehand using HttpURLConnection#getResponseCode() and in case of errors, read the HttpURLConnection#getErrorStream() instead. It may namely contain information about the problem.
If the host has blocked you, you would rather have gotten a 4nn status code like 401 or 403.
See also:
How to use URLConnection to fire and handle HTTP requests?
This Status Code 500 is an Internal Server Error. This code indicates that a part of the server (for example, a CGI program) has crashed or encountered a configuration error.
i think the problem does'nt lie on your side, but rather on the side of the Http server.
the resources you used to access may have been moved or get corrupted, or its configuration just may have altered or spoiled
I had this problem i.e. works fine when pasted into browser but 505s when done through java. It was simply the spaces that needed to be escaped/encoded.
The problem must be with the parameters you are passing(You must be passing blank parameters). For example : http://www.myurl.com?id=5&name=
Check if you are handling this at the server you are calling.
Change the content-type to "application/x-www-form-urlencoded", i solved the problem.
You may look within the first server response and see if the server sent you a cookie.
To check if the server sent you a cookie, you can use HttpURLConnection#getHeaderFields() and look for headers named "Set-Cookie".
If existing, here's the solution for your problem. 100% Working for this case!
In my case, I had changed the Content-Type to Accept and it resolved the issue.
URL url = new URL(GET_URL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("Accept", "application/json; charset=utf-8");