I would like to replace Location header in one condition. I use the following to enable Redirect
client.prepareGet(request.getUrl())
.setFollowRedirect(true)
What I did is to extended AsyncCompletionHandler class, and from there I #Override onHeadersReceived(),
#Override
public State onHeadersReceived(final HttpResponseHeaders headers) throws Exception {
String location = headers.getHeaders().get("Location").replace("itmss", "https"
);
DefaultHttpHeaders httpHeaders = new DefaultHttpHeaders();
httpHeaders.add(HEADER, location);
httpHeaders.add(headers.getHeaders());
return inner.onHeadersReceived(new HttpResponseHeaders(httpHeaders));
}
However, I see that the request still uses itms instead of https, which means the Location header is not replaced.
The current plan is to implement a interceptor that runs before Redirect30xInterceptor.java(https://github.com/AsyncHttpClient/async-http-client/blob/dd459294434a408cff3c65c9f5c402b82d60aaa2/client/src/main/java/org/asynchttpclient/netty/handler/intercept/Redirect30xInterceptor.java), and change the url.
The other plan is to do a while loop for the following code, and whenever receiving 3XX, set a new url in the next request.
client.prepareGet(request.getUrl())
.addHeader("User-Agent", request.getUserAgent())
.setProxyServer(proxyServer)
.execute(new ResponseAsyncCompletionHandler(request))
.toCompletableFuture();
The ultimate goal is to replace itms:// to https://, so the next request uses https instead of itms.
After studying more, okhttp supports interceptor easily.
If wanted to use AsyncHttp it's much more tricky. You have to implement the ResposeFilter, and from there you change the request, and setup replayRequest(true), which means a new request is made.
example code
builder.request(new RequestBuilder(nextRequest).build()).replayRequest(true);
https://github.com/AsyncHttpClient/async-http-client/blob/master/client/src/main/java/org/asynchttpclient/netty/handler/intercept/ResponseFiltersInterceptor.java#L62
I am trying to send a request using jsoup with manually inserted cookies for the purpose of detecting SQL injection vulnerability.
The problem seams to be that only one of the cookies works and i don't understand why.
I first authenticate myself manually and get the cookies. Example:
PHPSESSID : b74302c3c6af62d23047a450a40cbf5a
security : high
After i got the cookies i send my request (which whould look like this from the browser http://localhost:8090/dvwa/vulnerabilities/sqli/?id='&Submit=Submit#) using the same PHPSESSID but with Security : low. The purpose is to force a "You have an error in your SQL syntax" response that signals SQL Injection vulnerability. The problem is that the PHPSESSID is received good (since I retrieve the dvwa/vulnerabilities/sqli page and not Login page thus it recognizes the PHPSESSID as valid after authentication) but the "security : low" seems not to work. I can't find the problem.
The jsoup code for an initial connection so that i can parse the forms on the page looks like this. I supply the cookies manually.
Connection connection = Jsoup.connect(urlDTO.getUrl())
.userAgent(StringConstants.USER_AGENT)
.cookies(cookies)//Map<String,String>
.referrer(StringConstants.REFERRER);
Document htmlDocument = connection.get();
For sending the form i use this code:
Connection connection = Jsoup.connect(formDTO.getUrl())
.userAgent(StringConstants.USER_AGENT)
.cookies(cookies)
.data(listToMap(formDTO.getInputList()))// id = ' , Submit = Submit
.method(getMethod(formDTO.getMethod()))
.referrer(StringConstants.REFERRER);
Connection.Response res = connection.execute();
Document doc = res.parse();
Does anyone know what I'm doing wrong?
After much debugging I fund out the origin of the problem and the odd behavior. There was no problem with the cookies/headers/url, the problem was at the .method(). The default value of .method() is Method.GET. But since i was sending dynamic requests i had to construct this dynamically also. For this purpose i was parsing the forms to get the method and then adding the type needed in the connection construction.
if (method.equals("post")) {
return Method.POST;
}
if (method.equals("get")) {
return Method.GET;
}
return Method.POST;
This worked until now and surprisingly when sending the wrong method it still sent a response almost valid, so i overlooked it.
Here is the fix.
if (method.toLowerCase().equals("post")) {
return Method.POST;
}
if (method.toLowerCase().equals("get")) {
return Method.GET;
}
return Method.GET;
It was my mistake and not relay a Jsoup problem but since I overlooked it others might too so here is a reminder.
I'm building a REST API, but I've encountered a problem.
It seems that accepted practice in designing a REST API is that if the resource requested doesn't exist, a 404 is returned.
However, to me, this adds unnecessary ambiguity. HTTP 404 is more traditionally associated with a bad URI. So in effect we're saying "Either you got to the right place, but that specific record does not exist, or there's no such location on the Internets! I'm really not sure which one..."
Consider the following URI:
http://mywebsite/api/user/13
If I get a 404 back, is that because User 13 does not exist? Or is it because my URL should have been:
http://mywebsite/restapi/user/13
In the past, I've just returned a NULL result with an HTTP 200 OK response code if the record doesn't exist. It's simple, and in my opinion very clean, even if it's not necessarily accepted practice. But is there a better way to do this?
404 is just the HTTP response code. On top of that, you can provide a response body and/or other headers with a more meaningful error message that developers will see.
Use 404 if the resource does not exist. Don't return 200 with an empty body.
This is akin to undefined vs empty string (e.g. "") in programming. While very similar, there is definitely a difference.
404 means that nothing exists at that URI (like an undefined variable in programming). Returning 200 with an empty body means that something does exist there and that something is just empty right now (like an empty string in programming).
404 doesn't mean it was a "bad URI". There are special HTTP codes that are intended for URI errors (e.g. 414 Request-URI Too Long).
As with most things, "it depends". But to me, your practice is not bad and is not going against the HTTP spec per se. However, let's clear some things up.
First, URI's should be opaque. Even if they're not opaque to people, they are opaque to machines. In other words, the difference between http://mywebsite/api/user/13, http://mywebsite/restapi/user/13 is the same as the difference between http://mywebsite/api/user/13 and http://mywebsite/api/user/14 i.e. not the same is not the same period. So a 404 would be completely appropriate for http://mywebsite/api/user/14 (if there is no such user) but not necessarily the only appropriate response.
You could also return an empty 200 response or more explicitly a 204 (No Content) response. This would convey something else to the client. It would imply that the resource identified by http://mywebsite/api/user/14 has no content or is essentially nothing. It does mean that there is such a resource. However, it does not necessarily mean that you are claiming there is some user persisted in a data store with id 14. That's your private concern, not the concern of the client making the request. So, if it makes sense to model your resources that way, go ahead.
There are some security implications to giving your clients information that would make it easier for them to guess legitimate URI's. Returning a 200 on misses instead of a 404 may give the client a clue that at least the http://mywebsite/api/user part is correct. A malicious client could just keep trying different integers. But to me, a malicious client would be able to guess the http://mywebsite/api/user part anyway. A better remedy would be to use UUID's. i.e. http://mywebsite/api/user/3dd5b770-79ea-11e1-b0c4-0800200c9a66 is better than http://mywebsite/api/user/14. Doing that, you could use your technique of returning 200's without giving much away.
That is an very old post but I faced to a similar problem and I would like to share my experience with you guys.
I am building microservice architecture with rest APIs. I have some rest GET services, they collect data from back-end system based on the request parameters.
I followed the rest API design documents and I sent back HTTP 404 with a perfect JSON error message to client when there was no data which align to the query conditions (for example zero record was selected).
When there was no data to sent back to the client I prepared an perfect JSON message with internal error code, etc. to inform the client about the reason of the "Not Found" and it was sent back to the client with HTTP 404. That works fine.
Later I have created a rest API client class which is an easy helper to hide the HTTP communication related code and I used this helper all the time when I called my rest APIs from my code.
BUT I needed to write confusing extra code just because HTTP 404 had two different functions:
the real HTTP 404 when the rest API is not available in the given url, it is thrown by the application server or web-server where the rest API application runs
client get back HTTP 404 as well when there is no data in database based on the where condition of the query.
Important: My rest API error handler catches all the exceptions appears in the back-end service which means in case of any error my rest API always returns with a perfect JSON message with the message details.
This is the 1st version of my client helper method which handles the two different HTTP 404 response:
public static String getSomething(final String uuid) {
String serviceUrl = getServiceUrl();
String path = "user/" + , uuid);
String requestUrl = serviceUrl + path;
String httpMethod = "GET";
Response response = client
.target(serviceUrl)
.path(path)
.request(ExtendedMediaType.APPLICATION_UTF8)
.get();
if (response.getStatus() == Response.Status.OK.getStatusCode()) {
// HTTP 200
return response.readEntity(String.class);
} else {
// confusing code comes here just because
// I need to decide the type of HTTP 404...
// trying to parse response body
try {
String responseBody = response.readEntity(String.class);
ObjectMapper mapper = new ObjectMapper();
ErrorInfo errorInfo = mapper.readValue(responseBody, ErrorInfo.class);
// re-throw the original exception
throw new MyException(errorInfo);
} catch (IOException e) {
// this is a real HTTP 404
throw new ServiceUnavailableError(response, requestUrl, httpMethod);
}
// this exception will never be thrown
throw new Exception("UNEXPECTED ERRORS, BETTER IF YOU DO NOT SEE IT IN THE LOG");
}
BUT, because my Java or JavaScript client can receive two kind of HTTP 404 somehow I need to check the body of the response in case of HTTP 404. If I can parse the response body then I am sure I got back a response where there was no data to send back to the client.
If I am not able to parse the response that means I got back a real HTTP 404 from the web server (not from the rest API application).
It is so confusing and the client application always needs to do extra parsing to check the real reason of HTTP 404.
Honestly I do not like this solution. It is confusing, needs to add extra bullshit code to clients all the time.
So instead of using HTTP 404 in this two different scenarios I decided that I will do the following:
I am not using HTTP 404 as a response HTTP code in my rest application anymore.
I am going to use HTTP 204 (No Content) instead of HTTP 404.
In that case client code can be more elegant:
public static String getString(final String processId, final String key) {
String serviceUrl = getServiceUrl();
String path = String.format("key/%s", key);
String requestUrl = serviceUrl + path;
String httpMethod = "GET";
log(requestUrl);
Response response = client
.target(serviceUrl)
.path(path)
.request(ExtendedMediaType.APPLICATION_JSON_UTF8)
.header(CustomHttpHeader.PROCESS_ID, processId)
.get();
if (response.getStatus() == Response.Status.OK.getStatusCode()) {
return response.readEntity(String.class);
} else {
String body = response.readEntity(String.class);
ObjectMapper mapper = new ObjectMapper();
ErrorInfo errorInfo = mapper.readValue(body, ErrorInfo.class);
throw new MyException(errorInfo);
}
throw new AnyServerError(response, requestUrl, httpMethod);
}
I think this handles that issue better.
If you have any better solution please share it with us.
404 Not Found technically means that uri does not currently map to a resource. In your example, I interpret a request to http://mywebsite/api/user/13 that returns a 404 to imply that this url was never mapped to a resource. To the client, that should be the end of conversation.
To address concerns with ambiguity, you can enhance your API by providing other response codes. For example, suppose you want to allow clients to issue GET requests the url http://mywebsite/api/user/13, you want to communicate that clients should use the canonical url http://mywebsite/restapi/user/13. In that case, you may want to consider issuing a permanent redirect by returning a 301 Moved Permanently and supply the canonical url in the Location header of the response. This tells the client that for future requests they should use the canonical url.
So in essence, it sounds like the answer could depend on how the request is formed.
If the requested resource forms part of the URI as per a request to http://mywebsite/restapi/user/13 and user 13 does not exist, then a 404 is probably appropriate and intuitive because the URI is representative of a non-existent user/entity/document/etc. The same would hold for the more secure technique using a GUID http://mywebsite/api/user/3dd5b770-79ea-11e1-b0c4-0800200c9a66 and the api/restapi argument above.
However, if the requested resource ID was included in the request header [include your own example], or indeed, in the URI as a parameter, eg http://mywebsite/restapi/user/?UID=13 then the URI would still be correct (because the concept of a USER does exits at http://mywebsite/restapi/user/); and therefore the response could reasonable be expected to be a 200 (with an appropriately verbose message) because the specific user known as 13 does not exist but the URI does. This way we are saying the URI is good, but the request for data has no content.
Personally a 200 still doesn't feel right (though I have previously argued it does). A 200 response code (without a verbose response) could cause an issue not to be investigated when an incorrect ID is sent for example.
A better approach would be to send a 204 - No Contentresponse. This is compliant with w3c's description *The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation.*1 The confusion, in my opinion is caused by the Wikipedia entry stating 204 No Content - The server successfully processed the request, but is not returning any content. Usually used as a response to a successful delete request. The last sentence is highly debateable. Consider the situation without that sentence and the solution is easy - just send a 204 if the entity does not exist. There is even an argument for returning a 204 instead of a 404, the request has been processed and no content has been returned! Please be aware though, 204's do not allow content in the response body
Sources
http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
1. http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
This old but excellent article... http://www.infoq.com/articles/webber-rest-workflow says this about it...
404 Not Found - The service is far too lazy (or secure) to give us a real reason why our request failed, but whatever the reason, we need to deal with it.
This recently came up with our team.
We use both 404 Not found with a message body and 204 No Content based on the following rational.
If the request URI indicates the location of a single resource, we use 404 Not found. When the request queries a URI, we use 204 No Content
http://mywebsite/api/user/13 would return 404 when user 13 does not exist
http://mywebsite/api/users?id=13 would return 204 no content
http://mywebsite/api/users?firstname=test would return 204 no content
The idea here being, 'query routes' are expected to be able to return 1, many or no content.
Whatever pattern you choose, the most important things is to be consistent - so get buy in from your team.
The Uniform Resource Identifier is a unique pointer to the resource. A poorly form URI doesn't point to the resource and therefore performing a GET on it will not return a resource. 404 means The server has not found anything matching the Request-URI. If you put in the wrong URI or bad URI that is your problem and the reason you didn't get to a resource whether a HTML page or IMG.
Since this discussion seems to be able to survive the end of time I'll throw in the JSON:API Specifications
404 Not Found
A server MUST respond with 404 Not Found when processing a request to fetch a single resource that does not exist, except when the request warrants a 200 OK response with null as the primary data (as described above).
HTTP/1.1 200 OK
Content-Type: application/vnd.api+json
{
"links": {
"self": "http://example.com/articles/1/author"
},
"data": null
}
Also please see this Stackoverflow question
For this scenario HTTP 404 is response code for the response from the REST API
Like 400, 401, 404 , 422 unprocessable entity
use the Exception handling to check the full exception message.
try{
// call the rest api
} catch(RestClientException e) {
//process exception
if(e instanceof HttpStatusCodeException){
String responseText=((HttpStatusCodeException)e).getResponseBodyAsString();
//now you have the response, construct json from it, and extract the errors
System.out.println("Exception :" +responseText);
}
}
This exception block give you the proper message thrown by the REST API
I'm currently using the OAuth-Signpost Java library to sign requests sent from a client to a server which implements OAuth authentication. When making GET requests (using HttpURLConnection) everything works fine: requests are signed, parameters are included and signatures match in destination. However, it doesn't seem to work with POST requests. I'm aware of the issues that may come up when signing POST using HttpURLConnection, so I moved to the Apache HttpComponents library for these requests. The parameters I send in the following example are plain strings and a XML-like string ('rxml'). My code goes as follows:
public Response exampleMethod(String user, String sp, String ep, String rn, String rxml){
//All these variables are proved to be correct (they work right in GET requests)
String uri = "...";
String consumerKey = "...";
String consumerSecret = "...";
String token = "...";
String secret = "...";
//create the parameters list
List<NameValuePair> params = new ArrayList<NameValuePair>();
params.add(new BasicNameValuePair("user", user));
params.add(new BasicNameValuePair("sp", sp));
params.add(new BasicNameValuePair("ep", ep));
params.add(new BasicNameValuePair("rn", rn));
params.add(new BasicNameValuePair("rxml", rxml));
// create a consumer object and configure it with the access
// token and token secret obtained from the service provider
OAuthConsumer consumer = new CommonsHttpOAuthConsumer(consumerKey, consumerSecret);
consumer.setTokenWithSecret(token, secret);
// create an HTTP request to a protected resource
HttpPost request = new HttpPost(uri);
// sign the request
consumer.sign(request);
// set the parameters into the request
request.setEntity(new UrlEncodedFormEntity(params));
// send the request
HttpClient httpClient = new DefaultHttpClient();
HttpResponse response = httpClient.execute(request);
//if request was unsuccessful
if(response.getStatusLine().getStatusCode()!=200){
return Response.status(response.getStatusLine().getStatusCode()).build();
}
//if successful, return the response body
HttpEntity resEntity = response.getEntity();
String responseBody = "";
if (resEntity != null) {
responseBody = EntityUtils.toString(resEntity);
}
EntityUtils.consume(resEntity);
httpClient.getConnectionManager().shutdown();
return Response.status(200).entity(responseBody).build();
}
When I send a POST request to the server I get an error telling that the signatures (the one I send and the one the server calculates by itself) don't match, so I guess it has to do with the base string they are signing and the way the POST signing works, since they're handling the same keys and secrets in both sides (checked).
I've read that a way to go through this is setting the parameters as part of the URL (as in a GET request). It wouldn't work for me though, since the XML parameter may exceed the URL length so it needs to be sent as a POST parameter.
I suppose I'm doing something wrong either signing the POST requests or handling the parameters, but I don't know what it is. Please, could you help me out?
P.S: I apologize if I lack context, error traces or additional information regarding this issue, but I'm newbie around here. So please don't hesitate to ask me for more information if you need it.
A bit of backstory/explanation
I've been having a similar problem for the past couple of days, and had almost given up. Until I heard that the guy at my company that was putting up the services I was communicating with, had configured them to read the OAuth information from the query string instead of header parameters.
So instead of reading it from the header parameter Authorization that Signpost puts into the request when you pass it on to be signed, for instance [Authorization: OAuth oauth_consumer_key="USER", oauth_nonce="4027096421883800497", oauth_signature="Vd%2BJEb0KnUhEv1E1g3nf4Vl3SSM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1363100774", oauth_version="1.0"], the services where trying to read the query string, for example http://myservice.mycompany.com?oauth_consumer_key=USER&oauth_nonce=4027096421883800497&oauth_signature=Vd%2BJEb0KnUhEv1E1g3nf4Vl3SSM%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1363100774&oauth_version=1.0.
The problem with this is that when I tried to sign the url and then build a HttpPost request with it, the url got a basestring with the prefix GET instead of POST which gave another signature then the one the service computed. Signpost isn't doing anything wrong, its url signing method is just by default set to GET with no other possibility available out of the box. This is so because you should read header parameters when doing POST, not the query string (Going to egg the house of that "colleague" of mine), and Signpost adds these when signing request which you should do when doing POST.
The signingbasestring can be observed in the SigningBaseString class method generate in Signpost.
Solution
Now this is how I did it, but other ways may be possible or even better.
Get the signpost source code and add it to your project. Can get it here
Locate the OAuthConsumer class and change the signing method so that you can pass on information that the request should be POST. In my case I added a boolean like so public String sign(String url, boolean POST)
Now you need to change the sign method in the AbstractOAuthConsumer class, which CommonsHttpOAuthConsumer and DefaultOAuthConsumer extend. In my case I added the boolean variable to the method and the following if(POST) request.setMethod("POST"); right before the method calls sign(request);
Now the request is a Signpost specific object, HTTPRequest, so this will throw an error. You'll need to change it and add the method public void setMethod(String method);.
This will cause an error in the following classes HttpURLConnectionRequestAdapter, HttpRequestAdapter and UrlStringRequestAdapter. You'll need to add the method implementation to them all, but in different flavors. For the first you'll add
public void setMethod(String method){
try {
this.connection.setRequestMethod(method);
} catch (ProtocolException e) {
e.printStackTrace();
}
}
for the second you'll add
public void setMethod(String method){
try {
RequestWrapper wrapper = new RequestWrapper(this.request);
wrapper.setMethod(method);
request = wrapper;
} catch (org.apache.http.ProtocolException e) {
e.printStackTrace();
}
}
and for the last you'll add
public void setMethod(String method){
mMethod = method;
}
Be warned that I've only used and tried the first and last. But this at least gives you an idea about how to fix the problem, if you are having the same one as I.
Hope this helps in anyway.
-MrDresden
Signing POST requests using oauth-signpost and HttpURLConnection is doable, but it requires a bit of a hack:
The trick is to percent-encode the POST parameters, and add them to the OAuth library using method setAdditionalParameters().
See this article for an example.
This answer helped me.
but for PALINTEXT method, you do not need to have params and also url.
they don't change the signature. signature is constant and based on secrets.
but for SHA1 (and other methods) you can use the above answer.
I am developing a very simple http bot. I am using the javax.net.ssl.HttpsURLConnection class and I have to make multiple requests.
Snippet of code :
HttpURLConnection urlConnection =
(HttpURLConnection) new URL(url+"?"+firstParameters).openConnection();
urlConnection.setRequestProperty("Accept-Charset", "UTF-8");
headerFields = urlConnection.getHeaderFields();
keys = headerFields.keySet();
for(String key : keys){
if(key != null && key.contains("ookie")){
cookies = urlConnection.getHeaderField(key);
break;
}
}
for(String cookie : cookies.split(";")){
if(cookie.contains("JSESSION")){
JSESSION = cookie.split("=")[1];
break;
}
}
document = new InputSource(urlConnection.getInputStream());
parser.setDocument(document);
attributesId.put("name",new ArrayList<String>(Arrays.asList(attributesNames)));
elementsIds.put("INPUT",attributesId);
elements = parser.getValues(elementsIds);
for(String attr : attributesNames){
secondParameters = secondParameters.replaceAll("#r"+index,elements.get(attr));
}
urlConnection.getInputStream().close();
//Second call
urlConnection = (HttpURLConnection) new URL(url2).openConnection();
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", "JSESSIONID="+JSESSION);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
payload = new PrintWriter(urlConnection.getOutputStream());
payload.print(secondParameters);
payload.flush();
payload.close();
Summarizing the code above, first i do a request without any payload and i am able to see the correct response from the server, but the problem is when i make the second request (now with payload and with the JSESSION cookie), what i receive it his the same response that i received in the first request, it looks like i am making the first request again.
So my question is , what i am doing wrong ?
I just need to open one connection, and then change the headers and payload ?
There is any tutorial related with multiple http requests(with mixed methods , post and get)?
Thanks in advance
I've never used HttpURLConnection before. I usually use Apache's HTTPClient code. There are a lot of docs and tutorials about it on their home page.
Couple of things that I noticed about your code:
You code does not handle multiple Cookie headers on the response. Mine seems to handle that better.
Are you sure that all you need is JSESSION? Maybe there are other cookies you are missing?
Have you debugged your code to make sure that your JSESSION cookie gets set appropriately? I added some trim() calls in my cookie processing code to make sure some spaces didn't slip in there.
I can't see the real value of your secondParameters. I have no idea if they are valid. Have you debugged your code to verify the secondParamters value looks good. You can see in my code what I'm posting to the server. Btw, I'd use a StringBuilder instead of + to build them.
Hope this helps.