I'm trying to protect against a particular error case I'm running into. When I make an HttpResponse to build a request and execute it, I run the risk of hitting some rate limit issues (even using a Google RateLimiter to try and alleviate this). What I thought would make sense is to use an HttpBackOffUnsuccessfulResponseHandler with an ExponentialBackoff and that would solve the problem (as the wait times being requested are very small.)
However, in the course of testing, I've come to realize the api endpoint I'm hitting requires a nonce and you can't make the same call in succession with the same nonce (this data is part of the header on the request.)
I'm trying to determine if there is a way I can use the existing HttpBackOffUnsuccessfulResponseHandler and have it update one of the headers before each retry. Is this something easy to do, or am I basically going to have to create my own version of HttpBackOffUnsuccessfulResponseHandler? Or at this point, would it make sense that in the case of detecting a rate limit violation, to just catch then manually and rebuild/resend the request after waiting for the designated time? (The API endpoints I'm hitting will include how much longer I need to wait due to rate limiting.)
I considered just extending HttpBackOffUnsuccessfulResponseHandler but the way it is implemented, I cannot override the handleResponse call that might give me a chance to update the request before it is resent.
And example of how I'm currently building the request is as follows (sanitized):
//Please note these headers are just an example. This data should not be treated as real.
HttpHeaders headers = new HttpHeaders()
.setContentType("text/plain").setContentLength(0L)
.set("header-1", header1Data.toString)
.set("payload", payload.toString); // Contains the nonce that would need to be updated.
HttpResponse resp = httpRequestFactory.buildPostRequest(url, null)
.setHeaders(headers)
.setConnectTimeout(30000)
.setReadTimeout(30000)
.setUnsuccessfulResponseHandler(
new HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackoff())
//BlanketBackoffRequirement is a custom implementation of the BackOffRequired interface
.setBackOffRequired(BlanketBackoffRequirement.SERVER_ERROR_OR_RATE_LIMIT)
)
.execute();
Related
I want to use the new java.net.HttpClient to do some requests to another system.
For debug purposes I want to log (and later store in our db) the request that I send and the response that I receive.
How can I retrieve the effective http headers, that java is sending?
I tried to get the headers like this:
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:54113"))
.build();
System.out.println("HTTP-Headers:\n---");
request.headers().map()
.forEach((key, values) ->
values.forEach(value ->
System.out.println(key + ": " + value)
)
);
System.out.println("---");
HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
But it outputs:
HTTP-Headers:
---
---
My server, however, tells me, that it receives these Http headers:
HTTP-Headers:
---
Connection: Upgrade, HTTP2-Settings
User-Agent: Java-http-client/11
Host: localhost:54113
HTTP2-Settings: AAEAAEAAAAIAAAABAAMAAABkAAQBAAAAAAUAAEAA
Content-Length: 0
Upgrade: h2c
---
I have a multithreaded application and simultanious requests might occur. Using a log framework with custom appenders is therefore probably not reliable.
I have an unfortunate answer to your question: Regrettably, impossible.
Some background on why this is the case:
The actual implementation of HttpRequest used by your average OpenJDK-based java-core-library implementation is not java.net.http.HttpRequest - that is merely an interface. It's jdk.internal.net.http.HttpRequestImpl.
This code has 2 separate lists of headers to send; one is the 'user headers' and the other is the 'system headers'. Your .headers() call retrieves solely the user headers, which are headers you explicitly asked to send, and, naturally, as you asked for none to send, it is empty.
The system headers is where those 6 headers are coming from. I don't think there is a way to get at these in a supported fashion. If you want to dip into unsupported strategies (Where you write code that queries internal state and is thus has no guarantee to work on other JVM implementations, or a future version of a basic JVM implementation), it's still quite difficult, unfortunately! Some basic reflection isn't going to get the job done here. It's the worst news imaginable:
These 6 headers just aren't set, at all, until send is invoked. For example, the three headers that are HTTP2 related are set in the package-private setH2Upgrade method, and this method is passed the HttpClient object, which proves that this cannot possibly be called except in the chain of events started when you invoke send. An HttpClient object doesn't exist in the chain of code that makes HttpRequest objects, which proves this.
To make matters considerably worse, the default HttpClient impl will first clone your HttpRequest, then does a bunch of ops on this clone (including adding those system headers), and then sends the clone, which means the HttpRequest object you have doesn't have any of these headers. Not even after the send call completes. So even if you are okay with fetching these headers after the send and are okay with using reflecting to dig into internal state to get em, it won't work.
You also can't reflect into the client because the relevant state (the clone of your httprequest object) isn't in a field, it's in a local variable, and reflection can't get you those.
A HttpRequest can be configured with custom proxies, which isn't much of a solution either: That's TCP/IP level proxies, not HTTP proxies, and headers are sent encrypted with HTTPS. Thus, writing code that (ab)uses the proxy settings so that you can make a 'proxy' that just bounces the connection around your own code first before sending it out, in order to see the headers in transit, is decidedly non-trivial.
The only solution I can offer you is to ditch java.net.http.HttpClient entirely and use a non-java-lib-core library that does do what you want. perhaps OkHttp. (Before you sing hallelujah, I don't actually know if OkHttp can provide you with all the headers it intends to send, or give you a way to register a hook that is duly notified, so investigate that first!)
I am building a REST API using Spring and implementing the PUT functionality. I am trying to handle the scenario in which the client tries to PUT to a uri where the resource does not already exist. In this scenario, per the PUT spec, a new resource should be created at that ID. However because of the ID generation strategy I am using (#GeneratedValue(strategy = GenerationType.IDENTITY)), I cannot create resources with IDs out of sequence. The database must use the next available value. However, according to the w3 spec on PUT...
If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI.
If the server desires that the request be applied to a different URI, it MUST send a 301 (Moved Permanently) response; the user agent MAY then make its own decision regarding whether or not to redirect the request.
In this case, I can do neither of these. I cannot create a new resource at the existing URI due to the ID generation restrictions, and I cannot send a 301 Moved Permanently response because according to How do I know the id before saving an object in jpa it is impossible to know the next ID in a sequence before actually persisting the object. So I would have no way of telling the client what URI to redirect to in order to properly create the new resource.
I would imagine this problem has been solved many times over because it is the standard PUT functionality, yet I am having trouble finding any other people who have tried to do this. It seems most people just ignore the "create new resource" part of PUT, and simply use it as update only.
What I want to do is just go ahead and create the new resource, and then send the 301 Moved Permanently to redirect the client to the true location of the created resource - but as we see above, this violates the definition of PUT.
Is there a spring-y way to solve this problem? Or is the problem unsolved, and the true standard practice is to simply not allow creation of new resources via PUT?
If the server cant processes the request due to an error in the request, just return a 400.
400 Bad Request -
The server cannot or will not process the request due to an apparent client error (e.g., malformed request syntax, size too large, invalid request message framing, or deceptive request routing).
Is it enough to request one XSRF token from the server and reuse it over the whole session or should I request for each protect-worthy action like save, edit or delete a new XSRF token first and then perform the actual request?
This question arises because I don't understand why my XSRF protected request is working even though I am not requesting a new one:
public void saveName(Long shopId, Long languageId, String name, OnSuccessCallback<String> success, OnFailureCallback failure) {
Request.<String> doRequest(this.shopService,
asyncCallback -> {
this.shopService.saveName(shopId, languageId, name, asyncCallback);
},
(String result) -> {
// ..
success.onSuccess(result);
}, failure);
}
Here Request#doRequest() will simply perform a request but won't ask for a new XSRF token first. I would have to change it to XsrfRequest#doRequest() which does the same thing basically but will request a XSRF token first.
The thing is that saveName() is supposed to be protected:
#XsrfProtect
#RemoteServiceRelativePath("shop")
public interface ShopServlet extends RemoteService {
// ..
String saveName(Long shopId, Long languageId, String name);
}
Please note: Before saveName() is getting called there are several other requests of which some of them already get XSRF tokens. But since I can save a new name without requesting a new one I have the feeling that the previous requested token is getting reused here. Is this okay that way?
Another thing I noticed is if I add #NoXsrfProtect to saveName()
#NoXsrfProtect
String saveName(Long restaurantId, Long languageId, String name);
that the request will still contain XSRF token information:
7|2|9|http://localhost:8080/app/|424F33664CAA93E2F8A4A94C57DA5827|com.google.gwt.user.client.rpc.XsrfToken/4254043109|..ShopServlet|saveName|java.lang..
Why is the token being sent here even though the method is declared as #NoXsrfProtect?
Could somebody clarify this to me? I don't want to make any mistakes - especially in security related matters..
Is it enough to request one XSRF token from the server and reuse it over the whole session or should I request for each protect-worthy action like save, edit or delete a new XSRF token first and then perform the actual request?
Lets ignore GWT RPC's built-in XSRF protection for a moment and look at the title question and this remark: What is XSRF and how do we protect against it?
What is XSRF
XSRF stands for Cross Site Request Forgery - the idea is that a malicious site could somehow forge a request, and force one of our users to correctly send it to our application, as if they had intended to do it themselves. For example, if all it took to transfer money from one bank account to another was
GET /transfer?from=me&to=attacker&amount=10000USD
then an attacker could very simply make a request to our server as an image, css, or js file, from their own site:
<img src="https://securebank.com/transfer?from=me&to=attacker&amount=10000USD" />
Setting aside other details ("okay, that works for a GET, how did they manage to send a POST to my GWT RPC service?"), lets look at the idea of a XSRF "token" preventing this attack: what is it that a friendly client knows or can do, that an attacker cannot know or do?
Mitigation
The idea is that the client should have a way to indicate to the server that it is trusted - that it knows something which only a valid client could know, which indicates that the user is willing to make the specified action. One option is to require the user to perform a captcha, in such a way that the action cannot be scripted by the attacking site, and the user must consciously perform. Another option is to make some data available to a real/trusted client, such as cookies (that can only be read from a page loaded on the same domain), or as part of the HTML page as it loads or some other request (which might be possible to be sent by some other means, but the results can't be read).
OWASP refers to this latter piece of data as a "Synchronizer Token":
The synchronizer token pattern requires the generating of random "challenge" tokens that are associated with the user's current session. [...] When the user wishes to invoke these sensitive operations, the HTTP request should include this challenge token. It is then the responsibility of the server application to verify the existence and correctness of this token.
So, in this case, we could write some value to a cookie so that only the client can see it, and the client can then use this to generate a token. That token then should be passed to the server on each request that must be verified. But from this description, we see that it isn't necessarily important to only have one valid token at a time.
But if the attacker can get an XSS, they can just read the token and force the request again! Or MitM!
That's true, but if they have an XSS, then any request that your own JS can make, the attack can make as well. You've lost, pack up shop, time to go home. Likewise if they own the connection between the user and application and can read and write at will. XSRF protection isn't a magic wand that solves all problems, its a specific attack, and only needs to be addressed on its own: a lock in your home isn't considered faulty if the window can be broken.
Okay, back to GWT
So how does GWT do this? As you've noted, the #XsrfProtect annotation marks a service type as needing to be checked on the server. The client then must request a token, and then make sure the client's service is aware of that token for future requests.
So how does the server generate a token? The XsrfTokenServiceServlet RPC service on the server generates the token, as part of each call to the server, as you've observed, XsrfProtectedServiceServlet.validateXsrfToken then verifies that this is correct. If you wanted custom behavior, you'd have to modify each side of that system, and you could build it to invalidate each token once it is used, but this is not the default (nor, according to OWASP, is it encouraged).
Another thing I noticed is if I add #NoXsrfProtect to saveName()...
Note that validateXsrfToken is only called in one place in GWT, from in AbstractXsrfProtectedServiceServlet:
#Override
protected void onAfterRequestDeserialized(RPCRequest rpcRequest) {
if (shouldValidateXsrfToken(rpcRequest.getMethod())) {
validateXsrfToken(rpcRequest.getRpcToken(), rpcRequest.getMethod());
}
}
The method shouldValidateXsrfToken then checks if the method expressly has the protection disabled. If so, it will return false, and no check will be performed. But when building up the RPCRequest object, RPC.decodeRequest always appends the token, even if that token happens to be null:
RpcToken rpcToken = null;
if (streamReader.hasFlags(AbstractSerializationStream.FLAG_RPC_TOKEN_INCLUDED)) {
// Read the RPC token
rpcToken = (RpcToken) streamReader.deserializeValue(RpcToken.class);
}
So if the client is configured to send the token, it will always be sent, though the server might ignore it.
More reading:
https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF) and https://www.owasp.org/index.php/CSRF_Prevention_Cheat_Sheet
http://www.gwtproject.org/doc/latest/DevGuideSecurityRpcXsrf.html
I need to write a HTTP client which to communicate with Floodlight OpenFlow controller via its REST API.
For testing I did it in python, and it worked OK. But now I'm in a situation where it has to be done in Java, of which I'm admittedly still at the beginner's level. One of my apps uses AsyncHttpClient to dispatch async GET requests, and works just fine. Now as a Floodlight's REST client, it has to do POST and DELETE with JSON encoded body. My code for an async POST request works very much as expected.
But no luck with DELETE.
Somehow it doesn't write JSON string into its request body.
The code is almost identical with POST. For debugging, I don't feed an AsyncCompletionHandler instance to execute() method.
System.out.println(ofEntry.toJson()); // this returns {"name": "xyz"} as expected.
Future<Response> f = httpClient.prepareRequest(new RequestBuilder("DELETE")
.setUrl("http://" + myControllerBaseUrl + urlPathFlowPostDelete)
.setHeader("content-type", "application/json")
.setBody(ofEntry.toJson())
.build()).execute();
System.out.println(f.getStatusCode()); // returns 200.
System.out.println(f.getResponseBody()); // returns {"status" : "Error! No data posted."}.
Just to make sure, I peeped into packet dump with wireshark, and found out the server isn't lying :)
The author of the library has written an extensive amount of relevant, valuable information, but unfortunately I can't find example code specifically for building a DELETE request.
I'd very much appreciate any suggestions, pointers, and of course pinpoint solutions!
Not sure that replying to my own question is appropriate here, but I have just found a related thread at the floodlight-dev Google group.
Problem with Static Flow Pusher DELETE REST method
So this could be a problem with Floodlight REST API which requires message body for a DELETE request to identify what to be deleted, whereas AHC is simply compliant with RFC2616.
I will follow the thread at Google group, and see how it will conclude among developers.
First off, I'm using an older version of Restlet (1.1).
Secondly, I'm not sure I'm doing the correct thing. Here's what I'm trying to do...
I'm creating a reporting service (resource). I'd like my service to listen for POST requests. The body of the request will contain the report definition. I'd like the response to be the CSV file generated by the service (the report). Is responding to a POST request in this manner OK from a REST standpoint (if not, then how to refine this resource)?
I can't seem to figure out how the acceptRepresentation() generates the response. I've tried setting the Representation parameter passed into the method to a new FileRepresentation. I've also tried to utilize the represent() method, but it doesn't seem like that method is called as part of the POST processing.
How can I accomplish this seeming easy task?
Calling the getResponse().setEntity() method from acceptRepresentation() will accept the new FileRepresentation and accomplish what I'd like to.