I want to use the new java.net.HttpClient to do some requests to another system.
For debug purposes I want to log (and later store in our db) the request that I send and the response that I receive.
How can I retrieve the effective http headers, that java is sending?
I tried to get the headers like this:
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("http://localhost:54113"))
.build();
System.out.println("HTTP-Headers:\n---");
request.headers().map()
.forEach((key, values) ->
values.forEach(value ->
System.out.println(key + ": " + value)
)
);
System.out.println("---");
HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
But it outputs:
HTTP-Headers:
---
---
My server, however, tells me, that it receives these Http headers:
HTTP-Headers:
---
Connection: Upgrade, HTTP2-Settings
User-Agent: Java-http-client/11
Host: localhost:54113
HTTP2-Settings: AAEAAEAAAAIAAAABAAMAAABkAAQBAAAAAAUAAEAA
Content-Length: 0
Upgrade: h2c
---
I have a multithreaded application and simultanious requests might occur. Using a log framework with custom appenders is therefore probably not reliable.
I have an unfortunate answer to your question: Regrettably, impossible.
Some background on why this is the case:
The actual implementation of HttpRequest used by your average OpenJDK-based java-core-library implementation is not java.net.http.HttpRequest - that is merely an interface. It's jdk.internal.net.http.HttpRequestImpl.
This code has 2 separate lists of headers to send; one is the 'user headers' and the other is the 'system headers'. Your .headers() call retrieves solely the user headers, which are headers you explicitly asked to send, and, naturally, as you asked for none to send, it is empty.
The system headers is where those 6 headers are coming from. I don't think there is a way to get at these in a supported fashion. If you want to dip into unsupported strategies (Where you write code that queries internal state and is thus has no guarantee to work on other JVM implementations, or a future version of a basic JVM implementation), it's still quite difficult, unfortunately! Some basic reflection isn't going to get the job done here. It's the worst news imaginable:
These 6 headers just aren't set, at all, until send is invoked. For example, the three headers that are HTTP2 related are set in the package-private setH2Upgrade method, and this method is passed the HttpClient object, which proves that this cannot possibly be called except in the chain of events started when you invoke send. An HttpClient object doesn't exist in the chain of code that makes HttpRequest objects, which proves this.
To make matters considerably worse, the default HttpClient impl will first clone your HttpRequest, then does a bunch of ops on this clone (including adding those system headers), and then sends the clone, which means the HttpRequest object you have doesn't have any of these headers. Not even after the send call completes. So even if you are okay with fetching these headers after the send and are okay with using reflecting to dig into internal state to get em, it won't work.
You also can't reflect into the client because the relevant state (the clone of your httprequest object) isn't in a field, it's in a local variable, and reflection can't get you those.
A HttpRequest can be configured with custom proxies, which isn't much of a solution either: That's TCP/IP level proxies, not HTTP proxies, and headers are sent encrypted with HTTPS. Thus, writing code that (ab)uses the proxy settings so that you can make a 'proxy' that just bounces the connection around your own code first before sending it out, in order to see the headers in transit, is decidedly non-trivial.
The only solution I can offer you is to ditch java.net.http.HttpClient entirely and use a non-java-lib-core library that does do what you want. perhaps OkHttp. (Before you sing hallelujah, I don't actually know if OkHttp can provide you with all the headers it intends to send, or give you a way to register a hook that is duly notified, so investigate that first!)
Related
I'm trying to protect against a particular error case I'm running into. When I make an HttpResponse to build a request and execute it, I run the risk of hitting some rate limit issues (even using a Google RateLimiter to try and alleviate this). What I thought would make sense is to use an HttpBackOffUnsuccessfulResponseHandler with an ExponentialBackoff and that would solve the problem (as the wait times being requested are very small.)
However, in the course of testing, I've come to realize the api endpoint I'm hitting requires a nonce and you can't make the same call in succession with the same nonce (this data is part of the header on the request.)
I'm trying to determine if there is a way I can use the existing HttpBackOffUnsuccessfulResponseHandler and have it update one of the headers before each retry. Is this something easy to do, or am I basically going to have to create my own version of HttpBackOffUnsuccessfulResponseHandler? Or at this point, would it make sense that in the case of detecting a rate limit violation, to just catch then manually and rebuild/resend the request after waiting for the designated time? (The API endpoints I'm hitting will include how much longer I need to wait due to rate limiting.)
I considered just extending HttpBackOffUnsuccessfulResponseHandler but the way it is implemented, I cannot override the handleResponse call that might give me a chance to update the request before it is resent.
And example of how I'm currently building the request is as follows (sanitized):
//Please note these headers are just an example. This data should not be treated as real.
HttpHeaders headers = new HttpHeaders()
.setContentType("text/plain").setContentLength(0L)
.set("header-1", header1Data.toString)
.set("payload", payload.toString); // Contains the nonce that would need to be updated.
HttpResponse resp = httpRequestFactory.buildPostRequest(url, null)
.setHeaders(headers)
.setConnectTimeout(30000)
.setReadTimeout(30000)
.setUnsuccessfulResponseHandler(
new HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackoff())
//BlanketBackoffRequirement is a custom implementation of the BackOffRequired interface
.setBackOffRequired(BlanketBackoffRequirement.SERVER_ERROR_OR_RATE_LIMIT)
)
.execute();
I want to retrieve the server's response as is, with all headers. The first thing that comes to mind is to use raw sockets. As I have learned from the search, there are 3 ways to indicate the end of response:
(1) closing the connection;
(2) examining Content-Length;
(3) getting all chunks in the case of Transfer-Encoding: Chunked.
There is also
(4) the timeout method: assume that the timeout means end of data, but the latter is not really reliable.
I want a general-case solution and do not want to
add a Connection: close line to the request itself.
In addition, it is recommended to use an existing library rather than re-invent the wheel.
Question:
How do I use an existing package, preferably, something already present in Android, to detect the end of HTTP response while having access (without interference) to the raw data stream?
UPD: forgot to mention that the HTTP request is given to me as a sequence of bytes. Yes, it is for testing.
PS
relevant reading:
End of an HTTP Response
Detect the end of an HTTP Request in Java
Detect end of HTTP request body
How HTTP Server inform its clients that the response has ended
Proper handling of chuncked Http Response within Socket
Detect the end of a HTTP packet
Android socket & HTTP response headers
Java HTTP GET response waits until timeout
I suggest to use a the Apache HTTP client package (http://hc.apache.org/httpclient-3.x/ ) so you don't need to implement all the finicky details of the HTTP protocol.
The Apache Http Client will give you access to the headers and their content, which may be enough for you.
If you really need access to the actual character sequence sent by the server (e.g. for debugging purposes), you could then intercept the communication by replacing the connection socket factory with your own to create "intercepting" sockets which store all data transferred in a buffer where your code can access it later on. See http://hc.apache.org/httpcomponents-client-4.3.x/tutorial/html/connmgmt.html#d5e418
I need to write a HTTP client which to communicate with Floodlight OpenFlow controller via its REST API.
For testing I did it in python, and it worked OK. But now I'm in a situation where it has to be done in Java, of which I'm admittedly still at the beginner's level. One of my apps uses AsyncHttpClient to dispatch async GET requests, and works just fine. Now as a Floodlight's REST client, it has to do POST and DELETE with JSON encoded body. My code for an async POST request works very much as expected.
But no luck with DELETE.
Somehow it doesn't write JSON string into its request body.
The code is almost identical with POST. For debugging, I don't feed an AsyncCompletionHandler instance to execute() method.
System.out.println(ofEntry.toJson()); // this returns {"name": "xyz"} as expected.
Future<Response> f = httpClient.prepareRequest(new RequestBuilder("DELETE")
.setUrl("http://" + myControllerBaseUrl + urlPathFlowPostDelete)
.setHeader("content-type", "application/json")
.setBody(ofEntry.toJson())
.build()).execute();
System.out.println(f.getStatusCode()); // returns 200.
System.out.println(f.getResponseBody()); // returns {"status" : "Error! No data posted."}.
Just to make sure, I peeped into packet dump with wireshark, and found out the server isn't lying :)
The author of the library has written an extensive amount of relevant, valuable information, but unfortunately I can't find example code specifically for building a DELETE request.
I'd very much appreciate any suggestions, pointers, and of course pinpoint solutions!
Not sure that replying to my own question is appropriate here, but I have just found a related thread at the floodlight-dev Google group.
Problem with Static Flow Pusher DELETE REST method
So this could be a problem with Floodlight REST API which requires message body for a DELETE request to identify what to be deleted, whereas AHC is simply compliant with RFC2616.
I will follow the thread at Google group, and see how it will conclude among developers.
I'm reasonably confident I know the answer to this, but am struggling to find any concrete information out there. I'm aware that the client submits requests to an http server optionally supplying reqeust parameters. The server has the additional capability to store information in request attributes via Objects. My question is, does the client have any access to the attributes in an http request object? We have a lot of poorly written code which looks something like this:
if (request.getAttribute("name") != null)
name = request.getAttribute("name);
else if (request.getParameter("name") != null)
name = request.getParameter("name");
I'm guess this is because the original developer didn't fully understand how client side http requests submitted data to the server. In any case, I'm currently working on implementing additional valiadation and encoding of request data to prevent XSS vulnerabilities and wondered if it was possible for the client to corrupt/hack/take advantage of request attributes (assuming they aren't ever populated with data sourced from the client)?
No. The attributes are something the servlet spec adds, and can be used to communicate between different entities operating on the request. They don't travel over the wire, so they don't exist client side.
The client can set the body, the parameters (i.e. the url) and the headers, and that's pretty much it.
See:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5
How the attribute field of a HttpServletRequest maps to a raw HTTP request?
I'm using java's HTTP Server object with web service implemeted by WebServiceProvider.
I see that no matter of the client request, the answer is chunked and i need it to be with content length.
so i'm assuming the problem is in the server and not the web server provider, right?
and how can i configure the http header to use content length and not chunked?
HttpServer m_server = HttpServer.create();
Endpoint ep= Endpoint.create(new ep());
HttpContext epContext = m_server.createContext("/DownloadFile");
ep.publish(downloadFileContext);
I assume you're talking about the com.sun.net.httpserver HTTPServer. I further assume that you're connecting the server to the service with a call to Endpoint.publish, using some service provider which supports HTTPServer.
The key is in the HttpExchange.sendResponseHeaders method:
If the response length parameter is greater than zero, this specifies an exact number of bytes to send and the application must send that exact amount of data. If the response length parameter is zero, then chunked transfer encoding is used and an arbitrary amount of data may be sent. The application terminates the response body by closing the OutputStream.
So, as long as the handler is passing a positive value for responseLength, Content-Length is used. Of course, to do that, it will have to know how much data it is going to send ahead of time, which it might well not. Whether it does or not depends entirely on the implementation of the binding, i'm afraid. I don't believe this is standardised - indeed, i don't believe that the WebServiceProvider/HTTPServer is standardised at all.
However, even if your provider is uncooperative, you have a recourse: write a Filter which adds buffering, and add it to the HttpContext which you are using to publish the service. I think that to do this, you would have to write an implementation of HttpExchange which buffers the data written to it, pass that down the filter chain for the handler to write its response to, then when it comes back, write the buffered content, setting the responseLength when it does so.