I am seeing some strange behaviour with the following client-server interaction and I am wondering whether I am running into HTTP/1.1 semantics or my Reactive programming skills need work (or both).
I am attempting to create a client-server interaction where both the request and response bodies are long-running streams of data.
The client is a Spring Reactive WebClient that sends an infinite stream in the request body. It expects to receive (and log) an infinite stream of results.
Flux<Long> requests = Flux.interval(Duration.ofSeconds(2));
return WebClient.create()
.post()
.uri("/instructions")
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(requests, Long.class)
.retrieve()
.bodyToFlux(Object.class)
.map(response -> {
log.info("Received Response Object {}", response);
return response;
});
The server is a spring-boot-starter-webflux application with a route handler to log the request objects as they are received and provide an infinite stream of results:
public Mono<ServerResponse> instructions(ServerRequest request) {
// Log the request objects as they are received
Flux<Object> requestStream = request.bodyToFlux(Object.class)
.map(r -> {
log.info("Received Request Object: {}", r);
return r;
});
requestStream.subscribe();
// Infinite stream of responses
Flux<Long> responses = Flux.interval(Duration.ofSeconds(5));
return ServerResponse.ok()
.contentType(MediaType.APPLICATION_STREAM_JSON)
.body(responses, Long.class);
}
When the above code is run, the server logs the infinite stream of request objects but the client never logs any response objects.
If I bound the request stream by doing something like so: Flux<Long> requests = Flux.interval(Duration.ofSeconds(2)).take(20); then the client begins to log the responses after all requests are received.
What are the problems here?
* Is there something wrong with the reactive code?
* Is this part of the HTTP/1.1 specification where a response header should not be sent until the request body is completely received?
See this question: WebClient doesn't read response until request write is completed.
Apparently, Netty-based WebClient starts processing the response only after it is finished sending the request
Netty server might also behave similarly (only start sending the response after it is finished reading the request body), I'm not sure
Interestingly, Jetty-based WebClient is able to process request and response simultaneously
Related
how to re-use webclient client response? I am using webclient for synchronous request and response. I am new to webclient and not sure how to extract response body in multiple places
WebClient webClient = WebClient.builder().baseUrl("http://localhost:8080").build();
below is my call to API which returns valid response
ClientResponse clientResponse;
clientResponse = webClient.get()
.uri("/api/v1/data")
.accept(MediaType.APPLICATION_JSON)
.exchange()
.block();
How to use clientResponse in multiple places? only one time I am able to extract response body
String response = clientResponse.bodyToMono(String.class).block(); // response has value
When I try to extract the response body second time (in a different class), it's null
String response = clientResponse.bodyToMono(String.class).block(); // response is null
So, can someone explain why response is null second time and how to extract the response body multiple times?
WebClient is based on Reactor-netty and the buffer received is one time thing.
One thing you could do is to cache the result at the first time and then reuse it.
You can refer to this issue in spring cloud gateway: https://github.com/spring-cloud/spring-cloud-gateway/issues/1861
Or refer to what Spring Cloud gateway do for caching request body: https://github.com/spring-cloud/spring-cloud-gateway/blob/master/spring-cloud-gateway-core/src/main/java/org/springframework/cloud/gateway/filter/AdaptCachedBodyGlobalFilter.java
Or you can write your code like:
String block = clientResponse.bodyToMono(String.class).block();
And next time you can use this body:
Mono.just(block);
I call below code within my application. The first request is always working fine. My issue is that every following request is not sent, it runs into timeout, when I specify a timeout value. Otherwise it seems to wait endlessly. It seems the first request blocks the connection for every following attempt. How can I ensure the connection is properly released again? Maybe some headers? Maybe some properties (defaults are used for http https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/doc-files/net-properties.html)?
HttpRequest request = HttpRequest.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.GET()
.uri(URI.create(url))
.build();
try {
HttpResponse<Path> response = client.send(request, HttpResponse.BodyHandlers.ofFile(Paths.get(outfile)));
} catch (IOException | InterruptedException e) {
// ...
}
used: java.net.http.HttpClient (AdoptOpenJDK 11.0.7_10)
The send method is a Sync method, so the request get block until you get a response, in this case maybe you are not getting the response.
send(HttpRequest, BodyHandler) blocks until the request has been sent and the response has been received.
Try using the Async method
sendAsync(HttpRequest, BodyHandler) sends the request and receives the response asynchronously.
The sendAsync method returns immediately with a CompletableFuture. The CompletableFuture completes when the response becomes available. The returned CompletableFuture can be combined in different ways to declare dependencies among several asynchronous tasks.
Example of an async request (taken from the apidoc):
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("https://example.com/"))
.timeout(Duration.ofMinutes(2))
.header("Content-Type", "application/json")
.POST(BodyPublishers.ofFile(Paths.get("file.json")))
.build();
client.sendAsync(request, BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.thenAccept(System.out::println);
I'm trying to make a POST call to an api through WebClient without a request body (the api does not need a body):
webClient.post()
.uri("url")
.retrieve()
.bodyToMono(CustomResponse.class);
But the call returns an exception:
The underlying HTTP client completed without emitting a
response.java.lang.IllegalStateException: The underlying HTTP client
completed without emitting a response.
For another API which requires a request body, I can successfully make a POST call with no issue:
webClient.post()
.uri("url")
.bodyValue("value")
.retrieve()
.bodyToMono(CustomResponse.class);
Is the problem related to the WebClient’s request body? How do I fix it?
UPDATE
The error happens because I added a ContentType header for Webclient through .defaultHeader("ContentType", JSON).
The problem has gone after I've removed the header.
It looks like the downstream webservice responding too slow and exceeding connection timeout hence you're getting this issue.
In case you don't need (or response doesn't have body) you can use .toBodilessEntity() instead. So you code snippet can be rewritten:
webClient.post()
.uri("url")
.retrieve()
.toBodilessEntity();
Why should you use it? Because you're always responsible for resources release. Please, read NOTE section.
When using exchange(), it is the responsibility of the application to
consume any response content regardless of the scenario (success,
error, unexpected data, etc). Not doing so can cause a memory leak.
See ClientResponse for a list of all the available options for
consuming the body.
I receive a Request from the Client which returns a SendRequest-Object that has a HttpMethod, a path and data to send.
Now I would like to send the Request depending on the object I get to the API.
After sending I will get a Response.
The problem now is how can I send the payload and receive the response.
#Bean
public IntegrationFlow httpPostSendRawData() {
return IntegrationFlows.from(
Http.inboundGateway("/api/data/send")
.requestMapping(r -> r.methods(HttpMethod.POST))
.statusCodeExpression(dataParser().parseExpression("T(org.springframework.http.HttpStatus).BAD_REQUEST"))
.requestPayloadType(ResolvableType.forClass(DataSend.class))
.crossOrigin(cors -> cors.origin("*"))
.headerMapper(dataHeaderMapper())
)
.channel("http.data.send.channel")
.handle("rawDataEndpoint", "send")
.transform(/* here i have some transformations*/)
.handle(Http.outboundGateway((Message<SendRequest> r)->r.getPayload().getPath())
.httpMethod(/* Here I would like to get the Method of SendRequest*/)
//add payload
.extractPayload(true))
.get();
It's not clear what is your SendRequest, but the idea for the method is exactly the same what you have done for the url:
.httpMethodFunction((Message<SendRequest> r)->r.getPayload().getMethod())
Although, since you want to have some extraction for the request body, you need to do that in advance in the transform() and move values for url and method to the headers.
There is just no any payload extraction in the Http.outboundGateway: it deals with the whole request payload as an entity for HTTP request body.
I am writing a simple HTTP client using netty-4.0x.
Build the pipeline as below :
pipeline.addLast("codec", new HttpClientCodec());
pipeline.addLast("inflater", new HttpContentDecompressor());
pipeline.addLast("handler", new HttpResponseHandler());
where HttpResponseHandler provides implementation of messageReceived(),
Now there is a thread-pool which call the client and keep sending http message,
I understand that ChannelFuture future = channel.write(request); is async call and will come out without blocking
The query which i am having is, is there a way to link request-response, without calling
future.sync() call.
Thanks for all the help in advance !!!
If you code the server too, you could have the client add a unique id to the HTTP header and have the server echo it back in the response.
If you are following strict HTTP pipelining rules then, on any given channel, the responses will be returned in the order the requests were sent in. It should be enough to maintain a request queue, removing the front of the queue for each response received.
If you are creating a new channel, and a new pipeline for that channel, for each request, it's even easier. Either way you can add a handler to your pipeline which remembers the request (or queue of requests), and returns the request and response to your application when the response is received.