We Keep Coding

WebClient Request and Response body logging

I am trying to make a POJO out of request and response data received when making WebClient calls. But I am not getting the request body in string/JSON readable form instead I am getting a BodyInsertor. I am making use of Exchange Filters.
public ExchangeFilterFunction logWebRequest() {
return (request, next) -> {
log.info("Entered in logWebRequest for WebClient");
long startTime = System.currentTimeMillis();
Mono<ClientResponse> response = next.exchange(request);
long processingTimeInMs = System.currentTimeMillis() - startTime;
// request.body() -> Gives Body Insertor
WebRequestLog webRequestLog = webRequestService.makeWebRequestLog(request, response.block());
webRequestLog.setProcessingTimeInMs(processingTimeInMs);
log.info("WebRequest to be produced to kafka topic: " + webRequestLog);
kafkaService.produceAuditLog(webRequestLog);
return response;
};
}
I followed some articles such as https://andrew-flower.com/blog/webclient-body-logging and https://www.gitmemory.com/issue/spring-projects/spring-framework/24262/570245788 but nothing worked for me.
My end goal is to capture requests and responses with their bodies and produce the data collected for Kafka.

Inside ExchangeFilterFunction, you can access HTTP method, URL, headers, cookies but request or response body can not be accessed directly from this filter.
Refer to the answer here. It provides a way to get access to the request and response body. It also provides a link to This blog post. It explains how to get the body in JSON/String format in Web Client.

You can do tracing of request and response payloads with small manipulations with request and responses:
public class TracingExchangeFilterFunction implements ExchangeFilterFunction {
return next.exchange(buildTraceableRequest(request))
.flatMap(response ->
response.body(BodyExtractors.toDataBuffers())
.next()
.doOnNext(dataBuffer -> traceResponse(response, dataBuffer))
.thenReturn(response)) ;
}
private ClientRequest buildTraceableRequest(
final ClientRequest clientRequest) {
return ClientRequest.from(clientRequest).body(
new BodyInserter<>() {
#Override
public Mono<Void> insert(
final ClientHttpRequest outputMessage,
final Context context) {
return clientRequest.body().insert(
new ClientHttpRequestDecorator(outputMessage) {
#Override
public Mono<Void> writeWith(final Publisher<? extends DataBuffer> body) {
return super.writeWith(
from(body).doOnNext(buffer ->
traceRequest(clientRequest, buffer)));
}
}, context);
}
}).build();
}
private void traceRequest(ClientRequest clientRequest, DataBuffer buffer) {
final ByteBuf byteBuf = NettyDataBufferFactory.toByteBuf(buffer);
final byte[] bytes = ByteBufUtil.getBytes(byteBuf);
// do some tracing e.g. new String(bytes)
}
private void traceResponse(ClientResponse response, DataBuffer dataBuffer) {
final byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
// do some tracing e.g. new String(bytes)
}
}

To add to Vicky Ajmera answer best way to get and log a request is with ExchangeFilterFunction.
private ExchangeFilterFunction logRequest() {
return (clientRequest, next) -> {
logger.info("Request: {} {} {}", clientRequest.method(), clientRequest.url(), clientRequest.body());
clientRequest.headers()
.forEach((name, values) -> values.forEach(value -> logger.info("{}={}", name, value)));
return next.exchange(clientRequest);
};
}
But to log a response body you will have to go to the lower level of ClientHttpResponse which then allows you to intercept the body.
First extend ClientHttpResponseDecorator like this:
public class LoggingClientHttpResponse extends ClientHttpResponseDecorator {
private static final Logger logger = LoggerFactory.getLogger(LoggingClientHttpResponse.class);
private static final DataBufferFactory bufferFactory = new DefaultDataBufferFactory();
private final DataBuffer buffer = bufferFactory.allocateBuffer();
public LoggingClientHttpResponse(ClientHttpResponse delegate) {
super(delegate);
}
#Override
public Flux<DataBuffer> getBody() {
return super.getBody()
.doOnNext(this.buffer::write)
.doOnComplete(() -> logger.info("Response Body: {}", buffer.toString(StandardCharsets.UTF_8)));
}
}
Then create your implementation of ClientHttpConnector like this:
public class LoggingClientHttpConnector implements ClientHttpConnector {
private final ClientHttpConnector delegate;
public LoggingClientHttpConnector(ClientHttpConnector delegate) {
this.delegate = delegate;
}
#Override
public Mono<ClientHttpResponse> connect(HttpMethod method, URI uri, Function<? super ClientHttpRequest, Mono<Void>> requestCallback) {
return this.delegate.connect(method, uri, requestCallback).map(LoggingClientHttpResponse::new);
}
}
And last when building your WebClient add a connector:
HttpClient httpClient = HttpClient.create();
ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
WebClient.builder()
.baseUrl("http://localhost:8080")
.clientConnector(new LoggingClientHttpConnectorDecorator(connector))
.filter(logRequest())
.build();

How to implement custom error handling with retroift2

I calling to the api with the basic retrofit Call object:
public interface dataApi {
#GET("animal/cats")
Call<AllAnimals> getAllData(
#Query("api_key") String apiKey
);
}
And I can get the response inside my view model like this:
call.enqueue(new Callback<AllAnimals>() {
#Override
public void onResponse(Call<AllAnimals> call, Response<AllAnimals> response) {
animals.setValue(response.body());
}
#Override
public void onFailure(Call<AllAnimals> call, Throwable t) {
Log.i(TAG, "onFailure: " + t);
}
});
Nothing speical here.
I've several problem with this approach
FIRST - if I give the wrong api key for example, the response should give me a response with the code of the problem, instead I just get null body.
SECOND I am planning to have more api calls, and it's a huge code duplication to handle errors every call I wrote.
How can I implement custom error handling for this situation, that will be apply to other calls too?

I think you can use okhttp interceptor and define yourself ResponseBody converter to fix your problem.
First,intercept you interested request and response;
Second,check the response,if response is failed then modify the response body to empty。
define a simple interceptor
Interceptor interceptor = new Interceptor() {
#Override
public okhttp3.Response intercept(Chain chain) throws IOException {
Request request = chain.request();
String url = request.url().toString();
System.out.println(request.url());
okhttp3.Response response = chain.proceed(request);
if (!response.isSuccessful() && url.contains("animal/cats")) {
// request failed begin to modify response body
response = response.newBuilder()
.body(ResponseBody.create(MediaType.parse("application/json"), new byte[] {}))
.build();
}
return response;
}
};
define self ResponseBody converter
most code from com.squareup.retrofit2:converter-jackson we just add two lines:
final class JacksonResponseBodyConverter<T> implements Converter<ResponseBody, T> {
private final ObjectReader adapter;
JacksonResponseBodyConverter(ObjectReader adapter) {
this.adapter = adapter;
}
#Override public T convert(ResponseBody value) throws IOException {
try {
if (value.contentLength() == 0) {
return null;
}
return adapter.readValue(value.charStream());
} finally {
value.close();
}
}
}
the below code is added：
if (value.contentLength() == 0) {
return null;
}

Requests and response from netty server hangs

I have the following code to create a netty web server based on http server created in the netty's example. My buisness logic is the following.
public class HttpServerHandler extends SimpleChannelInboundHandler<Object> {
private final static Logger LOG = LogManager
.getLogger(HttpServerHandler.class);
private WorkflowService workflowService;
private HttpRequest request;
private final StringBuffer buff = new StringBuffer();
private API avalancheApi;
public HttpServerHandler(WorkflowService workflowService) {
this.workflowService = workflowService;
this.avalancheApi = new API(this.workflowService);
}
#Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
LOG.debug("channelActive");
LOG.debug(ctx.toString());
};
#Override
public void channelReadComplete(ChannelHandlerContext ctx) {
ctx.flush();
}
#Override
public void channelRead0(ChannelHandlerContext ctx, Object msg)
throws IOException {
avalancheApi.setContext(ctx);
if (msg instanceof HttpRequest) {
HttpRequest request = this.request = (HttpRequest) msg;
if (HttpHeaders.is100ContinueExpected(request)) {
send100Continue(ctx);
}
String command = getCommand(request);
LOG.debug(command);
Map<String, List<String>> parameters = getParameters(request);
LOG.debug(parameters);
switch (command) {
case "/login":
ctx = avalancheApi.login(parameters);
break;
case "/test":
ctx = avalancheApi.test();
break;
default:
break;
}
}
if (msg instanceof LastHttpContent) {
LOG.debug("msg is of LastHttpContent");
}
if (!HttpHeaders.isKeepAlive(request)) {
// If keep-alive is off, close the connection once the content is
// fully written.
ctx.writeAndFlush(Unpooled.EMPTY_BUFFER).addListener(
ChannelFutureListener.CLOSE);
}
}
public class API {
private static final Logger LOG = LogManager.getLogger(API.class);
private ChannelHandlerContext ctx;
private HttpResponse response;
private WorkflowService workflowService;
public API(WorkflowService workflowService) {
this.workflowService = workflowService;
this.ctx = null;
}
public void setContext(ChannelHandlerContext ctx) {
this.ctx = ctx;
}
public ChannelHandlerContext login(Map<String, List<String>> parameters) {
boolean success;
String username = getUsername(parameters);
String password = getPassword(parameters);
User user = null;
user = workflowService.login(username, password);
success = validateLogin(user);
this.response = writeLoginResponse(success);
this.ctx.write(this.response);
writeLoginContext(success, response);
return this.ctx;
}
private void writeLoginContext(boolean success, HttpResponse response) {
JsonObject jsonResponseMessage = new JsonObject();
jsonResponseMessage.addProperty("result", success);
LOG.debug(jsonResponseMessage.toString());
this.ctx.write(Unpooled.copiedBuffer(jsonResponseMessage.toString(),
CharsetUtil.UTF_8));
this.response.headers().set(HttpHeaders.Names.CONTENT_LENGTH,
jsonResponseMessage.toString().length());
}
private HttpResponse writeLoginResponse(boolean success) {
if (success)
return createSuccessfullLoginResponse();
else
return createLoginFailureResponse();
}
private HttpResponse createLoginFailureResponse() {
return Response.loginFailureResponse();
}
private HttpResponse createSuccessfullLoginResponse() {
return Response.loginSuccessResponse();
}
}
Response class is only creating the response and the content_type which is of application/json. Content Length is set in the API class. Using python client with requests, results in the request made in http://localhost/login?username=name&password=pass works only once. The second time everything works, but it doesn't finish processing the request and send the response object. Api calls get executed normally, and I also get the message of LastHttpContext message getting print. The problem sometimes happens with browser too. Am I missing something? Maybe the content data and the content length doesn't match? Could it be that when making requests from python client, the content of the previous context isn't flushed and the content_length value of the header and content length of the context doesn't match?

Just wild guess
this.response.headers().set(HttpHeaders.Names.CONTENT_LENGTH,
jsonResponseMessage.toString().length());
Instead, shouldn't you be doing jsonResponseMessage.toString().getBytes().length ?? Sometimes, one character is not just one byte.

My guess is that you have overwritten the context in your API class, and as a result, are writing the response to the wrong context. Is your HttpServerHandler marked with #Shareable?

How to get the response header in a RestEasy client?

i´m implementing a Restful service using Jax-RS 2.0 (Resteasy 3.0.7.Final) and share the interface between client and service.
The return value is void because ClientResponse is deprecated since RestEasy introduced JAX-RS 2.0 in version 3+.
To return the location of the new created object i inject the response, using the #Context annotation, and add the Content-Location header.
For example:
Shared Interface:
#Path("/")
#Consumes("application/xml")
#Produces("application/xml")
interface Resource {
#Path("createSomething")
void createSomething(AnyObject object);
...
}
Implementation class (The Service):
class ResourceImpl {
...
#Context org.jboss.resteasy.spi.HttpResponse response;
...
#Override
void createSomething(AnyObject object) throws AnyException {
String id = service.create(object);
response.getOutputHeaders().putSingle("Content-Location",
"/createSomething/" + id);
response.setStatus(Response.Status.CREATED.getStatusCode());
}
}
The client (build with the Resteasy Proxy Framework):
...
ResteasyClient client = new ResteasyClientBuilder().build();
ResteasyWebTarget target = client.target(baseUrl);
Resource resource = (Resource) target.proxy(Resource.class);
resource.createSomething(anyObject);
...
How can i retrieve Header information (and others, like Atom Links) which has been injected by the service?
Is it reasonable to use client side Filters and Interceptors?
Thank You

The best solution i found was to use a Filter to process the incoming response header.
public class HeaderFilter implements ClientResponseFilter {
private Map<String, String> headers = new HashMap<>();
private List<String> headerFilter = new ArrayList<>();
public final void addHeaderFilter(final String header) {
headerFilter.add(header);
}
public final void removeHeaderFilter(final String header) {
headerFilter.remove(header);
}
public final String getHeader(final String header) {
return headers.get(header);
}
#Override
public final void filter(final ClientRequestContext requestContext,
final ClientResponseContext responseContext)
throws IOException {
headers = new HashMap<>();
for (String headerToLookFor : headerFilter) {
String header = responseContext.getHeaderString(headerToLookFor);
if (header != null) {
headers.put(headerToLookFor, header);
} else {
...
}
}
}
}

Java CXF WS Client - gZip HTTP REQUEST HEADER being ignored

I am trying to set/request gZip in HTTP REQUEST HEADERS inside my Java CXF WS Client BUT for some reason its being IGNORED. I don't get back gZipped response. Here is how I am trying to set. I am using Apache CXF 2.3.2. What am I missing?
public class LoggerXML implements SOAPHandler<SOAPMessageContext> {
private String uniqueIdentifier;
private String sessionId;
public LoggerXML(String sessionId, String uniqueIdentifier) {
this.sessionId = sessionId;
this.uniqueIdentifier = uniqueIdentifier;
}
protected final void setLogStream(PrintStream ps) {
// out = ps;
}
public void init(Map c) {
uniqueIdentifier = "";
}
public Set<QName> getHeaders() {
return null;
}
public boolean handleMessage(SOAPMessageContext smc) {
Boolean outboundProperty = (Boolean)
smc.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
if(outboundProperty){
// Creating HTTP headers & setting gZip.
Map<String, List<String>> headers = (Map<String,
List<String>>) smc.get(MessageContext.HTTP_REQUEST_HEADERS);
if(headers == null){
//System.out.println("LoggerXML.handleMessage: headers = null");
headers = new HashMap<String, List<String>>();
}
// Add HTTP headers to the web service request
headers.put("Accept-Encoding", Collections.singletonList("gzip,deflate"));
//headers.put("Content-Encoding", Collections.singletonList("gzip"));
//headers.put("Accept-Encoding", Collections.singletonList("gzip"));
smc.put(MessageContext.HTTP_REQUEST_HEADERS, headers);
//smc.put("org.apache.cxf.transport.common.gzip.GZIPOutInterceptor.UseGzip","YES");
}
return true;
}
public boolean handleFault(SOAPMessageContext smc) {
return true;
}
// nothing to clean up
public void close(MessageContext messageContext) {
}
// nothing to clean up
public void destroy() {
}
// Other Methods....
}

This code works for me
// Get the underlying Client object from the proxy object of service interface
Client proxy = ClientProxy.getClient(stub);
// Creating HTTP headers
Map<String, List<String>> headers = new HashMap<String, List<String>>();
headers.put("Accept-Encoding", Arrays.asList("gzip"));
// Add HTTP headers to the web service request
proxy.getRequestContext().put(Message.PROTOCOL_HEADERS, headers);
Refer:http://singztechmusings.wordpress.com/2011/09/17/apache-cxf-how-to-add-custom-http-headers-to-a-web-service-request/