I'd like to log the original 'raw' request body (e.g. JSON) while using Camel Rest endpoints. What's the proper way to do this?
My setup (RouteBuilder) looks like this:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.contextPath(this.contextPath)
.bindingMode(RestBindingMode.json);
rest("myService/").post()
.produces("application/json; charset=UTF-8")
.type(MyServiceRequest.class)
.outType(MyServiceResponse.class)
.to(SERVICE_CONTEXT_IN);
from(SERVICE_CONTEXT_IN).process(this.serviceProcessor);
My problem here is that the mechanics such as storing the request as an Exchange property are 'too late' in terms of using this approach, any processors are too late in the route, i.e., the binding already took place and consumed the Request. Also the CamelHttpServletRequest's InputStream has already been read and contains no data.
The first place to use the log EIP is directly before the single processor:
from(SERVICE_CONTEXT_IN).log(LoggingLevel.INFO, "Request: ${in.body}")
.process(this.serviceProcessor);
but at that point the ${in.body} is already an instance of MyServiceRequest. The added log above simply yields Request: x.y.z.MyServiceRequest#12345678. What I'd like to log is the original JSON prior to being bound to a POJO.
There seems to be no built-in way of enabling logging of the 'raw' request in RestConfigurationDefinition nor RestDefinition.
I could get rid of the automatic JSON binding and manually read the HTTP Post request's InputStream, log and perform manual unmarshalling etc. in a dedicated processor but I would like to keep the built-in binding.
I agree there is no way to log the raw request (I assume you mean the payload going through the wire before any automatic binding) using Camel Rest endpoints.
But taking Roman Vottner into account, you may change your restConfiguration() as follows:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.componentProperty("handlers", "#yourLoggingHandler")
.contextPath(this.contextPath)
.bindingMode(RestBindingMode.json);
where your #yourLoggingHandler needs to be registered in your registry and implement org.eclipse.jetty.server.Handler. Please take a look at writing custom handlers at Jetty documentation http://www.eclipse.org/jetty/documentation/current/jetty-handlers.html#writing-custom-handlers.
In the end I 'solved' this by not using the REST DSL binding with a highly sophisticated processor for logging the payload:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.contextPath(this.contextPath);
rest("myService/").post()
.produces("application/json; charset=UTF-8")
.to(SERVICE_CONTEXT_IN);
from(SERVICE_CONTEXT_IN).process(this.requestLogProcessor)
.unmarshal()
.json(JsonLibrary.Jackson, MyServiceRequest.class)
.process(this.serviceProcessor)
.marshal()
.json(JsonLibrary.Jackson);
All the requestLogProcessor does is to read the in body as InputStream, get and log the String, and eventually pass it on.
You can solve this by:
Turning the RestBindingMode to off on your specific route and logging the incoming request string as is.
After which you can convert the JSON string to your IN type object using ObjectMapper.
At the end of the route convert the java object to JSON and put it in the exchange out body, as we turned off the RestBindingMode.
rest("myService/").post()
.bindingMode(RestBindingMode.off)
.to(SERVICE_CONTEXT_IN);
In my case, streamCaching did the trick because the Stream was readable only once. Thefore I was able log but was not able to forward the body any more. I hope this might be of help to someone
Related
I'm trying stream the data from an HTTP (GET) response to another HTTP (POST) request. With old HttpURLConnection I would take the responses OutputStream, read parts into a buffer and write them to the requests InputStream.
I've already managed to do the same with HttpClient in Java 11 by creating my own Publisher that is used in the POST to write the request body. The GET request has a BodyHandler with ofByteArrayConsumer that sends the chunks to the custom Publisher which itself then sends the chunks to the subscribing HTTP POST request.
But I think this is not the correct approach as it looks like there is something in the API that looks like this could be done directly without implementing publishers and subscribers myself.
There is HttpResponse.BodyHandlers.ofPublisher() which returns a Publisher<List<ByteBuffer> which I can use for the HTTP GET request. Unfortunately for my POST request, there is HttpRequest.BodyPublishers.fromPublisher which expects a Publisher<? extends ByteBuffer> so it seems that the fromPublisher only works for a publisher that holds a complete ByteBuffer and not one that sends several ByteBuffers for parts of the data.
Do I miss something here to be able to connect the BodyPublisher from one request to the other?
You're not missing anything. This is simply a use case that is not supported out of the box for now. Though the mapping from ByteBuffer to List<ByteBuffer> is trivial, the inverse mapping is less so. One easy (if not optimal) way to adapt from one to the other could be to collect all the buffers in the list into a single buffer - possibly combining HttpResponse.BodyHandlers.ofPublisher() with HttpResponse.BodyHandlers.buffering() if you want to control the amount of bytes in each published List<ByteBuffer> that you receive from upstream.
I've looked through the examples at https://doc.akka.io/docs/akka-http/current/introduction.html for Akka HTTP routing and strangely for something built on top of Akka Streams none of the examples connect to a stream.
Can somebody show a simple example of creating a Java DSL flow (not Scala please), and then connecting a Route directly to that flow?
Or am I missing the point and it's not possible but requires some CompletionStage code within the Route to wait for a result of glue code that calls a Flow?
Edit: to clarify the flow can do something like append a string to a posted request body.
Using akka streams to complete a route is definitely possible. It involves either:
a web socket route, see examples in the docs, or
a chunked http response (since you typically do not know the size of the response if it's fed from a stream). You can create a Chunked Entity from an akka stream Source of ByteStrings
you can also use other response types if the response size is known in advance, see docs for HttpEntity about their specifics
Edit: to clarify the flow can do something like append a string to a posted request body.
Michał's answer contains good links, so please give them a read. Akka HTTP is by default and always streaming with its data -- e.g. the entities. So for example to do a streaming "echo" which at the same time adds a suffix, you could do something like this:
path("test", () ->
// extract the request entity, it contains the streamed entity as `getDataBytes`
extractRequestEntity(requestEntity -> {
// prepare what to add as suffix to the incoming entity stream:
Source<ByteString, NotUsed> suffixSource =
Source.single(ByteString.fromString("\n\nADDS THIS AFTER INCOMING ENTITY"))
// concat the suffix stream to the incoming entity stream
Source<ByteString, Object> replySource = requestEntity.getDataBytes()
.concat(suffixSource);
// prepare and return the entity:
HttpEntity.Chunked replyEntity = HttpEntities.create(ContentTypes.TEXT_PLAIN_UTF8, replySource);
return complete(StatusCodes.OK, replyEntity);
})
);
Having that said, there is numerous ways to make use of the streaming capabilities, including framed JSON Streaming and more. You should also give the docs page about implications of streaming a read.
I am trying to forward a large file pulled as an input stream to another service using spring's resttemplate. I have followed the answer given by #artbristol in this topic: How to forward large files with RestTemplate?
And it looks like it is setting the body of the request properly (grabbing the request with charlesproxy). The problem is that I have not set the headers correctly since I believe I need to set the content-type as multipart/formdata which I tried by adding this in the callback:
request.getHeaders().setContentType(
new MediaType("multipart", "form-data"));
But in the http headers I am still missing the boundary, not sure how to set that and I am sure I am probably missing some other settings.
So I was able to figure this out. Basically I needed to create a Spring message converter that will take in the input stream and write out to the body. I also basically have to use the Form Message Converter to write out the response body as well.
So in the restTemplate I call an add message converter to add new input stream message converter. In the call back I create a multivaluemap that takes in a string and inputstream and wrap that around an HttpEntity. Then I create a new instance of the Form Message converter and call write, passing in request, and the mutlivaluemap.
It looks like the issue is that I did not include the path to htrace-core.jar in the spark class path:
spark-shell --driver-class-path /opt/cloudera/parcels/CDH/lib/hbase/hbase-server.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar:/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar:/etc/hbase/conf
Seems like this is new for spark 1.x
I've a simple java RESTful service which queries database and send the response back based on the request parameter. I need to generate some simple report based on the apache access_log, for example number of queries/day, number of similar queries, etc.
One of report I need to generate is to list queries which returns zero result. I'm wondering how to achieve this. I can't rely on the response size in apache log, since the the response xml with zero result will still be returned.
I'm thinking of setting a custom cookie if the query returns no result and have it printed in apache log..
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\"" combined-cookie
Not sure if this will work or to be honest, is this is the right approach.
Any pointers will be highly appreciated.
Thanks
If you know for sure that the "no results" response is NNN bytes, and you know that any other response would be different (larger), then you could potentially query your access log for responses of size NNN. But that's a bit of a hack, and it's brittle if the size of an empty response changes for whatever reason.
I don't think Apache has any built-in capability to inspect the content of a response and set variables based on some property of the data. (You could potentially do something very hacky with mod_ext_filter, but it's not worth the hassle and the performance would likely suffer.)
It sounds like you already have the ability to change the server code that's producing the response. Since that's the case, I would not try to use Apache logging. Instead, I would add some additional logging capability to your server. Every response could output a line to a different log file. The lines could look like this:
2012-06-14 14:02:15.345 count=0 status=Completed
2012-06-14 14:02:15.906 count=12 status=Completed
...
Then the type of reporting you need becomes easier.
But if you absolutely have to do it with Apache, then my suggestion would be to invent a new HTTP header, something like X-Query-Result, and then tweak your server to set that header on every response. For example:
X-Query-Result: count=0 status=Completed
Then similar to what you suggested, use \"%{X-Query-Result}i\" in your log format. I'd choose this over cookies just to avoid the extra "weight" associated with cookies.
First off, I'm using an older version of Restlet (1.1).
Secondly, I'm not sure I'm doing the correct thing. Here's what I'm trying to do...
I'm creating a reporting service (resource). I'd like my service to listen for POST requests. The body of the request will contain the report definition. I'd like the response to be the CSV file generated by the service (the report). Is responding to a POST request in this manner OK from a REST standpoint (if not, then how to refine this resource)?
I can't seem to figure out how the acceptRepresentation() generates the response. I've tried setting the Representation parameter passed into the method to a new FileRepresentation. I've also tried to utilize the represent() method, but it doesn't seem like that method is called as part of the POST processing.
How can I accomplish this seeming easy task?
Calling the getResponse().setEntity() method from acceptRepresentation() will accept the new FileRepresentation and accomplish what I'd like to.