We Keep Coding

How to log request payload when using Camel Rest?

I'd like to log the original 'raw' request body (e.g. JSON) while using Camel Rest endpoints. What's the proper way to do this?
My setup (RouteBuilder) looks like this:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.contextPath(this.contextPath)
.bindingMode(RestBindingMode.json);
rest("myService/").post()
.produces("application/json; charset=UTF-8")
.type(MyServiceRequest.class)
.outType(MyServiceResponse.class)
.to(SERVICE_CONTEXT_IN);
from(SERVICE_CONTEXT_IN).process(this.serviceProcessor);
My problem here is that the mechanics such as storing the request as an Exchange property are 'too late' in terms of using this approach, any processors are too late in the route, i.e., the binding already took place and consumed the Request. Also the CamelHttpServletRequest's InputStream has already been read and contains no data.
The first place to use the log EIP is directly before the single processor:
from(SERVICE_CONTEXT_IN).log(LoggingLevel.INFO, "Request: ${in.body}")
.process(this.serviceProcessor);
but at that point the ${in.body} is already an instance of MyServiceRequest. The added log above simply yields Request: x.y.z.MyServiceRequest#12345678. What I'd like to log is the original JSON prior to being bound to a POJO.
There seems to be no built-in way of enabling logging of the 'raw' request in RestConfigurationDefinition nor RestDefinition.
I could get rid of the automatic JSON binding and manually read the HTTP Post request's InputStream, log and perform manual unmarshalling etc. in a dedicated processor but I would like to keep the built-in binding.

I agree there is no way to log the raw request (I assume you mean the payload going through the wire before any automatic binding) using Camel Rest endpoints.
But taking Roman Vottner into account, you may change your restConfiguration() as follows:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.componentProperty("handlers", "#yourLoggingHandler")
.contextPath(this.contextPath)
.bindingMode(RestBindingMode.json);
where your #yourLoggingHandler needs to be registered in your registry and implement org.eclipse.jetty.server.Handler. Please take a look at writing custom handlers at Jetty documentation http://www.eclipse.org/jetty/documentation/current/jetty-handlers.html#writing-custom-handlers.

In the end I 'solved' this by not using the REST DSL binding with a highly sophisticated processor for logging the payload:
restConfiguration().component("jetty")
.host(this.host)
.port(this.port)
.contextPath(this.contextPath);
rest("myService/").post()
.produces("application/json; charset=UTF-8")
.to(SERVICE_CONTEXT_IN);
from(SERVICE_CONTEXT_IN).process(this.requestLogProcessor)
.unmarshal()
.json(JsonLibrary.Jackson, MyServiceRequest.class)
.process(this.serviceProcessor)
.marshal()
.json(JsonLibrary.Jackson);
All the requestLogProcessor does is to read the in body as InputStream, get and log the String, and eventually pass it on.

You can solve this by:
Turning the RestBindingMode to off on your specific route and logging the incoming request string as is.
After which you can convert the JSON string to your IN type object using ObjectMapper.
At the end of the route convert the java object to JSON and put it in the exchange out body, as we turned off the RestBindingMode.
rest("myService/").post()
.bindingMode(RestBindingMode.off)
.to(SERVICE_CONTEXT_IN);

In my case, streamCaching did the trick because the Stream was readable only once. Thefore I was able log but was not able to forward the body any more. I hope this might be of help to someone

HttpServletRequest.getInputStream() does not unwrap chunked HTTP request

I am in the process of sending a HTTP chunked request to an internal system. I've confirmed other factors are not at play by ensuring that I can send small messages without chunk encoding.
My process was basically to change the Transfer-Encoding header to be chunked and I've removed the Content-Length header. Additionally, I am utilising an in-house ChunkedOutputStream which has been around for quite some time.
I am able to connect, obtain an output stream and send the data. The recipient then returns a 200 response so it seems the request was received and successfully handled. The endpoint receives the HTTP Request, and streams the data straight into a table (using HttpServletRequest.getInputStream()).
On inspecting the streamed data I can see that the chunk encoding information in the stream has not been unwrapped/decoded by the Tomcat container automatically. I've been trawling the Tomcat HTTPConnector documentation and can't find anything that alludes to the chunked encoding w.r.t how a chunk encoded message should be handled within a HttpServlet. I can't see other StackOverflow questions querying this so I suspect I am missing something basic.
My question boils down to:
Should Tomcat automatically decode the chunked encoding from my request and give me a "clean" InputStream when I call HttpServletRequest.getInputStream()?
If yes, is there configuration that needs to be updated to enable this functionality? Am I sending something wrong in the headers that is causing it to return the non-decoded stream?
If no, is it common practice to wrap input stream in a ChunkedInputStream or something similar when the Transfer-Encoding header is present ?

This is solved. As expected it was basic in my case.
The legacy system I was using provided handrolled methods to simplify the process of opening a HTTP Connection, sending headers and then using an OutputStream to send the content via a POST. I didn't realise, and it was in a rather obscure location, but the behind-the-scenes helper's we're identifying that I was not specifying a Content-Length thus added the TRANSFER_ENCODING=chunked header and wrapped the OutputStream in a ChunkedOutputStream. This resulted in me double encoding the contents, hence my endpoints (seeming) inability to decode it.
Case closed.

Adding additonal Security to Website

I am running a Java Spring MVC based Web-Application. It is also based on the Hybris Platform.
Now, the basic functionality in terms of Authentication and Authorization is already implemented. Meaning we do have filters for sessions, a working user-system, etc.
However, we currently have no security measurements against things such as XSS and other kinds of possible attacks that are out there.
XSS is probably the biggest concern as it is the most common possible way of attacking.
Now, i wonder ... what steps would be smart to take?
I have taken a look around and i have seen that stuff like XSS-Filter exist.
Implementing such would be pretty easy, just copy past the source and add it as a in tomcats web.xml.
But i wonder if that is a satisfying amount of security from such filter?
There are also way more bloated solutions, for example i could use the spring-security.
However, reading the documentations, i feel like this is very bloated and a large part of it implements what is already implemented ( the two A's, for example). I feel like it would take a lot of work to configure it down to the amount of work that i need it to do. Am i wrong?
And:
How would you say is it recommended to deal with security issues, for example XSS? Do you use a certain predefined framework that suits the needs or is your security "hand-made" by following things like the cheat sheet?

Set Anti-XSS Headers (hint: use Spring Security or make your own Interceptor)
Content-Security-Policy: default-src 'self' --only allow content from your own site
X-XSS-Protection: 1; mode=block --prevent some reflective attacks in some browsers
X-Content-Type-Options: nosniff --can't trick browser into detecting and running js in other content types
Prevent malicious inbound HTML/JS/CSS
Use Hibernate Validator (you don't need to use Hibernate ORM to use this) with the #SafeHtml annotation on all user-supplied String fields.
You could validate all request headers, post params and query params in one Interceptor for simplistic XSS validation.
Escape all user-supplied data on output
Use OWASP's Java Encoder Project <e:forHtml value="${attr}" /> to escape output or JSTL's <c:out value="${attr}"/> and in web.xml set
<context-param>
<param-name>defaultHtmlEscape</param-name>
<param-value>true</param-value>
</context-param>
They are equally safe if escaping HTML node text, but OWASP is safer for HTML attribute or <script> escaping.
If you have too many files to edit, consider http://pukkaone.github.io/2011/01/03/jsp-cross-site-scripting-elresolver.html
Make your session cookie unreadable by JavaScript. In web.xml:
<session-config>
<cookie-config>
<!-- browser will disallow JavaScript access to session cookie -->
<http-only>true</http-only>
</cookie-config>
<tracking-mode>COOKIE</tracking-mode>
</session-config>
If you are hosting user-uploaded files, you need to use a different domain (not subdomain) for download links, so that evil content cannot clobber your session cookie (yes, this can happen even if it's httpOnly)

I'd like to add this answer, as i think it is a simple but important thing to notice:
In my case i realised that i do not need to allow any kind of critical special characters for user input. I analysed the situation and realised that it makes no sense and that there is no need to do so in any of my webpages.
So, instead of having to implement proper styled XSS-safe code on about 60k existing lines of code, i could simply install a filter that would purge out all special characters, that i do not want to allow.
You could see this a little critical in terms of user-friendliness, but it should be OK.
So: If you realise that you do not need to allow any special characters, that are going to be critical in one of the possible contexts (such as JS,HTML,attributes, ... ), then you could safe a lot of work, given that you are in the same situation that i am/was in.
Obviously implementing the steps mentioned by Neil is still proper style if you are doing your work from scratch.
If people had known these steps before all the JS and JSP stuff was written they surely would have implemented them, potentially needing them or not shouldnt matter.

Here is a list of things concerning hybris security that I've done on my projects :
First read documentations
Below links are full of resources and details about security in hybris.
Non fuctional requirement
Data protection
Deployment check list
SAP Commerce Security
XML parser
We often need to import data from xml.
All Sax parser should use below features :
XMLConstants.FEATURE_SECURE_PROCESSING = true
http://xml.org/sax/features/external-general-entities = false
http://xml.org/sax/features/external-parameter-entities = false
http://apache.org/xml/features/disallow-doctype-decl = true
It allows to
instructs the implementation to process XML securely. This may set limits on XML constructs to avoid conditions such as denial of
service attacks.
do not include external general entities.
do not include external parameter entities or the external DTD subset
throw a fatal error if the incoming document contains a DOCTYPE declaration
JSON
All input must be validated using the OWASP lib json-sanitizer. See https://www.owasp.org/index.php/OWASP_JSON_Sanitizer
Example :
String wellFormedJson = JsonSanitizer.sanitize(jsonData);
try
{
return new ObjectMapper().readValue(wellFormedJson, JsonNode.class).toString();
}
catch (final IOException ex)
{
LOG.error("Incorrect json data : " + wellFormedJson, ex);
}
LOG
String coming from outside the application must not be directly logged to prevent log injection.
Controller case
In web context all controllers must extends BaseController. This class contains the method logParam which should be used to log something unknown.
This method uses YSanitizer.sanitize(input).
public class YSanitizer
{
public static String sanitize(final String input) {
String output = StringUtils.defaultString(input).trim();
output = output.replaceAll("(\\r\\n|\\r|\\n)+", " ");
output = StringEscapeUtils.escapeHtml(output);
return output;
}
}
Other case
Calling StringEscapeUtils.escapeJava(valToLog) should be enough.
Protect sensitive data from heap inspection
Because heap can be inspected, sensitive data should not be stored in String objects.
Indeed Strings are immutable and can stay in the heap for a while.
To prevent this issue all sensitive strings must be stored in a char[].
This array should be filled with "0" as soon as possible (when the value is not needed).
Not that this method is not 100% safe but reduced the time window to find the password in the heap.
Cross-Site Scripting (XSS)
Make sure de.hybris.platform.servicelayer.web.XSSFilter is in the filters list of incoming requests
Deployment checks (from Go-Live Checklist)
Verify that the default passwords for all users have been changed
Change admin user password for production
Disable automatic login or pre-populated passwords for
Product cockpit
CMS cockpit
CS cockpit
hMC
Password encoding should be MD5 or better SHA256
Change default password encoder
Change SALT for MD5 and SHA256 password encoder
Verify the database password and the requirement to store them in plain text in local.properties.
Verify that user account and checkout pages are only accessiable via a secure SSL connection
Check that a Web application firewall is in place
Perform a code review to ensure that no sensitive data, like credit card information or passwords, are logged to the log file
Verify that the hybris application server is not running as root
Secure the JMX connected

dynamic http response based on http request in java

We are building small test simulators that need to respond to restful requests injected into a platform based on content we inject into the message:
Example: GET http://server/example-app/users
Content-Headers and/or query params = some value of pass with 200
Server respond with 200 and content
Example: GET http://server/example-app/users
Content-Headers and/or query params = some value of fail with 400
Server respond with 400 and error
I am looking to see if anyone knows of an open source tool that would be of usefulness to inspect / parse the http request and figure out response based on the look-up criteria. I'm sure I could write some parsers easily, but just interested if the community has used or knows of something that is available to do the parsing and response mappings.

It's possible that the OWASP Zed Attack Proxy covers your needs.

Fiddler is an excellent proxy to inspect code and perform filtering: http://fiddler2.com/

AS2: Does xml.sendAndLoad use POST or GET?

All,
I'm trying to find out, unambiguously, what method (GET or POST) Flash/AS2 uses with XML.sendAndLoad.
Here's what the help/docs (http://livedocs.adobe.com/flash/9.0/main/wwhelp/wwhimpl/common/html/wwhelp.htm?context=LiveDocs_Parts&file=00002340.html) say about the function
Encodes the specified XML object into
an XML document, sends it to the
specified URL using the POST method,
downloads the server's response, and
loads it into the resultXMLobject
specified in the parameters.
However, I'm using this method to send XML data to a Java Servlet developed and maintained by another team of developers. And they're seeing log entries that look like this:
GET /portal/delegate/[someService]?svc=setPayCheckInfo&XMLStr=[an encoded version of the XML I send]
After a Google search to figure out why the POST shows up as a GET in their log, I found this Adobe technote (http://kb2.adobe.com/cps/159/tn_15908.html). Here's what it says:
When loadVariables or getURL actions are
used to send data to Java servlets it
can appear that the data is being sent
using a GET request, when the POST
method was specified in the Flash
movie.
This happens because Flash sends the
data in a GET/POST hybrid format. If
the data were being sent using a GET
request, the variables would appear in
a query string appended to the end of
the URL. Flash uses a GET server
request, but the Name/Value pairs
containing the variables are sent in a
second transmission using POST.
Although this causes the servlet to
trigger the doGet() method, the
variables are still available in the
server request.
I don't really understand that. What is a "GET/POST hybrid format"?
Why does the method Flash uses (POST or GET) depend on whether the data is sent to a Java servlet or elsewhere (e.g., a PHP page?)
Can anyone make sense of this? Many thanks in advance!
Cheers,
Matt

Have you try doing something like that :
var sendVar=new LoadVars();
var xml=new XML("<r>test</r>");
sendVar.xml=xml;
sendVar.svc="setPayCheckInfo";
var receiveXML=new XML();
function onLoad(success) {
if (success) {
trace("receive:"+receiveXML);
} else {
trace('error');
}
}
receiveXML.onLoad=onLoad;
sendVar.sendAndLoad("http://mywebserver", receiveXML, "POST");

The hybrid format is just a term Macromedia invented to paint over its misuse of HTTP.
HTTP is very vague on what you can do with GET and POST. But the convention is that no message body is used in GET. Adobe violates this convention by sending parameters in the message body.
Flash sends the same request regardless of the server. You have problem in Servlet because most implementation (like Tomcat) ignores message body for GET. PHP doesn't care the verb and it processes the message body for GET too.