Java Jersey REST Request Parameter Sanitation - java
I'm trying to make sure my Jersey request parameters are sanitized.
When processing a Jersey GET request, do I need to filter non String types?
For example, if the parameter submitted is an integer are both option 1 (getIntData) and option 2 (getStringData) hacker safe? What about a JSON PUT request, is my ESAPI implementation enough, or do I need to validate each data parameter after it is mapped? Could it be validated before it is mapped?
Jersey Rest Example Class:
public class RestExample {
//Option 1 Submit data as an Integer
//Jersey throws an internal server error if the type is not Integer
//Is that a valid way to validate the data?
//Integer Data, not filtered
#Path("/data/int/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getIntData(#PathParam("data") Integer data){
return Response.ok("You entered:" + data).build();
}
//Option 2 Submit data as a String, then validate it and cast it to an Integer
//String Data, filtered
#Path("/data/string/{data}/")
#GET
#Produces(MediaType.TEXT_HTML)
public Response getStringData(#PathParam("data") String data) {
data = ESAPI.encoder().canonicalize(data);
if (ESAPI.validator().isValidInteger("data", data, 0, 999999, false))
{
int intData = Integer.parseInt(data);
return Response.ok("You entered:" + intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
//JSON data, HTML encoded
#Path("/post/{requestid}")
#POST
#Consumes({MediaType.APPLICATION_FORM_URLENCODED, MediaType.APPLICATION_JSON})
#Produces(MediaType.TEXT_HTML)
public Response postData(String json) {
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
//Is there a way to iterate through each JSON KeyValue and filter here?
ObjectMapper mapper = new ObjectMapper();
DataMap dm = new DataMap();
try {
dm = mapper.readValue(json, DataMap.class);
} catch (Exception e) {
e.printStackTrace();
}
//Do we need to validate each DataMap object value and is there a dynamic way to do it?
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
return Response.status(404).entity("404 Not Found").build();
}
}
Data Map Class:
public class DataMap {
public DataMap(){}
String strData;
Integer intData;
}
The short answer is yes, though by "filter" I interpret it as "validate," because no amount of "filtering" will EVER provide you with SAFE data. You can still run into integer overflows in Java, and while those may not have immediate security concerns, they could still put parts of your application in an unplanned for state, and hacking is all about perturbing the system in ways you can control.
You packed waaaaay too many questions into one "question," but here we go:
First off, the lines
json = ESAPI.encoder().canonicalize(json);
json = ESAPI.encoder().encodeForHTML(json);
Aren't doing what you think they're doing. If your JSON is coming in as a raw String right here, these two calls are going to be applying mass rules across the entire string, when you really need to handle these with more surgical precision, which you seem to at least be subconsciously aware of in the next question.
//Is there a way to iterate through each JSON KeyValue and filter
here?
Partial duplicate of this question.
While you're in the loop discussed here, you can perform any data transformations you want, but what you should really be considering is using the JSONObject class referenced in that first link. Then you'll have JSON parsed into an object where you'll have better access to JSON key/value pairs.
//Do we need to validate each DataMap object value and is there a
dynamic way to do it?
Yes, we validate everything that comes from a user. All users are assumed to be trained hackers, and smarter than you. However if you handled filtering before you do your data mapping transformation, you don't need to do it a second time. Doing it dynamically?
Something like:
JSONObject json = new JSONObject(s);
Iterator iterator = json.keys();
while( iterator.hasNext() ){
String data = iterator.next();
//filter and or business logic
}
^^That syntax is skipping typechecks but it should get you where you need to go.
/Is Integer validation needed or will the thrown exception be good
enough?
I don't see where you're throwing an exception with these lines of code:
if (ESAPI.validator().isValidInput("strData", dm.strData, "HTTPParameterValue", 25, false, true))
{
//Is Integer validation needed or will the thrown exception be good enough?
return Response.ok("You entered:" + dm.strData + " and " + dm.intData).build();
}
Firstly, in java we have autoboxing which means this:
int foo = 555555;
String bar = "";
//the code
foo + bar;
Will be cast to a string in any instance. The compiler will promote the int to an Integer and then silently call the Integer.toString() method. Also, in your Response.ok( String ); call, THIS is where you're going to want to encodeForHTML or whatever the output context may be. Encoding methods are ALWAYS For outputting data to user, whereas canonicalize you want to call when receiving data. Finally, in this segment of code we also have an error where you're assuming that you're dealing with an HTTPParameter. NOT at this point in the code. You'll validate http Parameters in instances where you're calling request.getParameter("id"): where id isn't a large blob of data like an entire JSON response or an entire XML response. At this point you should be validating for things like "SafeString"
Usually there are parsing libraries in Java that can at least get you to the level of Java objects, but on the validation side you're always going to be running through every item and punting whatever might be malicious.
As a final note, while coding, keep these principles in mind your code will be cleaner and your thought process much more focused:
user input is NEVER safe. (Yes, even if you've run it through an XSS filter.)
Use validate and canonicalize methods whenever RECEIVING data, and encode methods whenever transferring data to a different context, where context is defined as "Html field. Http attribute. Javascript input, etc...)
Instead of using the method isValidInput() I'd suggest using getValidInput() because it will call canonicalize for you, making you have to provide one less call.
Encode ANY time your data is going to be passed to another dynamic language, like SQL, groovy, Perl, or javascript.
Related
Using Optionals to Validate Incoming Requests
I have a use case where I am needing to validate incoming requests with a bunch of fields, and if certain ones are null or empty, I want to return a detailed response from my API letting the client know that they need to include a value for that field. I am new to Optionals, and think there may be a more efficient and streamlined way to do this.
I have an incoming request:
public class myRequest {
private String fieldOne;
private String fieldTwo:
..etc
}
And I have validations in place :
if(StringUtils.isEmpty(request.getFieldOne)){
return new MyResponse("Please include a field one value")
}else if(StringUtils.isEmpty(request.getFieldTwo)){
return new MyResponse("Please include a field two value")
}else if(StringUtils.isEmpty(request.getFieldThree){
//etc..
}
Now I would think i converted the incoming requests to an optional, I would be able to validate these fields more efficiently.
Optional<MyRequest> request = Optional.of(request);
request.ifPresent((req ) ->{
//checks if the request itself is null...but what about fields, or objects?
});
Some of the incoming fields are objects themselves as well. Wondering if anyone has come across this.
HttpServer Request get date range from query string
I am new to Java and Vertx and I have a query string with the following format: GET /examples/1/data?date_1[gt]=2021-09-28&date_1[lt]=2021-10-28 Here I have this date_1 parameter which is within a certain range. I have been using HttpServerRequest class to extract simple parameters like integers but not sure how to proceed with these kind of range parameters. With the simple parameters, I can do something like: String param = request.getParam(paramName); paramAsInteger = Integer.valueOf(paramAsString); However, confused as to how to deal with the gt and lt options and the fact that we have same parameter twice.
You say that you have difficulties parsing out these tokens. Here's how you can handle this.
The first thing to understand is that the parameter name is NOT "date1"
There are actually two parameters here
2.1. "date_1[gt]" with a value of "2021-09-28"
2.2. "date_1[lt]" with a value of "2021-10-28"
This is because in the URI parameter definition everything before the "=" sign is the parameter name and everything after is the parameter value.
You can just do
String dateAsString = request.getParam("date1[gt]");
paramAsInteger = toDate(dateAsString)
To implement the toDate() function read this simple article how to convert a string object into a data object using a standard library
(link)
Vert.x will treat these parameters as two separate ones. So RoutingContext#queryParam("date_1[gt]") will only give you the value for [gt]. If you want the value for [lt] you need to get that separately.
That being said, you can move this tedious logic into an extra handler and store the values in the RoutingContext. Something like this might be easier:
private void extractDates(RoutingContext ctx) {
var startDate = ctx.queryParam("date_1[gt]");
var endDate = ctx.queryParam("date_1[lt]");
var parsedStartDate = DateTimeFormatter.ISO_LOCAL_DATE.parse(startDate.get(0));
var parsedEndDate = DateTimeFormatter.ISO_LOCAL_DATE.parse(endDate.get(0));
// things we put in the context here can be retrieved by later handlers
ctx.put("startDate", parsedStartDate);
ctx.put("endDate", parsedEndDate);
ctx.next();
}
Then, in your actual handler you can access the two dates as follows:
router.get("/date")
.handler(this::extractDates)
.handler(ctx -> {
var responseBody = ctx.get("startDate") + " - " + ctx.get("endDate");
ctx.end(responseBody);
});
This allows you to keep your actual business logic concise.
Validating Query Params in REST API
I have a REST API which accepts query parameters. The query parameters are valid if and only if at a time only one query parameter is passed and it is among the list of valid query parameters.
Currently my logic for this is:
I am collecting the query params in a map. And then checking it's size. If size > 1 function throwing an error. If that is not the case then iterating through the map and if find a parameter other than valid ones the function throwing an error.
For example:
if(queryParam.size()>1) {
throw new FailureResponse();
}
queryParam.forEach(e->{
String key = e.getKey();
if(!key.equalsIgnoreCase("p1") && !key.equalsIgnoreCase("p2")) {
throw new FailureResponse();
}
});
But I think in this way I am violating the SOLID design principle which says a class should be open for extension but closed for modification.
I also thought of creating a file and then reading the acceptable params from it but that would add to the response time of the API as it involves reading a file.
Is there some way I can keep and read the valid query-params and it does not violate the design principles?
You could maintain an Enum of valid params and extend the enums as and when applicable like
public enum QueryParams{
PARAM_1("param1"),
PARAM_2("param2"),
private String paramValue;
QueryParams(String paramName){
this.paramValue = paramValue();
}
public void getParamValue(){
return this.value;
}
}
and then you could iterate over the set of values of this enum to filter out invalid values
List<String> validParams = Arrays.asList(QueryParams.values()).stream().map(QueryParams::getParamValue).collect(Collectors.toList());
queryParams.removeAll(validParams);
if(queryParams.size()!=0) {
throw new FailureResponse();
}
}
This helps you maintain the API class without any changes, whenever a new parameter is added, just extend the enum and all the rest is automatically extended as it all depends upon the value in the enum.
How to cast data in response with Oligo Odata V4 client?
I wrote a code in java to create a Olingo Odata client V4. I am able to send queries and receive a response and extract each elements. I want to go further and make some calculations with data I receive:
-I would like to know if it is possible in java to map data I received into a class.
-I am wondering if it is possible to cast data into String, Integer, Long, Double in another way I did below:
public List<ItemSales> getItemSale(ClientEntitySetIterator<ClientEntitySet, ClientEntity> iteratorItemSales)
{
List<ItemSales> itemSalesList = new ArrayList<ItemSales>();
while (iteratorItemSales.hasNext()) {
ClientEntity resultsIterator = iteratorItemSales.next();
List<ClientProperty> properties = resultsIterator.getProperties();
ItemSales itemsales = new ItemSales();
//EdmCast
for (ClientProperty property : properties)
{
switch( property.getName().toLowerCase())
{
case "id": itemsales.setId(Long.parseLong(property.getValue().toString())); break;
}
ClientValue value =property.getValue();
}
}
return itemSalesList;
}
Indeed, once I get in my example, I want to get a long value. Nevertheless all elements of the list I got are in "ClientValue" format.
Spring REST Controller understanding arrays of strings when having special characters like blank spaces or commas
I am trying to write a Spring REST Controller getting an array of strings as input parameter of a HTTP GET request.
The problem arises when in the GET request, in some of the strings of the array, I use special characters like commas ,, blank spaces or forward slash /, no matter if I URL encode the query part of the URL HTTP GET request.
That means that the string "1/4 cup ricotta, yogurt" (edit which needs to be considered as a unique ingredient contained as a string element of the input array) in either this format:
http://127.0.0.1:8080/[...]/parseThis?[...]&ingredients=1/4 cup ricotta, yogurt
This format (please note the blank spaces encoded as + plus, rather than the hex code):
http://127.0.0.1:8080/[...]/parseThis?[...]&ingredients=1%2F4+cup+ricotta%2C+yogurt
Or this format (please note the blank space encoded as hex code %20):
http://127.0.0.1:8080/[...]/parseThis?[...]&ingredients=1%2F4%20cup%20ricotta%2C%20yogurt
is not rendered properly.
The system does not recognize the input string as one single element of the array.
In the 2nd and 3rd case the system splits the input string on the comma and returns an array of 2 elements rather than 1 element. I am expecting 1 element here.
The relevant code for the controller is:
#RequestMapping(
value = "/parseThis",
params = {
"language",
"ingredients"
}, method = RequestMethod.GET, headers = HttpHeaders.ACCEPT + "=" + MediaType.APPLICATION_JSON_VALUE)
#ResponseBody
public HttpEntity<CustomOutputObject> parseThis(
#RequestParam String language,
#RequestParam String[] ingredients){
try {
CustomOutputObject responseFullData = parsingService.parseThis(ingredients, language);
return new ResponseEntity<>(responseFullData, HttpStatus.OK);
} catch (Exception e) {
// TODO
}
}
I need to perform HTTP GET request against this Spring controller, that's a requirement (so no HTTP POST can be used here).
Edit 1:
If I add HttpServletRequest request to the signature of the method in the controller, then I add a log statement like log.debug("The query string is: '" + request.getQueryString() + "'"); then I am seeing in the log a line like The query string is: '&language=en&ingredients=1%2F4+cup+ricotta%2C+yogurt' (So still URL encoded).
Edit 2:
On the other hand if I add WebRequest request to the signature of the method, the the log as log.debug("The query string is: '" + request.getParameter("ingredients") + "'"); then I am getting a string in the log as The query string is: '1/4 cup ricotta, yogurt' (So URL decoded).
I am using Apache Tomcat as a server.
Is there any filter or something I need to add/review to the Spring/webapp configuration files?
Edit 3:
The main problem is in the interpretation of a comma:
#ResponseBody
#RequestMapping(value="test", method=RequestMethod.GET)
public String renderTest(#RequestParam("test") String[] test) {
return test.length + ": " + Arrays.toString(test);
// /app/test?test=foo,bar => 2: [foo, bar]
// /app/test?test=foo,bar&test=baz => 2: [foo,bar, baz]
}
Can this behavior be prevented?
The path of a request parameter to your method argument goes through parameter value extraction and then parameter value conversion. Now what happens is:
Extraction:
The parameter is extracted as a single String value. This is probably to allow simple attributes to be passed as simple string values for later value conversion.
Conversion:
Spring uses ConversionService for the value conversion. In its default setup StringToArrayConverter is used, which unfortunately handles the string as comma delimited list.
What to do:
You are pretty much screwed with the way Spring handles single valued request parameters. So I would do the binding manually:
// Method annotations
public HttpEntity<CustomOutputObject> handlerMethod(WebRequest request) {
String[] ingredients = request.getParameterValues("ingredients");
// Do other stuff
}
You can also check what Spring guys have to say about this.. and the related SO question.
Well, you could register a custom conversion service (from this SO answer), but that seems like a lot of work. :) If it were me, I would ignore the declaration the #RequestParam in the method signature and parse the value using the incoming request object.
May I suggest you try the following format: ingredients=egg&ingredients=milk&ingredients=butter Appending &ingredients to the end will handle the case where the array only has a single value. ingredients=egg&ingredients=milk&ingredients=butter&ingredients ingredients=milk,skimmed&ingredients The extra entry would need to be removed from the array, using a List<String> would make this easier. Alternatively if you are trying to implement a REST controller to pipe straight into a database with spring-data-jpa, you should take a look at spring-data-rest. Here is an example. You basically annotate your repository with #RepositoryRestResource and spring does the rest :)
A solution from here
public String get(WebRequest req) {
String[] ingredients = req.getParameterValues("ingredients");
for(String ingredient:ingredients ) {
System.out.println(ingredient);
}
...
}
This works for the case when you have a single ingredient containing commas