Apparently, in the move from Spring Boot 1 to Spring Boot 2 (Spring 5), the encoding behavior of URL parameters for RestTemplates changed. It seems unusually difficult to get a general query parameter on rest templates passed so that characters that have special meanings such as "+" get properly escaped. It seems that, since "+" is a valid character, it doesn't get escaped, even though its meaning gets altered (see here). This seems bizarre, counter-intuitive, and against every other convention on every other platform. More importantly, I can't figure out how to easily get around it. If I encode the string first, it gets double-encoded, because the "%"s get re-encoded. Anyway, this seems like it should be something very simple that the framework does, but I'm not figuring it out.
Here is my code that worked in Spring Boot 1:
String url = "https://base/url/here";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), entry.getValue());
}
HttpEntity<TheResponse> resp = myRestTemplate.exchange(builder.toUriString(), ...);
However, now it won't encode the "+" character, so the other end is interpreting it as a space. What is the correct way to build this URL in Java Spring Boot 2?
Note - I also tried this, but it actually DOUBLE-encodes everything:
try {
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), URLEncoder.encode(entry.getValue(),"UTF-8" ));
}
} catch(Exception e) {
System.out.println("Encoding error");
}
In the first one, if I put in "q" => "abc+1#efx.com", then, exactly in the URL, I get "abc+1#efx.com" (i.e., not encoded at all). However, in the second one, if I put in "abc+1#efx.com", then I get "abc%252B1%2540efx.com", which is DOUBLE-encoded.
I could hand-write an encoding method, but this seems (a) like overkill, and (b) doing encoding yourself is where security problems and weird bugs tend to creep in. But it seems insane to me that you can't just add a query parameter in Spring Boot 2. That seems like a basic task. What am I missing?
Found what I believe to be a decent solution. It turns out that a large part of the problem is actually the "exchange" function, which takes a string for a URL, but then re-encodes that URL for reasons I cannot fathom. However, the exchange function can be sent a java.net.URI instead. In this case, it does not try to interpolate anything, as it is already a URI. I then use java.net.URLEncoder.encode() to encode the pieces. I still have no idea why this isn't standard in Spring, but this should work.
private String mapToQueryString(Map<String, String> query) {
List<String> entries = new LinkedList<String>();
for (Map.Entry<String, String> entry : query.entrySet()) {
try {
entries.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" + URLEncoder.encode(entry.getValue(), "UTF-8"));
} catch(Exception e) {
log.error("Unable to encode string for URL: " + entry.getKey() + " / " + entry.getValue(), e);
}
}
return String.join("&", entries);
}
/* Later in the code */
String endpoint = "https://baseurl.example.com/blah";
String finalUrl = query.isEmpty() ? endpoint : endpoint + "?" + mapToQueryString(query);
URI uri;
try {
uri = new URI(finalUrl);
} catch(URISyntaxException e) {
log.error("Bad URL // " + finalUrl, e);
return null;
}
}
/* ... */
HttpEntity<TheResponse> resp = myRestTemplate.exchange(uri, ...)
Related
I am getting a high severity issue in this method:
public void recordBadLogin(final String uid, final String reason, final String ip) throws DataAccessException {
if (Utils.isEmpty(uid)) {
throw new DataAccessException("User information needed to update , Empty user information passed");
}
try {
String sql = (String) this.queries.get(IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN);
Map<String, Object> paramMap = new HashMap<String, Object>();
paramMap.put("uid", uid.trim());
paramMap.put("reason", (reason != null ? reason.trim() : "Invalid userid/password"));
paramMap.put("ip", ip);
this.namedJdbcTemplate.update(sql, paramMap);
} catch (Exception e) {
throw new DataAccessException("Failed to record bad login for user " + uid, e);
}
}
This line of code is causing the issue:
String sql = (String) this.queries.get(IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN);
queries is a properties object and the prepared statement is being retrieved given IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN. And those 2 arguments are constants. Logically I don't see how this can cause an SQL injection issue as the prepared statement is being retrieved from a dictionary. Does anyone have an idea if this is a false positive or if there is an actual vulnerability present?
It's hard to tell from the example given, but I'd guess that the properties object was tainted by untrusted data. Most code flow analysis tools will taint the entire data structure if any untrusted data is placed in it.
Technically this is a "false positive". But architecturally it's something that should be fixed - it's generally a bad idea to mix trusted and untrusted data together in the same data structure. It makes it easy for future developers to misunderstand the status of a particular element, and makes it harder for both humans and tools to code review for security issues.
For the below mentioned code, I'm getting Trust Boundary Violation in the CheckMarx report.
Error description -
Method 'getResponse' gets user input from element request. This element’s value flows through the code without being properly sanitized or validated and is eventually stored in the server-side Session object, in 'parseRequest' method.**
Code -
#Context
HttpHeaders httpHeader;
void parseRequest(SomeRequestType inputRequest) {
HashMap<String, Data> requestData = inputRequest.getRequestData(httpHeader);
if (requestData != null) {
if (Strings.isNullOrEmpty(inputRequest.getId())) {
Data data = requestData.get("data");
var dataID = data.getID();
if ((dataID != null) && Pattern.matches("[0-9]+", dataID)) {
inputRequest.setId(dataID);
ThreadContext.put("ID", dataID);
}
}
}
}
I am getting checkmarx vulnerability at below line for without being properly sanitized or validated
ThreadContext.put("ID", dataID);
Could some please help me, how to properly sanitize the above line.
If you know for sure that dataID is a number, convert it to integer/long right away, like this:
int dataIDasNumber = Integer.parseInt(dataID);
And use it like int/long here:
inputRequest.setId(dataIDasNumber);
ThreadContext.put("ID", dataIDasNumber);
Then you don't need to do this:
Pattern.matches...
And your checkmarx violation should go away.
I hav following route:
from("quartz2:findAll//myGroup/myTimerName?cron=" + pushProperties.getQuartz())
//.setBody().constant("{ \"id\": \"FBJDBFJHSDBFJSBDfi\" }")
.to("mongodb:mongoBean?database=" + mongoDataConfiguration.getDatabase()
+ "&operation=findAll&collection=" + mongoDataConfiguration.getDataPointCollection())
.process(exchange -> {
exchange.getIn().setBody(objectMapper.writeValueAsString(exchange.getIn().getBody()));
}).streamCaching()
.setHeader(Exchange.HTTP_METHOD, constant(pushProperties.getHttpMethod()))
.setHeader(Exchange.CONTENT_TYPE, constant(MediaType.APPLICATION_JSON_VALUE))
.to(pushProperties.getUrl() + "&throwExceptionOnFailure=false").streamCaching()
As you can see I use throwExceptionOnFailure=false
and I take my url from configuration. But we found out that it works if
pushProperties.getUrl() = localhost:8080/url?action=myaction
and doesn't work in case of
pushProperties.getUrl() = localhost:8080/url
Is there universla way in camel to add request parameter to URL?
something like:
private String buildUrl() {
String url = pushProperties.getUrl();
return url + (url.contains("?") ? "&" : "?") + "throwExceptionOnFailure=false";
}
inside Camel api
That is because in case of localhost:8080/url, after appending it becomes like this
localhost:8080/url&throwExceptionOnFailure=false
which is wrong
It should be
localhost:8080/url?throwExceptionOnFailure=false,
In the first case it works you already have a requestpatam(?action=myaction) so the next one can be added with ampersand(&)
I think you have to add your own logic to compose the endpoint to the http component at the runtime. This is because the CamelContext will process it during the route itself. The parameter throwExceptionOnFailure is a property from the http component.
I don't think that adding the parameter via .setHeader(Exchange.HTTP_QUERY, constant("throwExceptionOnFailure=false")) shoud work because these parameters will be evaluated after the http component get processed, e.g. into the URL destination. Please, take a look at "How to use a dynamic URI in to()":
.toD(pushProperties.getUrl() + "&throwExceptionOnFailure=false")
You could use the simple expression to write a logic to do what you want based on the result of pushProperties.getUrl().
I don't like how Camel configure HTTP component in this case, but this is what it is.
What I suggest is to create a map of config, and append your args to it, and do a manual join with "&", then append it to the main url.
I do it like:
public class MyProcessor {
/**
* Part of Camel HTTP component config are done with URL query parameters.
*/
private static final Map<String, String> COMMON_QUERY_PARAMS = Map.of(
// do not throw HttpOperationFailedException; we handle them ourselves
"throwExceptionOnFailure", "false"
);
#Handler
void configure(Exchange exchange, ...) {
...
Map<String, String> queryParams = new HashMap<>();
queryParams.put("foo", "bar");
message.setHeader(Exchange.HTTP_QUERY, mergeAndJoin(queryParams));
...
}
private String mergeAndJoin(Map<String, String> queryParams) {
// make sure HTTP config params put after event params
return Stream.concat(queryParams.entrySet().stream(), COMMON_QUERY_PARAMS.entrySet().stream())
.map(entry -> entry.getKey() + "=" + entry.getValue())
.collect(Collectors.joining("&"));
}
}
Note that toD needs optimization but in that case, HTTP_QUERY cannot be used.
When the optimised component is in use, then you cannot use the headers Exchange.HTTP_PATH and Exchange.HTTP_QUERY to provide dynamic values to override the uri in toD. If you want to use these headers, then use the plain to DSL instead. In other words these headers are used internally by toD to carry the dynamic details of the endpoint.
https://camel.apache.org/components/3.20.x/eips/toD-eip.html
I'm looking for a way to catch any URL, such as:
mydomain.com/first/second/third
mydomain.com/first/second/third/fourth
mydomain.com/first/
and be able to catch it in a single method in my controller and build a string with the path, like:
for (String pathVar : pathVariableArray){
stringToBuild += pathVar + '.'
}
and get the following results:
stringToBuild = "first.second.third."
stringToBuild = "first.second.third.fourth."
stringToBuild = "first."
Is there any way? I don’t want to have to code various methods for a different length of the path.
#RequestMapping("/**")
public void method(HttpServletRequest request) {
String stringToBuild = request.getServletPath().replace("/", ".") + ".";
For alternate mapping methods, see Spring boot - Controller catching all URLs.
You may need to be more sophisticated to avoid an double . at the end if the path ends with a /.
In a program I'm working on in Java where I have to read data from a file. The data is formatted so that each line contains all the necessary information to construct a new object. When I parse the data, I have a block of code that looks something like this:
String[] parts = file.nextLine().split(",");
String attr1 = parts[0];
int attr2, attr3;
try{
attr2 = Integer.parseInt(parts[1]);
} catch (NumberFormatException ex){
System.out.println("Could not parse attr2, got " + parts[1] + ".");
return;
}
try{
attr3 = Integer.parseInt(parts[2]);
} catch (NumberFormatException ex){
System.out.println("Could not parse attr3, got " + parts[2] + ".");
return;
}
ClassA attr4 = null, attr5 = null, attr6 = null;
try{
...
} catch (SomeExceptionType ex){
System.out.println("Could not parse attr4, got " + parts[3] + ".");
}
...
I find myself repeating the same simple try block over and over again. In an attempt to mitigate the situation and adhere to the DRY principle a bit more, I introduced some attempt methods:
int attr2 = attemptGetInt(parts, 1, "attr2");
int attr3 = attemptGetInt(parts, 2, "attr3");
ClassA attr4 = attemptGetClassA(parts, 3, "attr4");
...
// Somewhere in the class
public int attemptGetInt(String[] parts, int index, String name) throws SomeOtherException1{
try{
return Integer.parseInt(parts[index]);
} catch (NumberFormatException ex){
throw new SomeOtherException1("Could not parse " + name + ", got " + parts[index] + ".");
}
}
public ClassA attemptGetClassA(String[] parts, int index, String name) throws SomeOtherException2{
try{
return ...
} catch (SomeExceptionType ex){
throw new SomeOtherException2("Could not parse " + name + ", got" + parts[index] + ".");
}
}
...
Even this feels weird though, because there are a lot of different types I have to return that all sort of have the same but slightly different code and need to catch a slightly different error each time (i.e. I have to create an attemptGetClassB and attemptGetClassC and so on, a bunch of times with similar code each time).
Is there an elegant way of writing code like this?
If you have control over the format of the input file you might wish to change it to XML with a schema. That way the parser itself takes care of a lot of this type of checking for you.
However from the nature of the question I assume the format is fixed. In that case I would suggest splitting the syntax checking and parsing into separate steps for each line.
An easy way to do the syntax checking is using a regexp. Fairly complex syntax can be encoded in a regular expression so unless the files contain some sort of nesting (in which case DEFINITELY use XML instead) then it should be fairly straightforward.
The second step of parsing should then only return exceptions by exception :-) You still need to catch them but it's perfectly good form to gather all of your catches into a single block because it should only be used when debugging: in normal operations the syntax check will catch errors before this step.
My view is that this design is more intuitive and obvious. It may have a downside in error reporting if you specifically want to report on each error separately. In that case you'll need to break the string into substrings first (using a Scanner for example) and then syntax check each substring.
As a final note, opinions vary on this but my personal preference is not to use exception handling for conditions that occur in normal operations. They are not well suited for that (in my opinion). Better to do what I'm suggesting here: have explicit code to check error conditions before processing and then use exceptions for things that should not normally occur.