We Keep Coding

Encoding a URL Query Parameter so it can have a '+'

Apparently, in the move from Spring Boot 1 to Spring Boot 2 (Spring 5), the encoding behavior of URL parameters for RestTemplates changed. It seems unusually difficult to get a general query parameter on rest templates passed so that characters that have special meanings such as "+" get properly escaped. It seems that, since "+" is a valid character, it doesn't get escaped, even though its meaning gets altered (see here). This seems bizarre, counter-intuitive, and against every other convention on every other platform. More importantly, I can't figure out how to easily get around it. If I encode the string first, it gets double-encoded, because the "%"s get re-encoded. Anyway, this seems like it should be something very simple that the framework does, but I'm not figuring it out.
Here is my code that worked in Spring Boot 1:
String url = "https://base/url/here";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), entry.getValue());
}
HttpEntity<TheResponse> resp = myRestTemplate.exchange(builder.toUriString(), ...);
However, now it won't encode the "+" character, so the other end is interpreting it as a space. What is the correct way to build this URL in Java Spring Boot 2?
Note - I also tried this, but it actually DOUBLE-encodes everything:
try {
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), URLEncoder.encode(entry.getValue(),"UTF-8" ));
}
} catch(Exception e) {
System.out.println("Encoding error");
}
In the first one, if I put in "q" => "abc+1#efx.com", then, exactly in the URL, I get "abc+1#efx.com" (i.e., not encoded at all). However, in the second one, if I put in "abc+1#efx.com", then I get "abc%252B1%2540efx.com", which is DOUBLE-encoded.
I could hand-write an encoding method, but this seems (a) like overkill, and (b) doing encoding yourself is where security problems and weird bugs tend to creep in. But it seems insane to me that you can't just add a query parameter in Spring Boot 2. That seems like a basic task. What am I missing?

Found what I believe to be a decent solution. It turns out that a large part of the problem is actually the "exchange" function, which takes a string for a URL, but then re-encodes that URL for reasons I cannot fathom. However, the exchange function can be sent a java.net.URI instead. In this case, it does not try to interpolate anything, as it is already a URI. I then use java.net.URLEncoder.encode() to encode the pieces. I still have no idea why this isn't standard in Spring, but this should work.
private String mapToQueryString(Map<String, String> query) {
List<String> entries = new LinkedList<String>();
for (Map.Entry<String, String> entry : query.entrySet()) {
try {
entries.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" + URLEncoder.encode(entry.getValue(), "UTF-8"));
} catch(Exception e) {
log.error("Unable to encode string for URL: " + entry.getKey() + " / " + entry.getValue(), e);
}
}
return String.join("&", entries);
}
/* Later in the code */
String endpoint = "https://baseurl.example.com/blah";
String finalUrl = query.isEmpty() ? endpoint : endpoint + "?" + mapToQueryString(query);
URI uri;
try {
uri = new URI(finalUrl);
} catch(URISyntaxException e) {
log.error("Bad URL // " + finalUrl, e);
return null;
}
}
/* ... */
HttpEntity<TheResponse> resp = myRestTemplate.exchange(uri, ...)

Android Programming - Post Google Form/Spreedsheet (REQUIRED FIELD)

So I have successfully post data onto a Google Spreadsheet using the Google Form source. Everything works perfect UNTIL I make the field (in the Google Form) "required." When I do that, the Android Emulator still responds as if the information sent was properly saved. But on the Google spreadsheet it isn't there.
Am I missing something?
This is my AsyncTask:
new BackgroundTask().execute(
"https://docs.google.com/forms/d/10QStmb9Nr-hcfv889FMSNTZdA_hNUErxeK7vISzkx0E/formResponse",
student.FirstName, "entry_2030274183=",
student.LastName, "entry_1558758483=",
student.Age, "entry_1871336861=",
student.Gender, "entry.2013677542=",
student.Grade, "entry_1921311866=");
This is my Background.
protected String doInBackground(String... params) {
HttpRequest reg = new HttpRequest();
String URL = params[0];
String FirstName = params[1];
String FirstNameEntry = params[2];
String LastName = params[3];
String LastNameEntry = params[4];
String Age = params[5];
String AgeEntry = params[6];
String Gender = params[7];
String GenderEntry = params[8];
String Grade = params[9];
String GradeEntry = params[10];
#SuppressWarnings("deprecation")
String data =
FirstNameEntry + URLEncoder.encode(FirstName) + "&" +
LastNameEntry + URLEncoder.encode(LastName) + "&" +
AgeEntry + URLEncoder.encode(Gender) + "&" +
GenderEntry + URLEncoder.encode(Age) + "&" +
GradeEntry + URLEncoder.encode(Grade);
String response = reg.sendPost(URL, data);
return response;
}
Do I need to put something in the entries if it is a required field?
If you want to look at the HttpRequest class go here (Not My Code):
Secure HTTP Post in Android
Much Appreciated

The only way I can immediately think of is by processing the response and then making your app behave accordingly.
For instance - I tried one test form and if the request send had some required field empty, then the HTTPResponse contains "Looks like you have a question or two that still need attention".
Another way would be to validate if the save was actually successful by searching for the text you gave in the "Confirmation Page".
In both cases, you should be able to differentiate between a successful post and a failed one.

Gwt update TextArea from Server

I have created a Textarea in my app. And I have one method in my server i.e, in
GreetingServiceImpl class
The sample code of my method in GreetingServiceImpl class is:
public String greetServer(String input) throws IllegalArgumentException {
System.out.println("input===>>" + input);
String serverInfo = getServletContext().getServerInfo();
System.out.println("serverinfo===>>" + serverInfo);
String userAgent = getThreadLocalRequest().getHeader("User-Agent");
System.out.println("User agent===" + userAgent);
input = escapeHtml(input);
System.out.println("2....input===>>" + input);
userAgent = escapeHtml(userAgent);
return "Hello, " + input + "!<br><br>I am running " + serverInfo
+ ".<br><br>It looks like you are using:<br>" + userAgent;
}
Now in my client I will call this method, after calling this method I want to setText to my textarea. The text should come from the server.
i.e, I have 4 sysout statements in this method.
System.out.println("input===>>" + input);
System.out.println("serverinfo===>>" + serverInfo);
System.out.println("User agent===" + userAgent);
System.out.println("2....input===>>" + input);
When these statements prints into eclipse console , at the same time I want to print this in order(one by one according to their execution)in my UI i.e, into my textarea. I have no Idea how to achieve this. Please tel me is this possible to do, if so how I can achieve?
PS: I'm looking for a logger type functionality which can update my textarea when ever any server side mathods executed.

It's possible to push data from server to client using Atmosphere which supports WebSockets and GWT.

Why don't you use RPC calls? It is straightforward and easy.
You just create an Example.java and ExampleAsync class at client, and the implementation at server. You can then call the implementation with an AsyncCallback.
For example:
callback = new AsyncCallback() {
public void onSuccess(Void result) {
// Make what ever you want! For example, set the textarea
}
On the other hand, if you want server to notify and update client, you can use Server-Push
You can also use gwt-comet , which also streams messages over http.

How to fix the HTTP Response splitting vulnerability with ESAPI

after a recent findbugs (FB) run it complains about a: Security - HTTP Response splitting vulnerability The following code triggers it:
String referrer = req.getParameter("referrer");
if (referrer != null) {
launchURL += "&referrer="+(referrer);
}
resp.sendRedirect(launchURL);
Basically the 'referrer' http parameter contains an url, to which, when clicking on a back button in our application the browser returns to. It is appended to the url as a parameter. After a bit research i know that i need to sanitize the referrer url. After a bit more research i found the esapi project which seem to offer this kind of functionality:
//1st canonicalize
import org.owasp.esapi.Encoder;
import org.owasp.esapi.Validator;
import org.owasp.esapi.reference.DefaultEncoder;
import org.owasp.esapi.reference.DefaultValidator;
[...]
Encoder encoder = new DefaultEncoder(new ArrayList<String>());
String cReferrer = encoder.canonicalize(referrer);
However I didn't figure out how to detect e.g. jscript code or other stuff which doesn't belong to a referrer url. So how can I achieve that with esapi?
I tried:
Validator validator = new DefaultValidator(encoder);
validator.isValidInput("Redirect URL",referrer,"HTTPParameterValue",512,false);
however this doesn't work. What I need is a function which results in:
http://www.google.com (ok)
http://www.google.com/login?dest=http://google.com/%0D%0ALocation: javascript:%0D%0A%0D%0Aalert(document.cookie) (not ok)
Or is it enough to call the following statement?
encoder.encodeForHTMLAttribute(referrer);
Any help appreciated.

Here's my final solution if anyone is interested. First I canonicalize and then URL decode the string. If a CR or LF exists (\n \r) I just cut of the rest of that potential 'attack' string starting with \n or \r.
String sanitize(String url) throws EncodingException{
Encoder encoder = new DefaultEncoder(new ArrayList<String>());
//first canonicalize
String clean = encoder.canonicalize(url).trim();
//then url decode
clean = encoder.decodeFromURL(clean);
//detect and remove any existent \r\n == %0D%0A == CRLF to prevent HTTP Response Splitting
int idxR = clean.indexOf('\r');
int idxN = clean.indexOf('\n');
if(idxN >= 0 || idxR>=0){
if(idxN<idxR){
//just cut off the part after the LF
clean = clean.substring(0,idxN);
}
else{
//just cut off the part after the CR
clean = clean.substring(0,idxR);
}
}
//re-encode again
return encoder.encodeForURL(clean);
}
Theoretically i could have later verified the value against 'HTTPParameterValue' regex which is defined in the ESAPI.properties however it didn't like colon in the http:// and I didn't investigated further.
And one more remark after testing it: Most modern browser nowadays (Firefox > 3.6, Chrome, IE10 etc.) detect this kind of vulnerability and do not execute the code...

I think you have the right idea, but are using an inappropriate encoder. The Referer [sic] header value is really a URL, not an HTML attribute, so you really want to use:
encoder.encodeForURL(referrer);
-kevin

I would suggest white-listing approach wherein you check the referrer string only for permissible characters. Regex would be a good option.
EDIT:
The class org.owasp.esapi.reference.DefaultEncoder being used by you is not really encoding anything. Look at the source code of the method encodeForHTMLAttribute(referrer) here at grepcode. A typical URL encoding (encoding carriage return and line feed) too wont help.
So the way forward would be device some validation logic which checks for valid set of characters. Here is another insightful article.

The accepted answer will not work if in case there is "\n\r" in the string.
Example:
If I have string: "This is str\n\rstr", it returns "This is str\nstr"
Rectified version of above accepted answer is:
String sanitizeCarriageReturns(String value) {
int idxR = value.indexOf('\r');
int idxN = value.indexOf('\n');
if (idxN >= 0 || idxR >= 0) {
if ((idxN > idxR && idxR<0) || (idxR > idxN && idxR>=0)) {
value = value.substring(0, idxN);
} else if (idxN < idxR){
value = value.substring(0, idxR);
}
}
return value;
}

JavaScript Error in BIRT

I have been assigned to clean up a project for a client that uses BIRT reporting. I have fixed most of the issues but I still have one report that is not working and is returning an error. The error is:
Row (id = 1467):
+ There are errors evaluating script "var fileName = row["Attached_File"];
params["HyperlinkParameter"].value = ImageDecoder.decodeDocs(row["Ecrash_Attach"],fileName);":
Wrapped java.lang.NullPointerException (/report/body/table[#id="61"]/detail/row[#id="70"]/cell[#id="71"]/grid[#id="1460"]/row[#id="1462"]/cell[#id="1463"]/table[#id="1464"]/detail/row[#id="1467"]/method[#name="onCreate"]#2)
I can post the full stack trace if someone wants it but for now I will omit it since it is very long.
Here is the source of the decodeDocs method:
public static String decodeDocs(byte[] source, String fileName) {
String randName = "";
byte[] docSource = null;
if ( Base64.isArrayByteBase64(source) ){
docSource = Base64.decodeBase64(source);
}
documentZipPath = writeByteStreamToFile(source);
randName = writeByteStreamToFile(docSource, fileName);
return randName;
}
I am pretty well lost on this one. The error looks to be telling me there is an error on line two of the script which is:
var fileName = row["Attached_File"];
params["HyperlinkParameter"].value = ImageDecoder.decodeDocs(row["Ecrash_Attach"],fileName);
This is written in the OnCreate method of the report. Any help, even clues would be greatly appreciated. If you would like to see the report just ask and I will post the xml for it.

A common mistake I make in BIRT is to access the value of a null report parameter.
In your case, could params["HyperlinkParameter"] be null?