We Keep Coding

Encoding a URL Query Parameter so it can have a '+'

Apparently, in the move from Spring Boot 1 to Spring Boot 2 (Spring 5), the encoding behavior of URL parameters for RestTemplates changed. It seems unusually difficult to get a general query parameter on rest templates passed so that characters that have special meanings such as "+" get properly escaped. It seems that, since "+" is a valid character, it doesn't get escaped, even though its meaning gets altered (see here). This seems bizarre, counter-intuitive, and against every other convention on every other platform. More importantly, I can't figure out how to easily get around it. If I encode the string first, it gets double-encoded, because the "%"s get re-encoded. Anyway, this seems like it should be something very simple that the framework does, but I'm not figuring it out.
Here is my code that worked in Spring Boot 1:
String url = "https://base/url/here";
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), entry.getValue());
}
HttpEntity<TheResponse> resp = myRestTemplate.exchange(builder.toUriString(), ...);
However, now it won't encode the "+" character, so the other end is interpreting it as a space. What is the correct way to build this URL in Java Spring Boot 2?
Note - I also tried this, but it actually DOUBLE-encodes everything:
try {
for (Map.Entry<String, String> entry : query.entrySet()) {
builder.queryParam(entry.getKey(), URLEncoder.encode(entry.getValue(),"UTF-8" ));
}
} catch(Exception e) {
System.out.println("Encoding error");
}
In the first one, if I put in "q" => "abc+1#efx.com", then, exactly in the URL, I get "abc+1#efx.com" (i.e., not encoded at all). However, in the second one, if I put in "abc+1#efx.com", then I get "abc%252B1%2540efx.com", which is DOUBLE-encoded.
I could hand-write an encoding method, but this seems (a) like overkill, and (b) doing encoding yourself is where security problems and weird bugs tend to creep in. But it seems insane to me that you can't just add a query parameter in Spring Boot 2. That seems like a basic task. What am I missing?

Found what I believe to be a decent solution. It turns out that a large part of the problem is actually the "exchange" function, which takes a string for a URL, but then re-encodes that URL for reasons I cannot fathom. However, the exchange function can be sent a java.net.URI instead. In this case, it does not try to interpolate anything, as it is already a URI. I then use java.net.URLEncoder.encode() to encode the pieces. I still have no idea why this isn't standard in Spring, but this should work.
private String mapToQueryString(Map<String, String> query) {
List<String> entries = new LinkedList<String>();
for (Map.Entry<String, String> entry : query.entrySet()) {
try {
entries.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" + URLEncoder.encode(entry.getValue(), "UTF-8"));
} catch(Exception e) {
log.error("Unable to encode string for URL: " + entry.getKey() + " / " + entry.getValue(), e);
}
}
return String.join("&", entries);
}
/* Later in the code */
String endpoint = "https://baseurl.example.com/blah";
String finalUrl = query.isEmpty() ? endpoint : endpoint + "?" + mapToQueryString(query);
URI uri;
try {
uri = new URI(finalUrl);
} catch(URISyntaxException e) {
log.error("Bad URL // " + finalUrl, e);
return null;
}
}
/* ... */
HttpEntity<TheResponse> resp = myRestTemplate.exchange(uri, ...)

Downloading attachments from unseen messages

I work on university project in java. I have to download attachments from new emails using GMAIL API.
I successfully connected to gmail account using OAuth 2.0 authorization.
private static final List<String> SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);
I tried to get unseen mails using
ListMessagesResponse listMessageResponse = service.users().messages().list(user).setQ("is:unseen").execute();
listMessageResponse is not null but when I call method .getResultSizeEstimate() it returns 0
also I tried to convert listMessageResponse to List < Message > (I guess this is more usable) using
List<Message> list = listMessageResponse.getMessages();
But list launches NullPointerException
Then tried to get each attachment with
for(Message m : list) {
List<MessagePart> part = m.getPayload().getParts();
for(MessagePart p: part) {
if(p.getFilename()!=null && p.getFilename().length()>0) {
System.out.println(p.getFilename()); // Just to check attachment filename
}
}
}
Is my approach correct (if not how to fix it) and how should I download those attachments.
EDIT 1:
Fixed q parameter, I mistakenly wrote is:unseen instead of is:unread.
Now app reaches unread mails successfully.
(For example there was two unread mails and both successfully reached, I can get theirs IDs easy).
Now this part trows NullPointerException
List<MessagePart> part = m.getPayload().getParts();
Both messages have attachments and m is not null (I get ID with .getID())
Any ideas how to overcome this and download attachment?
EDIT 2:
Attachments Downloading part
for(MessagePart p : parts) {
if ((p.getFilename() != null && p.getFilename().length() > 0)) {
String filename = p.getFilename();
String attId = p.getBody().getAttachmentId();
MessagePartBody attachPart;
FileOutputStream fileOutFile = null;
try {
attachPart = service.users().messages().attachments().get("me", p.getPartId(), attId).execute();
byte[] fileByteArray = Base64.decodeBase64(attachPart.getData());
fileOutFile = new FileOutputStream(filename); // Or any other dir
fileOutFile.write(fileByteArray);
fileOutFile.close();
}catch (IOException e) {
System.out.println("IO Exception processing attachment: " + filename);
} finally {
if (fileOutFile != null) {
try {
fileOutFile.close();
} catch (IOException e) {
// probably doesn't matter
}
}
}
}
}
Downloading working like charm, tested app with different type of emails.
Only thing left is to change label of unread message (that was reached by app) to read. Any tips how to do it?
And one tiny question:
I want this app to fetch mails on every 10 minutes using TimerTask abstract class. Is there need for manual "closing" of connection with gmail or that's done automatically after run() method iteration ends?
#Override
public void run(){
// Some fancy code
service.close(); // Something like that if even exists
}

I don't think ListMessagesResponse ever becomes null. Even if there are no messages that match your query, at least resultSizeEstimate will get populated in the resulting response: see Users.messages: list > Response.
I think you are using the correct approach, just that there is no message that matches your query. Actually, I never saw is:unseen before. Did you mean is:unread instead?
Update:
When using Users.messages: list only the id and the threadId of each message is populated, so you cannot access the message payload. In order to get the full message resource, you have to use Users.messages: get instead, as you can see in the referenced link:
Note that each message resource contains only an id and a threadId. Additional message details can be fetched using the messages.get method.
So in this case, after getting the list of messages, you have to iterate through the list, and do the following for each message in the list:
Get the message id via m.getId().
Once you have retrieved the message id, use it to call Gmail.Users.Messages.Get and get the full message resource. The retrieved message should have all fields populated, including payload, and you should be able to access the corresponding attachments.
Code sample:
List<Message> list = listMessageResponse.getMessages();
for(Message m : list) {
Message message = service.users().messages().get(user, m.getId()).execute();
List<MessagePart> part = message.getPayload().getParts();
// Rest of code
}
Reference:
Class ListMessagesResponse
Users.messages: list > Response

Checkmarx SQL injection high severity issue

I am getting a high severity issue in this method:
public void recordBadLogin(final String uid, final String reason, final String ip) throws DataAccessException {
if (Utils.isEmpty(uid)) {
throw new DataAccessException("User information needed to update , Empty user information passed");
}
try {
String sql = (String) this.queries.get(IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN);
Map<String, Object> paramMap = new HashMap<String, Object>();
paramMap.put("uid", uid.trim());
paramMap.put("reason", (reason != null ? reason.trim() : "Invalid userid/password"));
paramMap.put("ip", ip);
this.namedJdbcTemplate.update(sql, paramMap);
} catch (Exception e) {
throw new DataAccessException("Failed to record bad login for user " + uid, e);
}
}
This line of code is causing the issue:
String sql = (String) this.queries.get(IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN);
queries is a properties object and the prepared statement is being retrieved given IUtilDAO.queryKeyPrefix + UtilDAO.RECORD_FAILED_LOGIN. And those 2 arguments are constants. Logically I don't see how this can cause an SQL injection issue as the prepared statement is being retrieved from a dictionary. Does anyone have an idea if this is a false positive or if there is an actual vulnerability present?

It's hard to tell from the example given, but I'd guess that the properties object was tainted by untrusted data. Most code flow analysis tools will taint the entire data structure if any untrusted data is placed in it.
Technically this is a "false positive". But architecturally it's something that should be fixed - it's generally a bad idea to mix trusted and untrusted data together in the same data structure. It makes it easy for future developers to misunderstand the status of a particular element, and makes it harder for both humans and tools to code review for security issues.

How to fix the HTTP Response splitting vulnerability with ESAPI

after a recent findbugs (FB) run it complains about a: Security - HTTP Response splitting vulnerability The following code triggers it:
String referrer = req.getParameter("referrer");
if (referrer != null) {
launchURL += "&referrer="+(referrer);
}
resp.sendRedirect(launchURL);
Basically the 'referrer' http parameter contains an url, to which, when clicking on a back button in our application the browser returns to. It is appended to the url as a parameter. After a bit research i know that i need to sanitize the referrer url. After a bit more research i found the esapi project which seem to offer this kind of functionality:
//1st canonicalize
import org.owasp.esapi.Encoder;
import org.owasp.esapi.Validator;
import org.owasp.esapi.reference.DefaultEncoder;
import org.owasp.esapi.reference.DefaultValidator;
[...]
Encoder encoder = new DefaultEncoder(new ArrayList<String>());
String cReferrer = encoder.canonicalize(referrer);
However I didn't figure out how to detect e.g. jscript code or other stuff which doesn't belong to a referrer url. So how can I achieve that with esapi?
I tried:
Validator validator = new DefaultValidator(encoder);
validator.isValidInput("Redirect URL",referrer,"HTTPParameterValue",512,false);
however this doesn't work. What I need is a function which results in:
http://www.google.com (ok)
http://www.google.com/login?dest=http://google.com/%0D%0ALocation: javascript:%0D%0A%0D%0Aalert(document.cookie) (not ok)
Or is it enough to call the following statement?
encoder.encodeForHTMLAttribute(referrer);
Any help appreciated.

Here's my final solution if anyone is interested. First I canonicalize and then URL decode the string. If a CR or LF exists (\n \r) I just cut of the rest of that potential 'attack' string starting with \n or \r.
String sanitize(String url) throws EncodingException{
Encoder encoder = new DefaultEncoder(new ArrayList<String>());
//first canonicalize
String clean = encoder.canonicalize(url).trim();
//then url decode
clean = encoder.decodeFromURL(clean);
//detect and remove any existent \r\n == %0D%0A == CRLF to prevent HTTP Response Splitting
int idxR = clean.indexOf('\r');
int idxN = clean.indexOf('\n');
if(idxN >= 0 || idxR>=0){
if(idxN<idxR){
//just cut off the part after the LF
clean = clean.substring(0,idxN);
}
else{
//just cut off the part after the CR
clean = clean.substring(0,idxR);
}
}
//re-encode again
return encoder.encodeForURL(clean);
}
Theoretically i could have later verified the value against 'HTTPParameterValue' regex which is defined in the ESAPI.properties however it didn't like colon in the http:// and I didn't investigated further.
And one more remark after testing it: Most modern browser nowadays (Firefox > 3.6, Chrome, IE10 etc.) detect this kind of vulnerability and do not execute the code...

I think you have the right idea, but are using an inappropriate encoder. The Referer [sic] header value is really a URL, not an HTML attribute, so you really want to use:
encoder.encodeForURL(referrer);
-kevin

I would suggest white-listing approach wherein you check the referrer string only for permissible characters. Regex would be a good option.
EDIT:
The class org.owasp.esapi.reference.DefaultEncoder being used by you is not really encoding anything. Look at the source code of the method encodeForHTMLAttribute(referrer) here at grepcode. A typical URL encoding (encoding carriage return and line feed) too wont help.
So the way forward would be device some validation logic which checks for valid set of characters. Here is another insightful article.

The accepted answer will not work if in case there is "\n\r" in the string.
Example:
If I have string: "This is str\n\rstr", it returns "This is str\nstr"
Rectified version of above accepted answer is:
String sanitizeCarriageReturns(String value) {
int idxR = value.indexOf('\r');
int idxN = value.indexOf('\n');
if (idxN >= 0 || idxR >= 0) {
if ((idxN > idxR && idxR<0) || (idxR > idxN && idxR>=0)) {
value = value.substring(0, idxN);
} else if (idxN < idxR){
value = value.substring(0, idxR);
}
}
return value;
}

insert a domain name into servlet authentication token

I am using Waffle for an SSO solution in my web-app.
Everything works fine but I would like to modify some functionality slightly:
Currently, if a user is not connected to the domain the SSO fails and opens a little authorization dialog:
The windows authorization requires the user name formatted like Domain\Username but most of my users will not know to add the domain in front of their username. So I would like to provide a default domain name if one is not specified.
I found a waffle function that I can override which will give me access to the decoded authentication token, I added a println to the waffle function and it shows the username in plain text (either with or without the domain depending on what is typed in the dialog):
public IWindowsSecurityContext acceptSecurityToken(String connectionId, byte[] token, String securityPackage) {
// I can see the passed username in the logs with this
System.out.println(new String(token));
// I don't understand any of the JNA stuff below this comment:
IWindowsCredentialsHandle serverCredential = new WindowsCredentialsHandleImpl(
null, Sspi.SECPKG_CRED_INBOUND, securityPackage);
serverCredential.initialize();
SecBufferDesc pbServerToken = new SecBufferDesc(Sspi.SECBUFFER_TOKEN, Sspi.MAX_TOKEN_SIZE);
SecBufferDesc pbClientToken = new SecBufferDesc(Sspi.SECBUFFER_TOKEN, token);
NativeLongByReference pfClientContextAttr = new NativeLongByReference();
CtxtHandle continueContext = _continueContexts.get(connectionId);
CtxtHandle phNewServerContext = new CtxtHandle();
int rc = Secur32.INSTANCE.AcceptSecurityContext(serverCredential.getHandle(),
continueContext, pbClientToken, new NativeLong(Sspi.ISC_REQ_CONNECTION),
new NativeLong(Sspi.SECURITY_NATIVE_DREP), phNewServerContext,
pbServerToken, pfClientContextAttr, null);
WindowsSecurityContextImpl sc = new WindowsSecurityContextImpl();
sc.setCredentialsHandle(serverCredential.getHandle());
sc.setSecurityPackage(securityPackage);
sc.setSecurityContext(phNewServerContext);
switch (rc)
{
case W32Errors.SEC_E_OK:
// the security context received from the client was accepted
_continueContexts.remove(connectionId);
// if an output token was generated by the function, it must be sent to the client process
if (pbServerToken != null
&& pbServerToken.pBuffers != null
&& pbServerToken.cBuffers.intValue() == 1
&& pbServerToken.pBuffers[0].cbBuffer.intValue() > 0) {
sc.setToken(pbServerToken.getBytes());
}
sc.setContinue(false);
break;
case W32Errors.SEC_I_CONTINUE_NEEDED:
// the server must send the output token to the client and wait for a returned token
_continueContexts.put(connectionId, phNewServerContext);
sc.setToken(pbServerToken.getBytes());
sc.setContinue(true);
break;
default:
sc.dispose();
WindowsSecurityContextImpl.dispose(continueContext);
_continueContexts.remove(connectionId);
throw new Win32Exception(rc);
}
return sc;
}
That whole function is from the Waffle API I only added the println at the beginning.
The passed username prints in plain text inside this token between a bunch of random byte chars (ÉsR=ÍtÍö?æ¸+Û-).
I am admittedly in very far over my head with JNA and java in general but I thought that because I can see the username here there must be a way to prepend the domain name to the username part of this token? I could be wrong.
My other idea was to add the domain to the pbClientToken that is created from the raw byte[] token this method is passed.
The pbClientToken is a JNA Structure object derivative. It has the Stucture method writeField which looked promising but I can't seem to figure out what field I should write. The Structure.getFields method doesn't seem to be available from pbClientToken.
I was hoping that this was a simple problem for someone more familiar with byte[] processing or JNA.

You cannot do this. What happens behind this dialog is a call to LogonUser on the user's machine, which gives you a ticket, which is then sent to the server. Unfortunately the server is not in the same domain, so even if you manage to extract the username it's completely useless.