We Keep Coding

Java encode image URL correctly [duplicate]

Say I have a URL
http://example.com/query?q=
and I have a query entered by the user such as:
random word £500 bank $
I want the result to be a properly encoded URL:
http://example.com/query?q=random%20word%20%A3500%20bank%20%24
What's the best way to achieve this? I tried URLEncoder and creating URI/URL objects but none of them come out quite right.

URLEncoder is the way to go. You only need to keep in mind to encode only the individual query string parameter name and/or value, not the entire URL, for sure not the query string parameter separator character & nor the parameter name-value separator character =.
String q = "random word £500 bank $";
String url = "https://example.com?q=" + URLEncoder.encode(q, StandardCharsets.UTF_8);
When you're still not on Java 10 or newer, then use StandardCharsets.UTF_8.toString() as charset argument, or when you're still not on Java 7 or newer, then use "UTF-8".
Note that spaces in query parameters are represented by +, not %20, which is legitimately valid. The %20 is usually to be used to represent spaces in URI itself (the part before the URI-query string separator character ?), not in query string (the part after ?).
Also note that there are three encode() methods. One without Charset as second argument and another with String as second argument which throws a checked exception. The one without Charset argument is deprecated. Never use it and always specify the Charset argument. The javadoc even explicitly recommends to use the UTF-8 encoding, as mandated by RFC3986 and W3C.
All other characters are unsafe and are first converted into one or more bytes using some encoding scheme. Then each byte is represented by the 3-character string "%xy", where xy is the two-digit hexadecimal representation of the byte. The recommended encoding scheme to use is UTF-8. However, for compatibility reasons, if an encoding is not specified, then the default encoding of the platform is used.
See also:
What every web developer must know about URL encoding

I would not use URLEncoder. Besides being incorrectly named (URLEncoder has nothing to do with URLs), inefficient (it uses a StringBuffer instead of Builder and does a couple of other things that are slow) Its also way too easy to screw it up.
Instead I would use URIBuilder or Spring's org.springframework.web.util.UriUtils.encodeQuery or Commons Apache HttpClient.
The reason being you have to escape the query parameters name (ie BalusC's answer q) differently than the parameter value.
The only downside to the above (that I found out painfully) is that URL's are not a true subset of URI's.
Sample code:
import org.apache.http.client.utils.URIBuilder;
URIBuilder ub = new URIBuilder("http://example.com/query");
ub.addParameter("q", "random word £500 bank \$");
String url = ub.toString();
// Result: http://example.com/query?q=random+word+%C2%A3500+bank+%24

You need to first create a URI like:
String urlStr = "http://www.example.com/CEREC® Materials & Accessories/IPS Empress® CAD.pdf"
URL url = new URL(urlStr);
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
Then convert that URI to an ASCII string:
urlStr = uri.toASCIIString();
Now your URL string is completely encoded. First we did simple URL encoding and then we converted it to an ASCII string to make sure no character outside US-ASCII remained in the string. This is exactly how browsers do it.

Guava 15 has now added a set of straightforward URL escapers.

The code
URL url = new URL("http://example.com/query?q=random word £500 bank $");
URI uri = new URI(url.getProtocol(), url.getUserInfo(), IDN.toASCII(url.getHost()), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
String correctEncodedURL = uri.toASCIIString();
System.out.println(correctEncodedURL);
Prints
http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$
What is happening here?
1. Split URL into structural parts. Use java.net.URL for it.
2. Encode each structural part properly!
3. Use IDN.toASCII(putDomainNameHere) to Punycode encode the hostname!
4. Use java.net.URI.toASCIIString() to percent-encode, NFC encoded Unicode - (better would be NFKC!). For more information, see: How to encode properly this URL
In some cases it is advisable to check if the URL is already encoded. Also replace '+' encoded spaces with '%20' encoded spaces.
Here are some examples that will also work properly
{
"in" : "http://نامه‌ای.com/",
"out" : "http://xn--mgba3gch31f.com/"
},{
"in" : "http://www.example.com/‥/foo",
"out" : "http://www.example.com/%E2%80%A5/foo"
},{
"in" : "http://search.barnesandnoble.com/booksearch/first book.pdf",
"out" : "http://search.barnesandnoble.com/booksearch/first%20book.pdf"
}, {
"in" : "http://example.com/query?q=random word £500 bank $",
"out" : "http://example.com/query?q=random%20word%20%C2%A3500%20bank%20$"
}
The solution passes around 100 of the test cases provided by Web Platform Tests.

Using Spring's UriComponentsBuilder:
UriComponentsBuilder
.fromUriString(url)
.build()
.encode()
.toUri()

The Apache HttpComponents library provides a neat option for building and encoding query parameters.
With HttpComponents 4.x use:
URLEncodedUtils
For HttpClient 3.x use:
EncodingUtil

Here's a method you can use in your code to convert a URL string and map of parameters to a valid encoded URL string containing the query parameters.
String addQueryStringToUrlString(String url, final Map<Object, Object> parameters) throws UnsupportedEncodingException {
if (parameters == null) {
return url;
}
for (Map.Entry<Object, Object> parameter : parameters.entrySet()) {
final String encodedKey = URLEncoder.encode(parameter.getKey().toString(), "UTF-8");
final String encodedValue = URLEncoder.encode(parameter.getValue().toString(), "UTF-8");
if (!url.contains("?")) {
url += "?" + encodedKey + "=" + encodedValue;
} else {
url += "&" + encodedKey + "=" + encodedValue;
}
}
return url;
}

In Android, I would use this code:
Uri myUI = Uri.parse("http://example.com/query").buildUpon().appendQueryParameter("q", "random word A3500 bank 24").build();
Where Uri is a android.net.Uri

In my case I just needed to pass the whole URL and encode only the value of each parameters.
I didn't find common code to do that, so (!!) so I created this small method to do the job:
public static String encodeUrl(String url) throws Exception {
if (url == null || !url.contains("?")) {
return url;
}
List<String> list = new ArrayList<>();
String rootUrl = url.split("\\?")[0] + "?";
String paramsUrl = url.replace(rootUrl, "");
List<String> paramsUrlList = Arrays.asList(paramsUrl.split("&"));
for (String param : paramsUrlList) {
if (param.contains("=")) {
String key = param.split("=")[0];
String value = param.replace(key + "=", "");
list.add(key + "=" + URLEncoder.encode(value, "UTF-8"));
}
else {
list.add(param);
}
}
return rootUrl + StringUtils.join(list, "&");
}
public static String decodeUrl(String url) throws Exception {
return URLDecoder.decode(url, "UTF-8");
}
It uses Apache Commons' org.apache.commons.lang3.StringUtils.

Use this：
URLEncoder.encode(query, StandardCharsets.UTF_8.displayName());
or this:
URLEncoder.encode(query, "UTF-8");
You can use the following code.
String encodedUrl1 = UriUtils.encodeQuery(query, "UTF-8"); // No change
String encodedUrl2 = URLEncoder.encode(query, "UTF-8"); // Changed
String encodedUrl3 = URLEncoder.encode(query, StandardCharsets.UTF_8.displayName()); // Changed
System.out.println("url1 " + encodedUrl1 + "\n" + "url2=" + encodedUrl2 + "\n" + "url3=" + encodedUrl3);

Basic Auth Authorization header and base 64 encoding

I have a vendor that I wish to exchange data with. They want me to take the username and password that they gave me and use it on an Authorization header for a get request.
And now my dirty little secret. I've never created an Authorization header before.
So I do a bunch of research and figure out the following code.
public static final String AUTH_SEPARATOR = ":";
private static final String AUTH_TYPE = "Basic ";
public static final String HEADER_AUTHORIZATION = "Authorization";
public static void addAuthHeader(Map<String, String> headers, String user, String password) {
String secretKey = user
+ AUTH_SEPARATOR
+ password;
byte[] tokenBytes = secretKey.getBytes();
String token64 = Base64.getEncoder().encodeToString(tokenBytes);
String auth = AUTH_TYPE + token64;
headers.put(HEADER_AUTHORIZATION, auth);
}
I run it and I get no response back. Hmmm. So I go to open a support request with them and I want to create an example, so I open postman and use the APIs they gave me for postman. First, I run the one for the API I'm replicating and it works. Hmm.
So then I modify that API and use my username and password instead of the one included in the example and it works fine. Crikey!
So I bang around a bit and notice that the Base64 string in the auth created by postman is slightly different at the end than the one I created.
So, back to the research and all the code I find looks a lot like mine, although I had to update it some because of version differences. The string is still different and now I'm asking for help. Surely someone has solved this problem.
String from postman "Basic THVKZ...FvTg=="
String from code above "Basic THVKZ...FvTiA="
How did I do something wrong and end up with only a three byte difference?

Tg== is base64 for N.
TiA= is base64 for N (as in, N, then a space).
Sooo, it sounds like postman is sticking a space up there and you aren't. Hopefully, if you add a space at the very end of whatever you are base64 encoding, you should get the exact same string as postman is giving you, and, hopefully, it'll all just work out at that point :)

Postman using UTF-8 for basic auth encoding, check from https://github.com/postmanlabs/postman-app-support/issues/4070
change your code like this
secretKey.getBytes(StandardCharsets.UTF_8);
https://www.base64encode.org/ for test

The problem is caused by padding. I still don't understand exactly why, but the string I'm encoding is 49 bytes long, which is not evenly divisible by 3, which means that padding comes into play.
When I changed my code to the following:
String token64 = Base64.getEncoder().withoutPadding().encodeToString(tokenBytes);
and then ran it, I got the same string minus the two == at the end that base64 uses as a pad character. Sending that to the server got the answer I was looking for.
Why do they call it software when it's so damned hard?

Esteemed developer, are they not running on OAuth2? If so, kindly make use of the code blocks below and kill it
public HttpHeaders getHeaders() {
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
headers.add("Authorization", "Bearer " + token you are either generating or getting from Eureka discovery service);
return headers;
}
AuthResponseObjectType getAuthorization() {
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Collections.singletonList(MediaType.APPLICATION_JSON));
String auth = basicAuthUsername + ":" + basicAuthPassword;
byte[] encodedAuth = org.apache.commons.net.util.Base64.encodeBase64(auth.getBytes(StandardCharsets.US_ASCII));
headers.add("Authorization", "Basic " + new String(encodedAuth));
MultiValueMap<String, String> map = new LinkedMultiValueMap<>();
map.add("grant_type", grantType);
map.add("username", username);
map.add("password", password);
map.add("scope", scope);
HttpEntity<MultiValueMap<String, String>> request = new HttpEntity<>(map, headers);
return restTemplate.exchange(authUrl, HttpMethod.POST, request, AuthResponseObjectType.class).getBody();
}
Let me know if you are facing any challenges then we can do it together

Correctly compare code_verifier with code_challenge in Java

I'm using passport-oauth2 (passportjs.org and https://github.com/jaredhanson/passport-oauth2/blob/master/lib/strategy.js) for OAuth2+PKCE integration in a nodejs application.
The backend it's authenticating against is written in Java.
The problem is that I can't seem to decode->hash the code_verifier to correctly match the code_challenge that comes from passport-oauth2.
I know that the Base64 encoding that comes from passport has been generated to be URL safe (no padding, no wrapping, replacements for + or /), so I'm using a Url Decoder:
Base64.getUrlDecoder().decode(...)
Then I'm using commons DigestUtils to generate a SHA256 of the decoded verifier and comparing it with the challenge. So the whole thing looks something like this:
java.util.Base64.Decoder decoder = java.util.Base64.getUrlDecoder();
String codeChallenge = // get the code challenge from my cache
byte[] decodedCodeChallenge = decoder.decode(codeChallenge);
byte[] decodedCodeVerifier = decoder.decode(codeVerifier);
if (!Arrays.equals(sha256(decodedCodeVerifier), decodedCodeChallenge)) {
return Response.status(400).entity(ERROR_INVALID_CHALLENGE_VERIFIER).build();
}
Example:
This code verifier: 5CFCAiZC0g0OA-jmBmmjTBZiyPCQsnq_2q5k9fD-aAY
should match this code challenge: Fw7s3XHRVb2m1nT7s646UrYiYLMJ54as0ZIU_injyqw once both have been Base64-url-decoded and the verifier has been SHA256 hashed, but it doesn't.
What am I doing wrong?

Just 5 minutes later I figured it out.
In passport-oauth2, the code verifier is Base64-url-encoded(random bytes):
verifier = base64url(crypto.pseudoRandomBytes(32))
See: https://github.com/jaredhanson/passport-oauth2/blob/master/lib/strategy.js#L236
The challenge is then Base64-url-encoded(sha256(verifier)), which expands to Base64-url-encoded(sha256(Base64-url-encoded(random bytes))):
challenge = base64url(crypto.createHash('sha256').update(verifier).digest());
See: https://github.com/jaredhanson/passport-oauth2/blob/master/lib/strategy.js#L242
So to do the verification, I don't need to decode anything. It was sha256-d in it's encoded state.
This worked in the end:
java.util.Base64.Encoder encoder = java.util.Base64.getUrlEncoder();
String codeChallenge = // get code challenge from my cache;
String encodedVerifier = new String(encoder.encode(sha256(codeVerifier))).split("=")[0]; // Remember to remove padding
if (!encodedVerifier.equals(codeChallenge)) {
return Response.status(400).entity(ERROR_INVALID_CHALLENGE_VERIFIER).build();
}

Creating a URL Using Java - What's the Best Practice?

I have an application that is calling a rest service. I need to pass it a URL and right now I'm creating the URL by concatenating a string.
I'm doing it this way:
String urlBase = "http:/api/controller/";
String apiMethod = "buy";
String url = urlBase + apiMethod;
The above is fake obviously, but the point is I'm using simple string concats.
Is this the best practice? I'm relatively new to Java. Should I be building a URL object instead?
Thanks

if you have a base path which needs some additional string to be added to it you have 2 options:
First is, using String.format():
String baseUrl = "http:/api/controller/%s"; // note the %s at the end
String apiMethod = "buy";
String url = String.format(baseUrl, apiMethod);
Or using String.replace():
String baseUrl = "http:/api/controller/{apiMethod}";
String apiMethod = "buy";
String url = baseUrl.replace("\\{apiMethod}", apiMethod);
The nice thing about both answers is, that the string that needs to be inserted, doesn't have to be at the end.

If you are using jersey-client. The following would be the best practice to access the subresources without making the code ugly
Resource: /someApp
Sub-Resource: /someApp/getData
Client client = ClientBuilder.newClient();
WebTarget webTarget = client.target("https://localhost:7777/someApp/").path("getData");
Response response = webTarget.request().header("key", "value").get();

If you are using plain Java, it's better to use dedicated class for URL building which throws exception if provided data are invalid in semantic way.
It has various constructors, you can read about it here.
Example
URL url = new URL(
"http",
"stackoverflow.com",
"/questions/50989746/creating-a-url-using-java-whats-the-best-practive"
);
System.out.println(url);

Properly encoding percent sign (%) in Apachi Http Client form post using UrlEncode

This seems like an extremely easy problem but alas I cannot figure it out nor find a solution anywhere else. I'm concatenating a string that has a % within and for some reason it adds the number 25 after the %. Anyone know of a solution to this easy problem?
String buttonCheck = "%26" + DATABASE.getValue("buttonCheck") + "%26";
Comes out to
"%2526value%2526"
EDIT: It has become apparent that the issue is actually within URL encoding and I will add more relevant data to the issue.
I am developing an Android App that parses HTML from a site and allows the user to interact with it through the Android UI. I am having an issue with encoding % into a parameter for a form.
public class CLASS extends Activity
{
DefaultHttpClient client = new DefaultHttpClient;
String url = "http://www.url.com"
HttpRequestBase method = new HttpPost(url);
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add("buttoncheck", "%26" + DATABASE.getValue("buttonCheck") + "%26");
//DATABASE is simply a class that handles a HashMap
HttpPost methodPost = (HttpPost) method;
methodPost.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8));
execute(method);
}
Rather than sending the form value of
buttoncheck=%26value%26
I get
buttoncheck=%2526value%2526

It won't come out as "%2526value%2526" in buttonCheck. I strongly suspect you're looking at a value later on - for example, after URI encoding. Work out what's doing that encoding, and what you actually want to be encoded.
Just to be clear, the 25 is completely distinct from the 26. You'll see the same thing if you get rid of the 26 completely, with
String buttonCheck = "%" + DATABASE.getValue("buttonCheck") + "%";
At that point I suspect you'll get
%25value%25
Basically something is just encoding the % as %25. For example, this would do it:
import java.net.*;
public class Test {
public static void main(String[] args) throws Exception {
String input = "%value%";
String encoded = URLEncoder.encode(input, "utf-8");
System.out.println(encoded); // Prints %25value%25
}
}

apparently the string went through "percent-encoding" when it becomes part of a URI.
If that's the case, you should not do percent-encoding so early. instead
String buttonCheck = "&" + DATABASE.getValue("buttonCheck") + "&";
which will end up in the URI as
"%26value%26"

Based on the answer to this question, I would guess that the literal % is being encoded to %25 in your string. Which explains the added 25. Without seeing relevant code, we won't be able to know why it gets there in the first place.