We Keep Coding

Decode multiple times encoded String

I have written Java code to decode a string encoded with "UTF-8". That String was encoded three times. I am using this code in the ETL. so, I can use an ETL step three times in a row, but it will be a little inefficient. I researched over the internet but didn't find anything promising. Is there any way in Java to decode the String encoded multiple times?
Here's my input string "uri":
file:///C:/Users/nikhil.karkare/dev/pentaho/data/ba-repo-content-original/public/Development+Activity/Defects+Unresolved+%252528by+Non-Developer%252529.xanalyzer
Here's my code which is decoding this string:
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.io.*;
String decodedValue;
public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException {
// First, get a row from the default input hop
//
Object[] r = getRow();
// If the row object is null, we are done processing.
//
if (r == null) {
setOutputDone();
return false;
}
// It is always safest to call createOutputRow() to ensure that your output row's Object[] is large
// enough to handle any new fields you are creating in this step.
//
Object[] outputRow = createOutputRow(r, data.outputRowMeta.size());
String newFileName = get(Fields.In, "uri").getString(r);
try{
decodedValue = URLDecoder.decode(newFileName, "UTF-8");
}
catch (UnsupportedEncodingException e) {
throw new AssertionError("UTF-8 is unknown");
}
// Set the value in the output field
//
get(Fields.Out, "decodedValue").setValue(outputRow, decodedValue);
// putRow will send the row on to the default output hop.
//
putRow(data.outputRowMeta, outputRow);
return true;}
Output of this code is following:
file:///C:/Users/nikhil.karkare/dev/pentaho/data/ba-repo-content-original/public/Development Activity/Defects Unresolved %2528by Non-Developer%2529.xanalyzer
When I run this code in the ETL three times, I get the output I want, which is this:
file:///C:/Users/nikhil.karkare/dev/pentaho/data/ba-repo-content-original/public/Development Activity/Defects Unresolved (by Non-Developer).xanalyzer

URL encoding replaces %, ( and ) with resp. %25.%28 and %29.
String s = "file:///C:/Users/nikhil.karkare/dev/pentaho/data/"
+ "ba-repo-content-original/public/Development+Activity/"
+ "Defects+Unresolved+%252528by+Non-Developer%252529.xanalyzer";
// %253528 ... %252529
s = URLDecoder.decode(s, "UTF-8");
// %2528 ... %2529
s = URLDecoder.decode(s, "UTF-8");
// %28 .. %29
s = URLDecoder.decode(s, "UTF-8");
// ( ... )

Just a for loop did the job:
String newFileName = get(Fields.In, "uri").getString(r);
decodedValue = newFileName;
for (int i=0; i<=3; i++){
try{
decodedValue = URLDecoder.decode(decodedValue, "UTF-8");
}
catch (UnsupportedEncodingException e) {
throw new AssertionError("UTF-8 is unknown");
}
}

PHP's hash() in Java

I am currently trying to get Java to generate the same hash for a string as PHP's hash algorithm does.
I have come close enough:
hash('sha512', 'password');
outputs:
b109f3bbbc244eb82441917ed06d618b9008dd09b3befd1b5e07394c706a8bb980b1d7785e5976ec049b46df5f1326af5a2ea6d103fd07c95385ffab0cacbc86
Java code:
public static void main(String[] args) {
hash("password");
}
private static String hash(String salted) {
byte[] digest;
try {
MessageDigest mda = MessageDigest.getInstance("SHA-512");
digest = mda.digest(salted.getBytes("UTF-8"));
} catch (Exception e) {
digest = new byte[]{};
}
String str = "";
for (byte aDigest : digest) {
str += String.format("%02x", 0xFF & aDigest);
}
return str;
}
This outputs the same.
My problem is when I use the third argument within PHP's hash function. On PHP's site it's described as following:
raw_output
When set to TRUE, outputs raw binary data. FALSE outputs lowercase hexits.
I am not quite sure how to implement this extra parameter. I think mainly my question would be, how do I convert a String object into a binary String object? Currently, running it with PHP generates the following: http://sandbox.onlinephpfunctions.com/code/a1bd9b399b3ac0c4db611fe748998f18738d19e3

This should reproduce the outcome from your link:
String strBinary = null;
try {
strBinary = new String(digest, "UTF-8");
} catch (UnsupportedEncodingException e) {
}
and you'll need these imports at the top of your file:
import java.nio.charset.Charset;
import java.io.UnsupportedEncodingException;
I hope I understood your issue correctly.

How to handle UNICODE URL?

If I want to pase the following URL's in Java:
... what handle should I have with the String.
So far I have been unable to handle that String's, all I've got are ???? chars.
Thanks.
Modified in 2012.09.09:
package pruebas;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.util.Vector;
public class Prueba03
{
public static void main(String argumentos[])
{
Vector<String> listaURLs = new Vector<String>();
listaURLs.add("http://президент.рф/");
listaURLs.add("http://www.中国政府.政务.cn");
listaURLs.add("http://www.原來我不帥.cn/");
listaURLs.add("http://وزارة-الأتصالات.مصر/");
URL currentURL;
URLConnection currentConnection;
int currentSize;
for(int i=0; i<listaURLs.size(); i++)
{
try
{
System.out.println(URLDecoder.decode(listaURLs.get(i), URLEncoder.encode(listaURLs.get(i), "UTF-8")));
} // End of the try.
catch(UnsupportedEncodingException uee)
{
uee.printStackTrace();
} // End of the catch.
catch(Exception e)
{
e.printStackTrace();
} // End of the catch.
try
{
currentURL = new URL(listaURLs.get(i));
System.out.println("currentURL" + " = " + currentURL);
currentConnection = currentURL.openConnection();
System.out.println("currentConnection" + " = " + currentConnection);
currentSize = currentConnection.getContentLength();
System.out.println("currentSize" + " = " + currentSize);
} // End of the try.
catch(Exception e)
{
e.printStackTrace();
} // End of the catch.
} // End of the for.
} // End of the main method.
} // End of the Prueba02 class.

For domain name, you should convert unicode host name using Punycode.
Punycode is a way to convert unicode string to ascii string.
The following link shows a JAVA method to convert Unicode Domain Name to International Domain Name.
https://docs.oracle.com/javase/6/docs/api/java/net/IDN.html#toASCII(java.lang.String)
URL u = new URL(url);
String host = u.getHost();
String[] labels = host.split("\\.");
for (int i = 0; i < labels.length; i++) {
labels[i] = java.net.IDN.toUnicode(labels[i]);
}
host = StringUtils.join(labels, ".");
System.out.println(host);
Also, you can test some unicode URL using online punycode converter.
https://www.punycoder.com/
For example, "http://www.中国政府.政务.cn" is converted into "http://www.xn--fiqs8sirgfmh.xn--zfr164b.cn/".

Based on #hyunjong answer its not working to use toUnicode, use toASCII instead. And if you prefer kotlin, you can use this code
val a = "http://www.中国政府.政务.cn"
val u = URL(a);
val labels = u.host.split("\\.");
val result = labels.joinToString(separator = ".") { s ->
java.net.IDN.toASCII(s)
}
print(result) //www.xn--fiqs8sirgfmh.xn--zfr164b.cn

You can try the follow code:
import java.net.URLDecoder;
import java.net.URLEncoder;
public class Test7 {
public static void main(String[] args) throws Exception {
String str = "http://www.中国政府.政务.cn";
System.out.println(URLDecoder.decode(str, URLEncoder.encode(str,
"UTF-8")));
}
}

Not sure what do you mean by "parse" - what do you intend to do with these parts?
Arabic and Russian, as far as I know are supported by UTF-8.
Not sure what is your source of data (some sort of Stream perhaps?) but String has a CTOR that accepts the desired encoding.
You should be able to obtain a string NOT containing ??? when it comes to Arabic and Russian if you use this CTOR (with the "UTF-8" argument)

You may try use the following:
String pageUrl = "http://www.中国政府.政务.cn";
try
{
URL url = new URL(pageUrl);
System.out.println(url.toURI().toASCIIString());
}
catch (MalformedURLException e1)
{
// TODO Auto-generated catch block
e1.printStackTrace();
}
catch (URISyntaxException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
The result is as expected:
http://www.%E4%B8%AD%E5%9B%BD%E6%94%BF%E5%BA%9C.%E6%94%BF%E5%8A%A1.cn
But converting to URI has its own disadvantage, you should replace manually the special characters like '|', '"', '#' to its URL encoding.

How to parse UTF-8 representation to String in Java?

Given the following code:
String tmp = new String("\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a");
String result = convertToEffectiveString(tmp); // result contain now "hello\n"
Does the JDK already provide some classes for doing this ?
Is there a libray that does this ? (preferably under maven)
I have tried with ByteArrayOutputStream with no success.

This works, but only with ASCII. If you use unicode characters outside of the ASCCI range, then you will have problems (as each character is being stuffed into a byte, instead of a full word that is allowed by UTF-8). You can do the typecast below because you know that the UTF-8 will not overflow one byte if you guaranteed that the input is basically ASCII (as you mention in your comments).
package sample;
import java.io.UnsupportedEncodingException;
public class UnicodeSample {
public static final int HEXADECIMAL = 16;
public static void main(String[] args) {
try {
String str = "\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a";
String arr[] = str.replaceAll("\\\\u"," ").trim().split(" ");
byte[] utf8 = new byte[arr.length];
int index=0;
for (String ch : arr) {
utf8[index++] = (byte)Integer.parseInt(ch,HEXADECIMAL);
}
String newStr = new String(utf8, "UTF-8");
System.out.println(newStr);
}
catch (UnsupportedEncodingException e) {
// handle the UTF-8 conversion exception
}
}
}
Here is another solution that fixes the issue of only working with ASCII characters. This will work with any unicode characters in the UTF-8 range instead of ASCII only in the first 8-bits of the range. Thanks to deceze for the questions. You made me think more about the problem and solution.
package sample;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
public class UnicodeSample {
public static final int HEXADECIMAL = 16;
public static void main(String[] args) {
try {
String str = "\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a\\u3fff\\uf34c";
ArrayList<Byte> arrList = new ArrayList<Byte>();
String codes[] = str.replaceAll("\\\\u"," ").trim().split(" ");
for (String c : codes) {
int code = Integer.parseInt(c,HEXADECIMAL);
byte[] bytes = intToByteArray(code);
for (byte b : bytes) {
if (b != 0) arrList.add(b);
}
}
byte[] utf8 = new byte[arrList.size()];
for (int i=0; i<arrList.size(); i++) utf8[i] = arrList.get(i);
str = new String(utf8, "UTF-8");
System.out.println(str);
}
catch (UnsupportedEncodingException e) {
// handle the exception when
}
}
// Takes a 4 byte integer and and extracts each byte
public static final byte[] intToByteArray(int value) {
return new byte[] {
(byte) (value >>> 24),
(byte) (value >>> 16),
(byte) (value >>> 8),
(byte) (value)
};
}
}

Firstly, are you just trying to parse a string literal, or is tmp going to be some user-entered data?
If this is going to be a string literal (i.e. hard-coded string), it can be encoded using Unicode escapes. In your case, this just means using single backslashes instead of double backslashes:
String result = "\u0068\u0065\u006c\u006c\u006f\u000a";
If, however, you need to use Java's string parsing rules to parse user input, a good starting point might be Apache Commons Lang's StringEscapeUtils.unescapeJava() method.

I'm sure there must be a better way, but using just the JDK:
public static String handleEscapes(final String s)
{
final java.util.Properties props = new java.util.Properties();
props.setProperty("foo", s);
final java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
try
{
props.store(baos, null);
final String tmp = baos.toString().replace("\\\\", "\\");
props.load(new java.io.StringReader(tmp));
}
catch(final java.io.IOException ioe) // shouldn't happen
{ throw new RuntimeException(ioe); }
return props.getProperty("foo");
}
uses java.util.Properties.load(java.io.Reader) to process the backslash-escapes (after first using java.util.Properties.store(java.io.OutputStream, java.lang.String) to backslash-escape anything that would cause problems in a properties-file, and then using replace("\\\\", "\\") to reverse the backslash-escaping of the original backslashes).
(Disclaimer: even though I tested all the cases I could think of, there are still probably some that I didn't think of.)

Decode base64Url in Java

https://web.archive.org/web/20110422225659/https://en.wikipedia.org/wiki/Base64#URL_applications
talks about base64Url - Decode
a modified Base64 for URL variant exists, where no padding '=' will be used, and the '+' and '/' characters of standard Base64 are respectively replaced by '-' and '_'
I created the following function:
public static String base64UrlDecode(String input) {
String result = null;
BASE64Decoder decoder = new BASE64Decoder();
try {
result = decoder.decodeBuffer(input.replace('-','+').replace('/','_')).toString();
}
catch (IOException e) {
System.out.println(e.getMessage());
}
return result;
}
it returns a very small set of characters that don't even resemble to the expected results.
any ideas?

Java8+
import java.util.Base64;
return Base64.getUrlEncoder().encodeToString(bytes);

Base64 encoding is part of the JDK since Java 8. URL safe encoding is also supported with java.util.Base64.getUrlEncoder(), and the "=" padding can be skipped by additionally using the java.util.Base64.Encoder.withoutPadding() method:
import java.nio.charset.StandardCharsets;
import java.util.Base64;
public String encode(String raw) {
return Base64.getUrlEncoder()
.withoutPadding()
.encodeToString(raw.getBytes(StandardCharsets.UTF_8));
}

With the usage of Base64 from Apache Commons, who can be configured to URL safe, I created the following function:
import org.apache.commons.codec.binary.Base64;
public static String base64UrlDecode(String input) {
String result = null;
Base64 decoder = new Base64(true);
byte[] decodedBytes = decoder.decode(input);
result = new String(decodedBytes);
return result;
}
The constructor Base64(true) makes the decoding URL-safe.

In the Android SDK, there's a dedicated flag in the Base64 class: Base64.URL_SAFE, use it like so to decode to a String:
import android.util.Base64;
byte[] byteData = Base64.decode(body, Base64.URL_SAFE);
str = new String(byteData, "UTF-8");

Guava now has Base64 decoding built in.
https://google.github.io/guava/releases/17.0/api/docs/com/google/common/io/BaseEncoding.html

public static byte[] encodeUrlSafe(byte[] data) {
byte[] encode = Base64.encode(data);
for (int i = 0; i < encode.length; i++) {
if (encode[i] == '+') {
encode[i] = '-';
} else if (encode[i] == '/') {
encode[i] = '_';
}
}
return encode;
}
public static byte[] decodeUrlSafe(byte[] data) {
byte[] encode = Arrays.copyOf(data, data.length);
for (int i = 0; i < encode.length; i++) {
if (encode[i] == '-') {
encode[i] = '+';
} else if (encode[i] == '_') {
encode[i] = '/';
}
}
return Base64.decode(encode);
}

Right off the bat, it looks like your replace() is backwards; that method replaces the occurrences of the first character with the second, not the other way around.

#ufk's answer works, but you don't actually need to set the urlSafe flag when you're just decoding.
urlSafe is only applied to encode operations. Decoding seamlessly
handles both modes.
Also, there are some static helpers to make it shorter and more explicit:
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
public static String base64UrlDecode(String input) {
StringUtils.newStringUtf8(Base64.decodeBase64(input));
}
Docs
newStringUtf8()
decodeBase64()

This class can help:
import android.util.Base64;
public class Encryptor {
public static String encode(String input) {
return Base64.encodeToString(input.getBytes(), Base64.URL_SAFE);
}
public static String decode(String encoded) {
return new String(Base64.decode(encoded.getBytes(), Base64.URL_SAFE));
}
}

I know the answer is already there, but still, if someone wants...
import java.util.Base64; public
class Base64BasicEncryptionExample {
publicstaticvoid main(String[] args) {
// Getting encoder
Base64.Encoder encoder = Base64.getUrlEncoder();
// Encoding URL
String eStr = encoder.encodeToString
("http://www.javatpoint.com/javatutorial/".getBytes());
System.out.println("Encoded URL: "+eStr);
// Getting decoder
Base64.Decoder decoder = Base64.getUrlDecoder();
// Decoding URl
String dStr = new String(decoder.decode(eStr));
System.out.println("Decoded URL: "+dStr);
}
}
Took help from: https://www.javatpoint.com/java-base64-encode-decode

In Java try the method Base64.encodeBase64URLSafeString() from Commons Codec library for encoding.