We Keep Coding

Decode and replace hex values in a string in Java

I have the a string in Java which contains hex values beneath normal characters. It looks something like this:
String s = "Hello\xF6\xE4\xFC\xD6\xC4\xDC\xDF"
What I want is to convert the hex values to the characters they represent, so it will look like this:
"HelloöäüÖÄÜß"
Is there a way to replace all hex values with the actual character they represent?
I can achieve what I want with this, but I have to do one line for every character and it does not cover unexcepted characters:
indexRequest = indexRequest.replace("\\xF6", "ö");
indexRequest = indexRequest.replace("\\xE4", "ä");
indexRequest = indexRequest.replace("\\xFC", "ü");
indexRequest = indexRequest.replace("\\xD6", "Ö");
indexRequest = indexRequest.replace("\\xC4", "Ä");
indexRequest = indexRequest.replace("\\xDC", "Ü");
indexRequest = indexRequest.replace("\\xDF", "ß");

public static void main(String[] args) {
String s = "Hello\\xF6\\xE4\\xFC\\xD6\\xC4\\xDC\\xDF\\xFF ";
StringBuffer sb = new StringBuffer();
Pattern p = Pattern.compile("\\\\x[0-9A-F]+");
Matcher m = p.matcher(s);
while(m.find()){
String hex = m.group(); //find hex values
int num = Integer.parseInt(hex.replace("\\x", ""), 16); //parse to int
char bin = (char)num; // cast int to char
m.appendReplacement(sb, bin+""); // replace hex with char
}
m.appendTail(sb);
System.out.println(sb.toString());
}

I would loop through every chacter to find the '\' and than skip one char and start a methode with the next two chars.
And than just use the code by Michael Berry
here:
Convert a String of Hex into ASCII in Java

You can use a regex [xX][0-9a-fA-F]+ to identify all the hex code in your string, convert them to there corresponding character using Integer.parseInt(matcher.group().substring(1), 16) and replace them in string. Below is a sample code for it
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class HexToCharacter {
public static void main(String[] args) {
String s = "HelloxF6xE4xFCxD6xC4xDCxDF";
StringBuilder sb = new StringBuilder(s);
Pattern pattern = Pattern.compile("[xX][0-9a-fA-F]+");
Matcher matcher = pattern.matcher(s);
while(matcher.find()) {
int indexOfHexCode = sb.indexOf(matcher.group());
sb.replace(indexOfHexCode, indexOfHexCode+matcher.group().length(), Character.toString((char)Integer.parseInt(matcher.group().substring(1), 16)));
}
System.out.println(sb.toString());
}
}
I have tested this regex pattern using your string. If there are other test-cases that you have in mind, then you might need to change regex accordingly

How to make control characters visible?

I have to display string with visible control characters like \n, \t etc.
I have tried quotations like here, also I have tried to do something like
Pattern pattern = Pattern.compile("\\p{Cntrl}");
Matcher matcher = pattern.matcher(str);
String controlChar = matcher.group();
String replace = "\\" + controlChar;
result = result.replace(controlChar, replace);
but I have failed

Alternative: Use visible characters instead of escape sequences.
To make control characters "visible", use the characters from the Unicode Control Pictures Block, i.e. map \u0000-\u001F to \u2400-\u241F, and \u007F to \u2421.
Note that this requires output to be Unicode, e.g. UTF-8, not a single-byte code page like ISO-8859-1.
private static String showControlChars(String input) {
StringBuffer buf = new StringBuffer();
Matcher m = Pattern.compile("[\u0000-\u001F\u007F]").matcher(input);
while (m.find()) {
char c = m.group().charAt(0);
m.appendReplacement(buf, Character.toString(c == '\u007F' ? '\u2421' : (char) (c + 0x2400)));
if (c == '\n') // Let's preserve newlines
buf.append(System.lineSeparator());
}
return m.appendTail(buf).toString();
}
Output using method above as input text:
␉private static String showControlChars(String input) {␍␊
␉␉StringBuffer buf = new StringBuffer();␍␊
␉␉Matcher m = Pattern.compile("[\u0000-\u001F\u007F]").matcher(input);␍␊
␉␉while (m.find()) {␍␊
␉␉␉char c = m.group().charAt(0);␍␊
␉␉␉m.appendReplacement(buf, Character.toString(c == '\u007F' ? '\u2421' : (char) (c + 0x2400)));␍␊
␉␉␉if (c == '\n')␍␊
␉␉␉␉buf.append(System.lineSeparator());␍␊
␉␉}␍␊
␉␉return m.appendTail(buf).toString();␍␊
␉}␍␊

Simply replace occurences of '\n' with the escaped version (i.e. '\\n'), like this:
final String result = str.replace("\n", "\\n");
For example:
public static void main(final String args[]) {
final String str = "line1\nline2";
System.out.println(str);
final String result = str.replace("\n", "\\n");
System.out.println(result);
}
Will yield the output:
line1
newline
line1\nnewline

just doing
result = result.replace("\\", "\\\\");
will work!!

String operations:Split

I have a string consisting of file separators; e.g. "Bancs\Bancs_CP_P&MB.xml".
I want to separate the string based on "\".
Here's what I'm trying:
public class Stringoperations {
public static void main(String[] args) {
try {
String fileSeparator = System.getProperty("file.separator");
String TestCaseName = "Bancs" + fileSeparator + "Bancs_CP_P&MB.xml";
String[] tukde = TestCaseName.split("\\");
System.out.println(tukde[0]);
System.out.println(tukde[1]);
} catch (Exception e) {
e.getMessage();
} finally {
System.out.println("here");
}
}
}
But this is not working.

First: add a e.printStackTrace(); or something similar to your catch block, so you'll see what's actually wrong:
java.util.regex.PatternSyntaxException: Unexpected internal error near index 1
\
^
at java.util.regex.Pattern.error(Pattern.java:1924)
A back slash in Java string literals allows you place special chars into a string:
String withTab = "a\tb";
would print as "a b". To get a backslash in a Java string you need to escape it:
String withBackslash = "a\\b";
So this is what you done in the split invocation: you passed one java string back slash. Since String.split() evaluates the passed string a regular expression (Java Doc for String.split()), the back slash is treated as a RegEx. Backslash has a special meaning in regular expressions and cannot appear alone (Java Doc for Pattern). If you want a literal back slash you need to escape the back slash again:
String[] tukde = TestCaseName.split("\\\\");

First, putting that code into IntelliJ IDEA causes it to fuss at me with an illegal escape sequence. You have to escape the escape, so you'd be using \\\\ as valid backslash escape syntax.
Second, you should be splitting on fileSeparator, not an arbitrary backslash. The backslash actually varies from system to system (e.g. I'm on Linux Mint, and my separators are all forward slashes).
String[] tukde = TestCaseName.split(fileSeparator);
As a further note, there's no exceptions here that could be thrown (save for runtime), and blindly catching all exceptions isn't a good practice.

Try this code :-
public static void main(String[] args) {
try
{
String fileSeparator = System.getProperty("file.separator");
String TestCaseName = "Bancs"+fileSeparator+"Bancs_CP_P&MB.xml";
String[] tukde = TestCaseName.split("\\\\");
System.out.println(tukde[0]);
System.out.println(tukde[1]);
}catch(Exception e)
{
e.getMessage();
}
finally
{
System.out.println("here");
}
}
Out put :-
Bancs
Bancs_CP_P&MB.xml
here

What platform or OS are you working on. Maybe your default file separator is not "\". Try this :
public class Stringoperations {
public static void main(String[] args) {
try {
String fileSeparator = System.getProperty("file.separator");
String TestCaseName = "Bancs" + fileSeparator + "Bancs_CP_P&MB.xml";
String[] tukde = TestCaseName.split(fileSeparator); //This uses the default separator returned by System.getProperty
System.out.println(tukde[0]);
System.out.println(tukde[1]);
} catch (Exception e) {
e.getMessage();
} finally {
System.out.println("here");
}
}
}
EDIT :
Also as Makoto points out if you are bent on using "\" to split you need to use "\\" and not "\"

Try this:
String pattern = Pattern.quote(System.getProperty("file.separator"));
String[] splittedFileName = fileName.split(pattern);

public class Stringoperations {
public static void main(String[] args) {
try {
String fileSeparator = System.getProperty("file.separator");
String TestCaseName = "Bancs" + fileSeparator + "Bancs_CP_P&MB.xml";
String[] tukde = TestCaseName.split("\\\\");
System.out.println(tukde[0]);
System.out.println(tukde[1]);
} catch (Exception e) {
e.getMessage();
} finally {
System.out.println("here");
}
}
}
try this

How to parse UTF-8 representation to String in Java?

Given the following code:
String tmp = new String("\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a");
String result = convertToEffectiveString(tmp); // result contain now "hello\n"
Does the JDK already provide some classes for doing this ?
Is there a libray that does this ? (preferably under maven)
I have tried with ByteArrayOutputStream with no success.

This works, but only with ASCII. If you use unicode characters outside of the ASCCI range, then you will have problems (as each character is being stuffed into a byte, instead of a full word that is allowed by UTF-8). You can do the typecast below because you know that the UTF-8 will not overflow one byte if you guaranteed that the input is basically ASCII (as you mention in your comments).
package sample;
import java.io.UnsupportedEncodingException;
public class UnicodeSample {
public static final int HEXADECIMAL = 16;
public static void main(String[] args) {
try {
String str = "\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a";
String arr[] = str.replaceAll("\\\\u"," ").trim().split(" ");
byte[] utf8 = new byte[arr.length];
int index=0;
for (String ch : arr) {
utf8[index++] = (byte)Integer.parseInt(ch,HEXADECIMAL);
}
String newStr = new String(utf8, "UTF-8");
System.out.println(newStr);
}
catch (UnsupportedEncodingException e) {
// handle the UTF-8 conversion exception
}
}
}
Here is another solution that fixes the issue of only working with ASCII characters. This will work with any unicode characters in the UTF-8 range instead of ASCII only in the first 8-bits of the range. Thanks to deceze for the questions. You made me think more about the problem and solution.
package sample;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
public class UnicodeSample {
public static final int HEXADECIMAL = 16;
public static void main(String[] args) {
try {
String str = "\\u0068\\u0065\\u006c\\u006c\\u006f\\u000a\\u3fff\\uf34c";
ArrayList<Byte> arrList = new ArrayList<Byte>();
String codes[] = str.replaceAll("\\\\u"," ").trim().split(" ");
for (String c : codes) {
int code = Integer.parseInt(c,HEXADECIMAL);
byte[] bytes = intToByteArray(code);
for (byte b : bytes) {
if (b != 0) arrList.add(b);
}
}
byte[] utf8 = new byte[arrList.size()];
for (int i=0; i<arrList.size(); i++) utf8[i] = arrList.get(i);
str = new String(utf8, "UTF-8");
System.out.println(str);
}
catch (UnsupportedEncodingException e) {
// handle the exception when
}
}
// Takes a 4 byte integer and and extracts each byte
public static final byte[] intToByteArray(int value) {
return new byte[] {
(byte) (value >>> 24),
(byte) (value >>> 16),
(byte) (value >>> 8),
(byte) (value)
};
}
}

Firstly, are you just trying to parse a string literal, or is tmp going to be some user-entered data?
If this is going to be a string literal (i.e. hard-coded string), it can be encoded using Unicode escapes. In your case, this just means using single backslashes instead of double backslashes:
String result = "\u0068\u0065\u006c\u006c\u006f\u000a";
If, however, you need to use Java's string parsing rules to parse user input, a good starting point might be Apache Commons Lang's StringEscapeUtils.unescapeJava() method.

I'm sure there must be a better way, but using just the JDK:
public static String handleEscapes(final String s)
{
final java.util.Properties props = new java.util.Properties();
props.setProperty("foo", s);
final java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
try
{
props.store(baos, null);
final String tmp = baos.toString().replace("\\\\", "\\");
props.load(new java.io.StringReader(tmp));
}
catch(final java.io.IOException ioe) // shouldn't happen
{ throw new RuntimeException(ioe); }
return props.getProperty("foo");
}
uses java.util.Properties.load(java.io.Reader) to process the backslash-escapes (after first using java.util.Properties.store(java.io.OutputStream, java.lang.String) to backslash-escape anything that would cause problems in a properties-file, and then using replace("\\\\", "\\") to reverse the backslash-escaping of the original backslashes).
(Disclaimer: even though I tested all the cases I could think of, there are still probably some that I didn't think of.)

How to replace characters in a java String?

I like to replace a certain set of characters of a string with a corresponding replacement character in an efficent way.
For example:
String sourceCharacters = "šđćčŠĐĆČžŽ";
String targetCharacters = "sdccSDCCzZ";
String result = replaceChars("Gračišće", sourceCharacters , targetCharacters );
Assert.equals(result,"Gracisce") == true;
Is there are more efficient way than to use the replaceAll method of the String class?
My first idea was:
final String s = "Gračišće";
String sourceCharacters = "šđćčŠĐĆČžŽ";
String targetCharacters = "sdccSDCCzZ";
// preparation
final char[] sourceString = s.toCharArray();
final char result[] = new char[sourceString.length];
final char[] targetCharactersArray = targetCharacters.toCharArray();
// main work
for(int i=0,l=sourceString.length;i<l;++i)
{
final int pos = sourceCharacters.indexOf(sourceString[i]);
result[i] = pos!=-1 ? targetCharactersArray[pos] : sourceString[i];
}
// result
String resultString = new String(result);
Any ideas?
Btw, the UTF-8 characters are causing the trouble, with US_ASCII it works fine.

You can make use of java.text.Normalizer and a shot of regex to get rid of the diacritics of which there exist much more than you have collected as far.
Here's an SSCCE, copy'n'paste'n'run it on Java 6:
package com.stackoverflow.q2653739;
import java.text.Normalizer;
import java.text.Normalizer.Form;
public class Test {
public static void main(String... args) {
System.out.println(removeDiacriticalMarks("Gračišće"));
}
public static String removeDiacriticalMarks(String string) {
return Normalizer.normalize(string, Form.NFD)
.replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
}
}
This should yield
Gracisce
At least, it does here at Eclipse with console character encoding set to UTF-8 (Window > Preferences > General > Workspace > Text File Encoding). Ensure that the same is set in your environment as well.
As an alternative, maintain a Map<Character, Character>:
Map<Character, Character> charReplacementMap = new HashMap<Character, Character>();
charReplacementMap.put('š', 's');
charReplacementMap.put('đ', 'd');
// Put more here.
String originalString = "Gračišće";
StringBuilder builder = new StringBuilder();
for (char currentChar : originalString.toCharArray()) {
Character replacementChar = charReplacementMap.get(currentChar);
builder.append(replacementChar != null ? replacementChar : currentChar);
}
String newString = builder.toString();

I'd use the replace method in a simple loop.
String sourceCharacters = "šđćčŠĐĆČžŽ";
String targetCharacters = "sdccSDCCzZ";
String s = "Gračišće";
for (int i=0 ; i<sourceCharacters.length() ; i++)
s = s.replace(sourceCharacters.charAt[i], targetCharacters.charAt[i]);
System.out.println(s);