We Keep Coding

Character Encoding Conversion In Groovy From UTF-8 to EUC-JP

We require character encoding conversion for one of our service, our requirement is to fetch characters in UTF-8 encoded format and should convert to EUC-JP then prepare some hashing on (Groovy based on) jdk8.
In php, similar solution works fine for us and coded as,
$encodedToEucJp = mb_convert_encoding($inputStringWithUtf8, “EUC-JP”);
Print_r(md5($encodedToEucJp));
We have tried many ways for the solution, e.g.,
Java.security.MessageDigest.getInstance(‘MD5’)
.digest(New String(inputStringWithUtf8.getBytes(“UTF-8”), “EUC-JP”)
.getBytes(“EUC-JP”))
.encodeHex()
.toString();
But, this solution failed for some of the characters that produces different digest then from our php coded solution. Here few characters are mentioned ―, ĭ, ? etc. That’s the reason why we couldn't product same digest with same input both in php and java system.
Thanks, in advance.

The error is in this part of the code:
New String(inputStringWithUtf8.getBytes(“UTF-8”), “EUC-JP”)
Basically, you try to interpret an UTF-8 byte array as if it were encoded in EUC-JP, which is a non-sense.
The following code should do the job
Java.security.MessageDigest.getInstance(‘MD5’)
.digest(inputStringWithUtf8.getBytes(“EUC-JP”))
.encodeHex()
.toString();

MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence

My file is already encoded with UTF-8 in notepad++, why I am still getting this error?
by the way, I'm using jasperreports. .jrxml
Oct 30, 2017 12:35:07 AM frmStart jButton_repActionPerformed
SEVERE: null
net.sf.jasperreports.engine.JRException: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:249)
at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:228)
at net.sf.jasperreports.engine.xml.JRXmlLoader.load(JRXmlLoader.java:216)
at net.sf.jasperreports.engine.xml.JRXmlLoader.load(JRXmlLoader.java:170)
at net.sf.jasperreports.engine.xml.JRXmlLoader.load(JRXmlLoader.java:154)
at net.sf.jasperreports.engine.JasperCompileManager.compileReportToFile(JasperCompileManager.java:85)
at frmStart.jButton_repActionPerformed(frmStart.java:556)

I believe the situation is that Xerces will accept UTF-8 input with a BOM if given a (binary) InputStream to read from, but if you give it a (character-based) reader as input, it will ask the Java I/O library to do the decoding, and that will fail.
So it depends how Jasper Reports is invoking Xerces. I don't know if you have any control over that.

How to find Invalid UTF-8 Character in oracle column

I have oracle table in which I am storing XML file , column is of CLOB type . Then we picked that xml file for further processing . It is somewhere breaking with below exception
"com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 start byte 0xa0 (at char #931, byte #20)"
When we copy the content in notepad++ ,it didn't show any invalid UTF-8 Character.
Could any one help how to find invalid UTF-8 character in XML file in oracle column , request you to considering column is of CLOB type.
ANy help is greatly appreciated

Do you have access to Unix? You can use iconv -f utf-8 -t utf-8 -c yourfile.xml. You can find more possible options in this thread.

java.io.UnsupportedEncodingException for UCS-2

I have some Hungarian text and I would like it to be encoded with UCS2 encoding
String stringEncoding = "UCS-2";
String contentHardCoded = new String("szigorúan bejelentkezési azonosításhoz".getBytes(),stringEncoding);
But I am getting the following exception
Exception in thread "main" java.io.UnsupportedEncodingException: UCS-2
at java.lang.StringCoding.decode(StringCoding.java:170)
at java.lang.String.<init>(String.java:443)
at java.lang.String.<init>(String.java:515)
at com.gtl.mindmatics.sms.Main.sendSMS(Main.java:108)
at com.gtl.mindmatics.sms.Main.main(Main.java:180)
Java Result: 1
What could be wrong?
EDIT
I use the following command to run my jar
Actually my command is like
java -Dfile.encoding=UCS-2 -cp MyApp.jar com.sms.Main "9876543210" “UCS-2” > testApp.log
And also what should be the correct encoding that should be used, i used UTF-8 but the output not correct.

You're doing it wrong; a String is a set of characters and that is all. What you do here is:
you get the bytes of the string as decoded by your current JVM encoding,
you reencode these bytes using a different encoding.
Your string will therefore be completely corrupted. A String does not have an encoding.
See here for more details.
As to UCS-2, it has been superseded by UTF-16. You want to use UTF-16 instead.
Note that you MUST specify the endianness, which matters for UTF-16 unlike for UTF-8. Use:
StandardCharsets.UTF_16LE
(or BE for big endian), or, if you still use Java 6 or lower:
Charset.forName("UTF-16LE") // or BE

I have UTF-8 - but still get "Invalid byte 1 of 1-byte UTF-8 sequence"

I create a XML String on the fly (NOT reading from a file). Then I use Cocoon 3 to transform it via FOP to a PDF. Somewhere in the middle Xerces runs. When I use the hardcoded stuff everything works. As soon as I put a german Umlaut into the database and enrich my xml with that data I get:
Caused by: org.apache.cocoon.pipeline.ProcessingException: Can't parse the XML string.
at org.apache.cocoon.sax.component.XMLGenerator$StringGenerator.execute(XMLGenerator.java:326)
at org.apache.cocoon.sax.component.XMLGenerator.execute(XMLGenerator.java:104)
at org.apache.cocoon.pipeline.AbstractPipeline.invokeStarter(AbstractPipeline.java:146)
at org.apache.cocoon.pipeline.AbstractPipeline.execute(AbstractPipeline.java:76)
at de.grobmeier.tab.webapp.modules.documents.InvoicePipeline.generateInvoice(InvoicePipeline.java:74)
... 87 more
Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:684)
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:554)
I have then debugged my app and found out, my "Ä" (which comes frome the database) has the byte value of 196, which is C4 in hex. This is what I have expected according to this: http://www.utf8-zeichentabelle.de/
I do not know why my code fails.
I have then tried to add a BOM manually, like that:
byte[] bom = new byte[3];
bom[0] = (byte) 0xEF;
bom[1] = (byte) 0xBB;
bom[2] = (byte) 0xBF;
String myString = new String(bom) + inputString;
I know this is not exactly good, but I tried it - of course it failed. I have tried to add a xml header in front:
<?xml version="1.0" encoding="UTF-8"?>
Which failed too. Then I combined it. Failed.
After all I tried something like that:
xmlInput = new String(xmlInput.getBytes("UTF8"), "UTF8");
Which is doing nothing in fact, because it is already UTF-8. Still it fails.
So... any ideas what I am doing wrong and what Xerces is expecting from me?
Thanks
Christian

If your database contains only a single byte (with value 0xC4) then you aren't using UTF-8 encoding.
The character "LATIN CAPITAL LETTER A WITH DIAERESIS" has a code-point value U+00C4, but UTF-8 can't encode that in a single byte. If you check the third column "UTF-8 (hex.)" on UTF8-zeichentabelle.de you'll see that UTF-8 encodes that as 0xC3 84 (two bytes).
Please read Joel's article "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" for more info.
EDIT: Christian found the answer himself; turned out it was a problem in the Cocoon 3 SAX component (I guess it's the alpha 3 version). It turns out that if you pass an XML as a String into the XMLGenerator class, something will go wrong during SAX parsing causing this mess.
I looked up the code to find the actual problem in Cocoon-stax:
if (XMLGenerator.this.logger.isDebugEnabled()) {
XMLGenerator.this.logger.debug("Using a string to produce SAX events.");
}
XMLUtils.toSax(new ByteArrayInputStream(this.xmlString.getBytes()), XMLGenerator.this.getSAXConsumer();
As you can see, the call getBytes() will create a Byte array with the JRE's default encoding which will then fail to parse. This is because the XML declares itself to be UTF-8 whereas the data is now in bytes again, and likely using your Windows codepage.
As a workaround, one can use the following:
new org.apache.cocoon.sax.component.XMLGenerator(xmlInput.getBytes("UTF-8"),
"UTF-8");
This will trigger the right internal actions (as Christian found out by experimenting with the API).
I've opened an issue in Apache's bug tracker.
EDIT 2: The issue is fixed and will be included in an upcoming release.

The C4 you see on that page refers to the unicode code point, U+00C4. The byte sequence used to represent such a code point in UTF-8 is NOT "\xC4". What you want is what's in the UTF-8 (hex.) column, namely "\xC3\x84".
Therefore, your data is not in UTF-8.
You can read about how data is encoded in UTF-8 here.

I'm running Windows 7 with TextPad as a text editor for manually building the xml data file. I was getting the MalformedByteSequenceException. My spec in the xml file was UTF-8. After poking around, I found that my editor had a tool "Tools ... Convert to DOS". I did that, re-saved the file, and the exception went away and my code ran fine.
I then looked at the default encoding for that file type in my editor. It was ASCII, though when I changed the xml encoding parameter to ASCII, I got another different MalformedByteSequenceException.
So on Windows systems, you might try keeping the xml encoding to UTF-8, but save the file encoded DOS. I did not dig any further as to why this works.