We Keep Coding

How to find Invalid UTF-8 Character in oracle column

I have oracle table in which I am storing XML file , column is of CLOB type . Then we picked that xml file for further processing . It is somewhere breaking with below exception
"com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 start byte 0xa0 (at char #931, byte #20)"
When we copy the content in notepad++ ,it didn't show any invalid UTF-8 Character.
Could any one help how to find invalid UTF-8 character in XML file in oracle column , request you to considering column is of CLOB type.
ANy help is greatly appreciated

Do you have access to Unix? You can use iconv -f utf-8 -t utf-8 -c yourfile.xml. You can find more possible options in this thread.

Solr ERROR #500 IOE

Anyone can tell me what could be to cause this problem?
I tried to post with post.jar a file xml; i copt below the server log
118208 [qtp760665089-18] ERROR org.apache.solr.servlet.SolrDispatchFilter û nul
l:java.lang.RuntimeException: [was class java.io.CharConversionException] Invali
d UTF-8 middle byte 0x6c (at char #139212, byte #136949)
at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.j
ava:18)at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.j
ava:3657)at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:397)
at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java
:246)
[...]
Caused by: java.io.CharConversionException: Invalid UTF-8 middle byte 0x6c (at c
har #139212, byte #136949)
at com.ctc.wstx.io.UTF8Reader.reportInvalidOther(UTF8Reader.java:313)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:204)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.
java:57)...

You have 1 or more illegal (e.g. not UTF-8) characters in your document:
http://www.coderanch.com/t/433718/XML/Invalid-UTF-middle-byte-error
I'd take a close look at the document and consider stripping/filtering for only UTF-8
This previous stackoverflow answer has a couple of code snippets in Perl and Java for filtering out non UTF-8 characters:
How to remove bad characters that are not suitable for utf8 encoding in MySQL?

java.io.UnsupportedEncodingException for UCS-2

I have some Hungarian text and I would like it to be encoded with UCS2 encoding
String stringEncoding = "UCS-2";
String contentHardCoded = new String("szigorúan bejelentkezési azonosításhoz".getBytes(),stringEncoding);
But I am getting the following exception
Exception in thread "main" java.io.UnsupportedEncodingException: UCS-2
at java.lang.StringCoding.decode(StringCoding.java:170)
at java.lang.String.<init>(String.java:443)
at java.lang.String.<init>(String.java:515)
at com.gtl.mindmatics.sms.Main.sendSMS(Main.java:108)
at com.gtl.mindmatics.sms.Main.main(Main.java:180)
Java Result: 1
What could be wrong?
EDIT
I use the following command to run my jar
Actually my command is like
java -Dfile.encoding=UCS-2 -cp MyApp.jar com.sms.Main "9876543210" “UCS-2” > testApp.log
And also what should be the correct encoding that should be used, i used UTF-8 but the output not correct.

You're doing it wrong; a String is a set of characters and that is all. What you do here is:
you get the bytes of the string as decoded by your current JVM encoding,
you reencode these bytes using a different encoding.
Your string will therefore be completely corrupted. A String does not have an encoding.
See here for more details.
As to UCS-2, it has been superseded by UTF-16. You want to use UTF-16 instead.
Note that you MUST specify the endianness, which matters for UTF-16 unlike for UTF-8. Use:
StandardCharsets.UTF_16LE
(or BE for big endian), or, if you still use Java 6 or lower:
Charset.forName("UTF-16LE") // or BE

I have UTF-8 - but still get "Invalid byte 1 of 1-byte UTF-8 sequence"

I create a XML String on the fly (NOT reading from a file). Then I use Cocoon 3 to transform it via FOP to a PDF. Somewhere in the middle Xerces runs. When I use the hardcoded stuff everything works. As soon as I put a german Umlaut into the database and enrich my xml with that data I get:
Caused by: org.apache.cocoon.pipeline.ProcessingException: Can't parse the XML string.
at org.apache.cocoon.sax.component.XMLGenerator$StringGenerator.execute(XMLGenerator.java:326)
at org.apache.cocoon.sax.component.XMLGenerator.execute(XMLGenerator.java:104)
at org.apache.cocoon.pipeline.AbstractPipeline.invokeStarter(AbstractPipeline.java:146)
at org.apache.cocoon.pipeline.AbstractPipeline.execute(AbstractPipeline.java:76)
at de.grobmeier.tab.webapp.modules.documents.InvoicePipeline.generateInvoice(InvoicePipeline.java:74)
... 87 more
Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:684)
at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:554)
I have then debugged my app and found out, my "Ä" (which comes frome the database) has the byte value of 196, which is C4 in hex. This is what I have expected according to this: http://www.utf8-zeichentabelle.de/
I do not know why my code fails.
I have then tried to add a BOM manually, like that:
byte[] bom = new byte[3];
bom[0] = (byte) 0xEF;
bom[1] = (byte) 0xBB;
bom[2] = (byte) 0xBF;
String myString = new String(bom) + inputString;
I know this is not exactly good, but I tried it - of course it failed. I have tried to add a xml header in front:
<?xml version="1.0" encoding="UTF-8"?>
Which failed too. Then I combined it. Failed.
After all I tried something like that:
xmlInput = new String(xmlInput.getBytes("UTF8"), "UTF8");
Which is doing nothing in fact, because it is already UTF-8. Still it fails.
So... any ideas what I am doing wrong and what Xerces is expecting from me?
Thanks
Christian

If your database contains only a single byte (with value 0xC4) then you aren't using UTF-8 encoding.
The character "LATIN CAPITAL LETTER A WITH DIAERESIS" has a code-point value U+00C4, but UTF-8 can't encode that in a single byte. If you check the third column "UTF-8 (hex.)" on UTF8-zeichentabelle.de you'll see that UTF-8 encodes that as 0xC3 84 (two bytes).
Please read Joel's article "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" for more info.
EDIT: Christian found the answer himself; turned out it was a problem in the Cocoon 3 SAX component (I guess it's the alpha 3 version). It turns out that if you pass an XML as a String into the XMLGenerator class, something will go wrong during SAX parsing causing this mess.
I looked up the code to find the actual problem in Cocoon-stax:
if (XMLGenerator.this.logger.isDebugEnabled()) {
XMLGenerator.this.logger.debug("Using a string to produce SAX events.");
}
XMLUtils.toSax(new ByteArrayInputStream(this.xmlString.getBytes()), XMLGenerator.this.getSAXConsumer();
As you can see, the call getBytes() will create a Byte array with the JRE's default encoding which will then fail to parse. This is because the XML declares itself to be UTF-8 whereas the data is now in bytes again, and likely using your Windows codepage.
As a workaround, one can use the following:
new org.apache.cocoon.sax.component.XMLGenerator(xmlInput.getBytes("UTF-8"),
"UTF-8");
This will trigger the right internal actions (as Christian found out by experimenting with the API).
I've opened an issue in Apache's bug tracker.
EDIT 2: The issue is fixed and will be included in an upcoming release.

The C4 you see on that page refers to the unicode code point, U+00C4. The byte sequence used to represent such a code point in UTF-8 is NOT "\xC4". What you want is what's in the UTF-8 (hex.) column, namely "\xC3\x84".
Therefore, your data is not in UTF-8.
You can read about how data is encoded in UTF-8 here.

I'm running Windows 7 with TextPad as a text editor for manually building the xml data file. I was getting the MalformedByteSequenceException. My spec in the xml file was UTF-8. After poking around, I found that my editor had a tool "Tools ... Convert to DOS". I did that, re-saved the file, and the exception went away and my code ran fine.
I then looked at the default encoding for that file type in my editor. It was ASCII, though when I changed the xml encoding parameter to ASCII, I got another different MalformedByteSequenceException.
So on Windows systems, you might try keeping the xml encoding to UTF-8, but save the file encoded DOS. I did not dig any further as to why this works.

java unicode encoded file reading problem in jdk 1.3

I am using jdk1.3 for blackberry platform. Now I am facing a problem when I trying to read an Unicode encoded xml file.
My code :
java.io.BufferedReader br = new java.io.BufferedReader(new java.io.InputStreamReader(new java.io.FileInputStream(path),"UTF16"));
br.readLine();
Error:
sun.io.MalformedInputException: Missing byte-order mark
at sun.io.ByteToCharUnicode.convert(ByteToCharUnicode.java:123)
at java.io.InputStreamReader.convertInto(InputStreamReader.java:137)
at java.io.InputStreamReader.fill(InputStreamReader.java:186)
at java.io.InputStreamReader.read(InputStreamReader.java:249)
at java.io.BufferedReader.fill(BufferedReader.java:139)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
Thanks

You XML file is missing a byte order mark.
In JDK 1.3, the byte order mark is mandatory if you use UTF-16. Try the UTF16-LE or -BE if you know in advance what the endianness is.
(The BOM is not mandatory in 1.4.2 and above.)
Of course, if your file is not UTF-16 at all, use the correct encoding. See the above link to character encodings. The actual encodings supported, apart from a small set of core encodings, are implementation defined so you'll need to check the docs for your particular JDK.
The encoding the files are in is supposed to be in the <xml> header of your files, something like:
<?xml version="1.0" encoding="THIS IS THE ENCODING YOU NEED TO USE"?>
If the file is in a single character encoding, or UTF-8 (without a BOM), You can try reading the first line with plain US-ASCII, it shouldn't contain any data outside that range. Parse the encoding field, then re-open the file with the deduced encoding.
This will only work if the actual encoding is supported by your platform obviously.
BTW: JDK 1.3 is ancient. Are you sure that's your version? (Doesn't change anything to the problem anyway except for the BOM part)

Try this code:
java.io.BufferedReader br = new java.io.BufferedReader(new java.io.InputStreamReader(new java.io.FileInputStream(path),"Windows-1256"));
br.readLine();